Multi-modal user interface

1 2

language dialogue between a user and a computer dataMULTI-MODAL USER INTERFACE base system:

USER: Which sales staff report to the Head Office? CROSS REFERENCE TO RELATED COMPUTER: Alecia Andrews, Bill Bronson, Carl

APPLICATION(S) 5 demson.

This is a continuation of copending application Ser. USER: Which sales staff visited Pinewood in April? No. 07/507,340 filed on Apr. 10, 1990, now abandoned. COMPUTER: Marcia Martin, Nancy Novarro, Oscar

Ottoman.

BACKGROUND OF THE INVENTION USER; Wnich of these Bve m

The present invention relates generally to computer ^ The word "these" in the third query is ambiguous. Does

database systems and more particularly to a user inter- "these" refer to persons who report to the head office,

face which facilitates natural language communication or to those who visited Pinewood, or to both? Using the

between a user and a database by providing an easy way WOrd "these" creates a referential ambiguity, that is, an

for a user to remove ambiguities from a natural lan- ambiguous reference to a previous item of information,

guage query of the database. 15 A referential ambiguity can be resolved by asking the

Computer database systems are being used to store for more information, for example as follows:

and manage more and more different kinds of data. USER: Which sales staff report to the Head Office?

Airhne reservation systems, bank account information COMPUTER: Alecia Andrews, BUI Bronson, Carl

systems, and computerized library card indices are but a Clemson

few examples of such systems. Many of these systems 20 ugER ^ ^ ... m A ^

are now becoming available for use by the general pub- COMPUTER: Marcia Martin, Nancy Novarro, Oscar

he. For example, until recently airline reservations Ottoman

could only be booked by trained reservation clerks, but „._„ , ... ,. . T , .

now anyone having a home computer and a modem can VS^? °Tf theSe 1,Ve m JLond°f ,

have direct access to airline reservation computers. 25 COMPUTER: I cannot understand the question.

Similarly, many public libraries are placing computer TbeK ,s ambiguous. Please rephrase your question

terminals about the library for use by library patrons in or Press • for helP in usm« the system

finding books and other references. USER: which of those who visited Pinewood?

A common problem in learning to use computers is COMPUTER: I cannot understand the question,

that computers require strict compliance with a set of 30 "Those" is ambiguous. What do you want to know

precise grammatical rules governing communication about "those who visited Pinewood"? Please re

between the computer and the user. The SQL database phrase your question or press "?" for help in using the

language is an example of such a set of rules. This kind system.

of precise communication is very unnatural for humans USER: (exasperated) Which of the sales staff who vis

because most people are accustomed to speaking and 35 ited Pinewood in April live in London?

writing in "natural language", that is, everyday spoken COMPUTER: Nancy Novarro and Oscar Ottoman.

English which is often ungrammatical and ambiguous Such a method of resolving ambiguities can be highly

but which nevertheless is easily understandable by irritating to a user and often results in much wasted time

other humans. as the user tries to formulate unambiguous questions. To

As long as computers were used only by a relatively 40 avoid this problem, various attempts have been made to

small number of specialists, this need to use precise rules develop natural language processing technologies that

in communicating with computers was not perceived as can automatically resolve such ambiguities. However,

a major problem. Humans are perfectly capable of these methods generally involve complex reasoning

learning to use languages such as SQL to communicate about the user Md the information in the database and

with computers. Persons such as airline and travel 45 have not adequateiy solved the problem,

agency reservation clerks were trained to use such lan- ft wil] ^ a, from the foregoing that there is a

guages to communicate with computers, and these nee(J fQ[ a meI.Mendl t0 reso]ve a referential

trained specialists then took care of user-computer com- ^ it ^ a natural j of a ^p^r

munication for everyone else. database system

As computer database systems become more widely 50 J

accessible, more people wish to use them. In addition, SUMMARY OF THE INVENTION

many people are finding that they no longer have any . . ., ... , .

choice-for example, libraries are abolishing the tradi- . TMe P^nt invention provides a multi-modal user

tional card index in favor of the computerized index, mtel?ace wluch States natural language commumca

and patrons of such libraries can no longer locate books 55 t,on betwee" a user-and a comPuter database by provid

except by using the computerized index system. m8 means for a user to remove a referential ambiguity

Most people have neither the time nor the desire to from a WW by pointing to a textual reference on a learn computer languages. Accordingly a need has VKual display. ^ interface is easy for users to learn arisen for computer database systems that are usable by 80(1 ^ demands minimal computer resources, and persons having no formal training in computer database 60 avoids user irritation and wasted tune, languages. Computers, in other words, must be "user Briefly and in general terms, a multi-modal user interfriendly" to all comers, and to this end much effort is face according to the invention includes a keyboard or being devoted to designing computer systems that can the like for receiving a textual communication from a communicate with people in "natural language", that is user, a mouse or the like for receiving a referential input in ordinary human speech. 65 from the user, and computer resources that detect any

One of the problems of designing a computer system referential ambiguity in the text and use the referential

that can communicate in natural language is interpret- input to remove the ambiguity to provide an unambigu

ing ambiguous queries. Consider the following natural ous communication from the user.

3 4

In a typical interactive database session, a user asks a system uses the referential input to resolve any referenstring of questions some of which may refer back to tial ambiguity in the textual input and thereby provide responses to earlier questions. Such a reference is often an unambiguous input which can then be used, for exambiguous in that it is not clear which question or re- ample, to obtain information from a database. An intersponse is being referred to. In accordance with a pre- 5 face according to the invention provides a user-friendly, ferred embodiment of the present invention, the previ- easily-learned way to resolve referential ambiguities in a ous questions and responses are identified and displayed natural language textual input such as a database query, on a computer screen. The user points to any number of As shown in FIG. 1, a multi-modal user interface elements in one or more previous responses to which according to the invention comprises input means such he/she intends to refer. 10 as a keyboard 11 and a mouse 13 for receiving a textual

It will be apparent that speech recognition apparatus input and a referential input, respectively, from a user

or other means may be used to receive textual input (not shown); means, for example a portion of a com

from the user. Similarly, a touch screen, speech recogni- puter system generally designated 15 for detecting a

don apparatus, or other pointer including cursor keys or referential ambiguity in the textual input; and means

the like may be used to receive the referential input. 15 such as a portion of the computer system 15 for remov

A method of resolving ambiguities preferably utilizes ing the ambiguity according to the referential input to

query cards to arrange and display user queries and provide an unambiguous textual input,

system responses. The method includes the steps of Instead of (or in addition to) the keyboard 11, any of

receiving a database query and a referential input from a variety of other input means known in the art can be

a user, identifying any referential ambiguity in the 20 used to receive textual input from the user. One embodi

query, resolving the ambiguity according to the referen- ment, for example, includes means such as a micro

tial input, evaluating the query, and providing a re- phone 17 and associated hardware and software in the

sponse to the user. computer system 15 responsive to human speech to

A query is preferably displayed in the form of a query provide a digital representation of the spoken word. By

card which has a portion for displaying the query and a 25 utilizing this apparatus the user communicates textual

portion for displaying a response to the query. A list of information to the computer system by talking to the

previous queries is also displayed. The user selects one computer rather than by typing the information at the

or several previous query cards for display and uses the keyboard. The microphone receives the user's spoken

mouse or other pointer to identify elements in the re- words and provides an analog signal, and the hardware

sponses on the cards as the referential input for the next 30 and software convert the analog signal into a digital

query. signal which the computer uses just as it would use

Other aspects and advantages of the present inven- typed keyboard input,

tion will become apparent from the following detailed Likewise, rather than the mouse 13 alternate embodi

description, taken in conjunction with the accompany- ments provide a touch-sensitive screen (not shown),

ing drawings, illustrating by way of example the princi- 35 speech recognition means, designated keys such as cur

ples of the invention. sor keys on the keyboard 11, or other input means for

RRTFF nFSOR TPTION OF THF DR A WTNfiS receiving the referential input from the user. The refer

BRIEF DESCRIPTION OF THE DRAWINGS ential mput essentialiy amounts t0 a selection of one or

FIG. 1 shows a computer database system including a more textual elements in previous responses that are

multi-modal interface that displays database queries in 40 being displayed on the screen; the user makes the selec

query cards according to the present invention; tion by pointing to the desired elements rather than by

FIG. 2 is a detail view of a query card of the kind having to type any text,

being displayed on the screen of the system of FIG. 1; The invention is advantageously embodied in a user

FIG. 3 is a detail view of the screen display as shown interface of a database system which receives a database

in FIG. 1; 45 query from a user, evaluates the query, and provides a

FIG. 4 is a view of an alternative arrangement of the result of the evaluation to the user. The database system

display of FIG. 3; uses a referential input from the user to remove a refer

FIG. 5 is a flow chart of a method of resolving ambi- ential ambiguity in the query and thereby provide an

guities in queries of a database system such as that de- unambiguous query for evaluation,

picted in FIG. 1; and 50 A preferred way of displaying the user's query and

FIG. 6 is a view similar to that of FIG. 1 and showing the system's response according to the invention com

an ambiguous textual query and a referential input that prises a query card such as the card 19 as best shown in

is used to resolve the ambiguity according to the inven- FIG. 2. The card 19 has a first region 21 for displaying

tion. the query and a second region 23 for displaying the

nF^rRTPTTDTM OF Thf PRFFFRRFn 55 resPonse to the querv- The card 19 physically consists

DESCRIPTION OF THE PREFERRED of m area of a computer display screen such as the

fcMBODIMEN I 25) nQt a physical ^ card 19 preferabiy

As shown in the drawings for purposes of illustration, includes a title bar 27 for identifying the card; the card

the invention is embodied in a novel multi-modal user 19 is identified in the title bar by the legend "Query

interface that makes it easy for a user to resolve referen- 60 Card 2".

tial ambiguities in natural language database queries. As As shown in FIG. 1 and in more detail in FIG. 3, the

more and more people are using computer database display screen 25 typically shows a plurality of query

systems," systems that can respond to natural language cards such as the card 19 and another card 29 arranged

queries have been developed. There has been a need for in a workspace 31, each card corresponding with a

a user-friendly way to resolve ambiguities in such 65 different query. The screen also includes a list of succes

queries. sive queries 33, 35 and 37 in a list or "history" area 39.

In accordance with the invention, a user provides a In the example shown, the query 35 corresponds with

textual input and a referential input. The computer the query card 19 and the query 37 corresponds with