CA2007411C - Advanced user interface - Google Patents
Advanced user interfaceInfo
- Publication number
- CA2007411C CA2007411C CA002007411A CA2007411A CA2007411C CA 2007411 C CA2007411 C CA 2007411C CA 002007411 A CA002007411 A CA 002007411A CA 2007411 A CA2007411 A CA 2007411A CA 2007411 C CA2007411 C CA 2007411C
- Authority
- CA
- Canada
- Prior art keywords
- input
- user
- computer system
- active
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/10—Program control for peripheral devices
- G06F13/105—Program control for peripheral devices where the programme performs an input/output emulation function
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F3/023—Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
- G06F9/45508—Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
- G06F9/45512—Command shells
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/038—Indexing scheme relating to G06F3/038
- G06F2203/0381—Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
Abstract
An advanced user interface for use with a computer system operating on an integrated operating environment.
The integrated operating environment allows a plurality of application programs to be running simultaneously, one of which is designated the active application program to which all input data is directed. The advanced user interface allows a user to select among user-friendly input devices to operate any application program according to his individual preferences without change to the application program code. The advanced user interface includes alternate input modules which translate the input signals transmitted from the various input devices into input messages useable by the rest of the interface. The advanced user interface also includes interface profiles which contain mappings of the input messages against corresponding commands useable by the application programs, the integrated operating environment or other modules of the advanced user interface itself. An environment link module refers to the interface profiles and matches the input message against the corresponding command for the application program active at the time the input signal was transmitted and send the corresponding command to that application program.
The integrated operating environment allows a plurality of application programs to be running simultaneously, one of which is designated the active application program to which all input data is directed. The advanced user interface allows a user to select among user-friendly input devices to operate any application program according to his individual preferences without change to the application program code. The advanced user interface includes alternate input modules which translate the input signals transmitted from the various input devices into input messages useable by the rest of the interface. The advanced user interface also includes interface profiles which contain mappings of the input messages against corresponding commands useable by the application programs, the integrated operating environment or other modules of the advanced user interface itself. An environment link module refers to the interface profiles and matches the input message against the corresponding command for the application program active at the time the input signal was transmitted and send the corresponding command to that application program.
Description
~ X007~
ADVANCED USER INTERFACE
Background of the Invention This invention relates to improvements in entering input data into digital computers. More particularly, it relates to an advanced user interface which allows a user to select one or more input devices to input data into a computer r~nning a program originally written for a different input device.
In the past, computers were used only by scientists, mathematicians, and other high-level, sophisticated computer users. As computer technology progressed, and particularly with the advent of the personal computer, data processing has reached every level of society, and every level of user. The trend is for fewer computer users to be computer professionals or sophisticated in data processing techniques. Access to computers will increase even further in the future as computer hardware and software increase in power and efficiency.
~ owever, despite the prevalence of computers in our society, many are apprehensive about using them. The traditional text display and alphanumeric keyboard input device are not readily learned, re~uiring the memorization of the correct format of dozens, if not hundreds, of commands. Unless the user is a reasonably good typist, data entry can be inconvenient and'time consuming. The hours of experimentation which may be necessary to become proficient with a single piece of software becomes extremely frustrating for the unsophisticated user who is likely to avoid the effort altogether.
It has therefore become necessary to design what have become known in the art as "user friendly" input devices and computer programs. Such "user friendly" devices and software are designed to allow an unsophisticated user to perform desired tasks without extensive training. One of the more popular input devices is the mouse pointing device. The mouse generates signals corresponding to the direction and distance it i5 moved across a flat surface by the user. The computer responds to this information by changing the position of the cursor on the computer display to allow the user to point to any displayed object. Once the cursor is correctly positioned, the user can perform a number of functions by depressing one or more buttons on top of the mouse.
Human factor studies have shown that a device which allows the user to input data directly on the visual display screen of a computer, generally known in the art as a touch input device, achieves greatest immediacy and accuracy between man and machine. One of the first input devices for use at the display surface was the light pen.
The light pen is an optical detector in a hand held stylus, which is placed against the face of a cathode ray tube. The location of the light pen is determined by detecting the coordinates of the dot of light which is the scanning raster of the display. A second type of touch input device is a mechanical deformation membrane which .... .. . .
is placed over the display screen. The membrane is a transparent overlay which consists o~ two transparent conductor planss disposed on a flexible surface. When a selection is made, the user mechanically displaces one of the conductor planes to touch the other by a finger or stylus touch, thereby bringing the conductors into electrical contact with each other. Appropriate electronics and software translate the electrical signals generated by the finger or stylus touch to the position on the display surface. Another touch input device is a capacitive transparen~ overlay placed over the display screen, which includes transparent conductors driven by an electromagnetic signal. The input device can detect the location of a finger touch by the change in capacitance of the overlay or, alternately, a stylus is used to return the electromagnetic signals from the overlay back to the computer to determine the stylus position. Yet another touch input device uses a frame which fits around the display screen having a number of infrared or visible light transmitters and receptors arranged in parallel horizontal and vertical directions.
When the user's finger blocks the light beams, the horizontal and vertical receptors note the absence of the signals, thereby locating the position of the action desired by the user.
Many other user friendly input devices which respond to actions o~ the user such as voice recognition units or digital tablets, have been developed.
Z~07~
In addition to user friendly input devices, designers have made efforts to develop more user friendly software.
One technique which has been employed is to provide the user with a menu of choices of the particular tasks or functions which can be performed. In this way, the user is not required to commit long lists of commands to memory. The menu can be a full or partial screen display with spaces adjacent to the menu entries to which the cursor is moved by keyboard or by other cursor moving device to select a particular action. Alternatively, the user can select an action by entering an alphanumeric character associated with the menu selection on a command line.
Another recent trend is to provide some sort of integration of computer program applications. Without integration, the user must employ separate application programs for word processing, database manipulation, graphics and electronic mail functions, and so forth. It is often quite difficult to integrate the outputs of the different programs into a single desired output. One solutlon has been to write a single integrated piece of softwara which incorporates a variety of applications which is called a multiple-function program. Typically, these multiple-function programs include text, spreadsheet, and business graphing applications.
Another approach is to provide an integrated operating environment as implemented by Mlcrosoft WINDOWS*~ or IBM* Presentation Manager**. In this * Registered Trade ~Iar~
** Trade Mark ;~007~
approach, individual application programs share information and often appear on the display screen at the same time, each in its own window. By selecting the strongest individual application programs, a much more powerful environment can be tailored for the user's particular needs, in contrast to multi-function programs where the user is limited to whatever applications have been programmed into the package.
Unfortunately, while many user friendly input devices and many user friendly computer programs have been developed, these efforts have not been well integrated.
For example, there are a large number of programs which i have been written to accept keyboard cr mouse input which do not recogni~e information from a touch input device or voice recognition unit. This situation presents a severe obstacle to the introduction and widespread use of new user friendly input devices. Either a programmer must make extensive revisions to a large body of existing software, or must write a new multiple function program, or set of application programs each time a new input device is developed. Either alternative adds significantly to the costs and time of a development effort and may be beyond the capabilities of the organization developing the input device or render the effort too expensive for the expected benefits. In addition, copyrights or oth~r proprietary rights may prevent the input device developer from adapting existing software. Further, newly developed software may not be .
21~7fl~
as efficient or commercially acceptable to users as the established application programs already written for a particular purpose.
A final consideration is that the individual user is probably best suited to assessing his own needs, yet is least likely to be able or interested in undertaking extensive revisions in application programming. It would be advantageous to allow a user to select the particular input devices and application programs which would best serve his purposes and yet with only a relatively low level of expertise, allow him to assemble the components in a user friendly interface between himself and the computer. The general solution developed by the programmer may not be optimum for a particular user's needs. It would also be advantageous to assure a user that future advances in input devices and application programs can be accommodated by the interface with a minimum of revision.
Summary of the Invention It is therefore an object of the invention to allow a user to select between a plurality of input devices to input data into a computer system. ~
It i8 another object of the invention to input ' information to a computer system via a different input device than that for which the active application program was originally written without revising the application program code. ' MA9-8~-029 6 ~ , -;~07~
It is still another object of the invention to allow a relatively unsophisticated user to select among available application programs and input devices an~
construct a user interface designed particularly ko his needs.
It is yet another object of the invention to accommodate future advances in user friendly input devices and application programs in a user interface.
It is still yet another object of the invention to allow different users who operate a single computer to construct different user interfaces to accommodate individual preferences in inputting data.
These objects and others are accomplished by an advancad user interface which operates with an inte~rated operating environment capable of running a plurality of application programs and which utilizes the message and focusing functions provided by the environment. ~he advanced user interface is comprised of three major modules, an alternative input subsystem module, an interface profile module, and an environment link module.
The alternative input subsystem module provides communication between the attached user friendly input devices and the remainder of the advanced user interface as well as application programs through the integrated operating environment. The alternative input subsystem module receives the input signals generated by the input devices and translates them to input messages useable in the advanced user interface.
~, ' ~
ADVANCED USER INTERFACE
Background of the Invention This invention relates to improvements in entering input data into digital computers. More particularly, it relates to an advanced user interface which allows a user to select one or more input devices to input data into a computer r~nning a program originally written for a different input device.
In the past, computers were used only by scientists, mathematicians, and other high-level, sophisticated computer users. As computer technology progressed, and particularly with the advent of the personal computer, data processing has reached every level of society, and every level of user. The trend is for fewer computer users to be computer professionals or sophisticated in data processing techniques. Access to computers will increase even further in the future as computer hardware and software increase in power and efficiency.
~ owever, despite the prevalence of computers in our society, many are apprehensive about using them. The traditional text display and alphanumeric keyboard input device are not readily learned, re~uiring the memorization of the correct format of dozens, if not hundreds, of commands. Unless the user is a reasonably good typist, data entry can be inconvenient and'time consuming. The hours of experimentation which may be necessary to become proficient with a single piece of software becomes extremely frustrating for the unsophisticated user who is likely to avoid the effort altogether.
It has therefore become necessary to design what have become known in the art as "user friendly" input devices and computer programs. Such "user friendly" devices and software are designed to allow an unsophisticated user to perform desired tasks without extensive training. One of the more popular input devices is the mouse pointing device. The mouse generates signals corresponding to the direction and distance it i5 moved across a flat surface by the user. The computer responds to this information by changing the position of the cursor on the computer display to allow the user to point to any displayed object. Once the cursor is correctly positioned, the user can perform a number of functions by depressing one or more buttons on top of the mouse.
Human factor studies have shown that a device which allows the user to input data directly on the visual display screen of a computer, generally known in the art as a touch input device, achieves greatest immediacy and accuracy between man and machine. One of the first input devices for use at the display surface was the light pen.
The light pen is an optical detector in a hand held stylus, which is placed against the face of a cathode ray tube. The location of the light pen is determined by detecting the coordinates of the dot of light which is the scanning raster of the display. A second type of touch input device is a mechanical deformation membrane which .... .. . .
is placed over the display screen. The membrane is a transparent overlay which consists o~ two transparent conductor planss disposed on a flexible surface. When a selection is made, the user mechanically displaces one of the conductor planes to touch the other by a finger or stylus touch, thereby bringing the conductors into electrical contact with each other. Appropriate electronics and software translate the electrical signals generated by the finger or stylus touch to the position on the display surface. Another touch input device is a capacitive transparen~ overlay placed over the display screen, which includes transparent conductors driven by an electromagnetic signal. The input device can detect the location of a finger touch by the change in capacitance of the overlay or, alternately, a stylus is used to return the electromagnetic signals from the overlay back to the computer to determine the stylus position. Yet another touch input device uses a frame which fits around the display screen having a number of infrared or visible light transmitters and receptors arranged in parallel horizontal and vertical directions.
When the user's finger blocks the light beams, the horizontal and vertical receptors note the absence of the signals, thereby locating the position of the action desired by the user.
Many other user friendly input devices which respond to actions o~ the user such as voice recognition units or digital tablets, have been developed.
Z~07~
In addition to user friendly input devices, designers have made efforts to develop more user friendly software.
One technique which has been employed is to provide the user with a menu of choices of the particular tasks or functions which can be performed. In this way, the user is not required to commit long lists of commands to memory. The menu can be a full or partial screen display with spaces adjacent to the menu entries to which the cursor is moved by keyboard or by other cursor moving device to select a particular action. Alternatively, the user can select an action by entering an alphanumeric character associated with the menu selection on a command line.
Another recent trend is to provide some sort of integration of computer program applications. Without integration, the user must employ separate application programs for word processing, database manipulation, graphics and electronic mail functions, and so forth. It is often quite difficult to integrate the outputs of the different programs into a single desired output. One solutlon has been to write a single integrated piece of softwara which incorporates a variety of applications which is called a multiple-function program. Typically, these multiple-function programs include text, spreadsheet, and business graphing applications.
Another approach is to provide an integrated operating environment as implemented by Mlcrosoft WINDOWS*~ or IBM* Presentation Manager**. In this * Registered Trade ~Iar~
** Trade Mark ;~007~
approach, individual application programs share information and often appear on the display screen at the same time, each in its own window. By selecting the strongest individual application programs, a much more powerful environment can be tailored for the user's particular needs, in contrast to multi-function programs where the user is limited to whatever applications have been programmed into the package.
Unfortunately, while many user friendly input devices and many user friendly computer programs have been developed, these efforts have not been well integrated.
For example, there are a large number of programs which i have been written to accept keyboard cr mouse input which do not recogni~e information from a touch input device or voice recognition unit. This situation presents a severe obstacle to the introduction and widespread use of new user friendly input devices. Either a programmer must make extensive revisions to a large body of existing software, or must write a new multiple function program, or set of application programs each time a new input device is developed. Either alternative adds significantly to the costs and time of a development effort and may be beyond the capabilities of the organization developing the input device or render the effort too expensive for the expected benefits. In addition, copyrights or oth~r proprietary rights may prevent the input device developer from adapting existing software. Further, newly developed software may not be .
21~7fl~
as efficient or commercially acceptable to users as the established application programs already written for a particular purpose.
A final consideration is that the individual user is probably best suited to assessing his own needs, yet is least likely to be able or interested in undertaking extensive revisions in application programming. It would be advantageous to allow a user to select the particular input devices and application programs which would best serve his purposes and yet with only a relatively low level of expertise, allow him to assemble the components in a user friendly interface between himself and the computer. The general solution developed by the programmer may not be optimum for a particular user's needs. It would also be advantageous to assure a user that future advances in input devices and application programs can be accommodated by the interface with a minimum of revision.
Summary of the Invention It is therefore an object of the invention to allow a user to select between a plurality of input devices to input data into a computer system. ~
It i8 another object of the invention to input ' information to a computer system via a different input device than that for which the active application program was originally written without revising the application program code. ' MA9-8~-029 6 ~ , -;~07~
It is still another object of the invention to allow a relatively unsophisticated user to select among available application programs and input devices an~
construct a user interface designed particularly ko his needs.
It is yet another object of the invention to accommodate future advances in user friendly input devices and application programs in a user interface.
It is still yet another object of the invention to allow different users who operate a single computer to construct different user interfaces to accommodate individual preferences in inputting data.
These objects and others are accomplished by an advancad user interface which operates with an inte~rated operating environment capable of running a plurality of application programs and which utilizes the message and focusing functions provided by the environment. ~he advanced user interface is comprised of three major modules, an alternative input subsystem module, an interface profile module, and an environment link module.
The alternative input subsystem module provides communication between the attached user friendly input devices and the remainder of the advanced user interface as well as application programs through the integrated operating environment. The alternative input subsystem module receives the input signals generated by the input devices and translates them to input messages useable in the advanced user interface.
~, ' ~
- 2~)~)74~
The inter~ace profile module comprises a set of application profiles, one for each application program for which the user wishes to use an input device in a different manner from the way in which the application program was originally written. An entirely new input device may be used with the application program, or optionally, the user may change the input signals of the existing input device from those to which the application program originally responded. Each application profile contains a list of input messages mapped to corresponding commands useable by the application program. The interface profile module may also comprise a ~et of user profiles for each user who utili~es the computer system.
The user profile contains a list of input messa~es and their corresponding commands which can be common to several application programs.
The environment link module communicates with integrated operating environment and contains algorithms for the execution of the advanced user interface procedures. For example, it takes the input messages generated by the alternate input subsystem, queries the integrated operating environment as to which application program is active, matches the input messages to the corresponding commands in the appropriate application profile within the interface profile module, and initiates an action based on the set of instructions such as sending messages to the active application program via the operating environment. The advanced user interface may )7~
also include a set of utility programs such as a pop-up keyboard or an image magnifier utility.
Brief Description of the Drawings FIG. 1 is a block diagram showing the interaction of computer system software and hardware in response to actions of user using an integrated operating environment such as Presentation Manager.
FIG. 2 is an architectural block diagram of a computer system in accordance with the present invention, depicting the memory having read therein a series of application and operating system proqrams.
FIG. 3 is a block diagram showing the interaction of computer system software and hardware in response to actions of a user designed in accordance with a first embodim~nt of the present invention.
FIG. 4 is a flow diagram of the operation of the first embodiment of the present invention in response to a voice input signal.
FIG. 5 is a diagram of an application profile.
; FIG. 6 is a block diagram showing the interaction of computer system software and hardware in response to actions of a user designed in accordance with a second embodiment of the present invention.
FIG. 7 is a flow diagram of the operation of the second embodiment of the present invention in response to a voice input signal.
.
, z~
' --FIG. 8 is a flow diagram of the operation of a computer module which distinguishes between touch input meant to emulate a mouse pointing device and touch input meant to be a gesture or character.
Detailed Description of the Preferred Embodiment FIG. 1 shows the interaction of the user and the computer system utilizing application programs A, B, C and D 50-53 in a typical integrated operating environment such as Presentation Manager 56.
The user inp~ts data into the computer system via device A (keyboard) 32, device B (mouse) 34 or touch sensor 36. As the user inputs data, the selected device generates an interrupt to activate the appropriate driver.
In the case of the keyboard 32, driver A 60 is activated;
in the case of the mouse 34, driver B 62 is energized; and input to the touch sensor 36 activates the touch driver 63. The appropriate driver signals the integrated operating environment 56. The integrated operating environment 56 keeps track of which application program is active and sends all input data to that application program. Alternatively, the user can change application programs, and the integrated operating environment S6 will route the message to the new active application program.
In one preferred embodiment, application program A 50 is a spreadsheet pro~ram, application program B 51 is a word processor, application program C 52 is a drawing program, and application program D 53 is an electronic mail Xq~:)7~1 facility. Keeping track of the active program is a "focusing function", as the integrated operating environment 56 determines the "focus" o~ the user at a given time. For purposes of illustration, assume the user is inputting data via the keyboard 32 and the spreadsheet 50 is the active application program. The spreadsheet 50 then decides what action should be performed. Generally, this action will fall into one of three categories: the spreadsheet 50 will change the display 28, the spreadsheet 50 will communicate with other I/O devices such as a printer, or the spreadsheet 50 will communicate with other applications. All of the messages are handled by the integrated operating environment 56.
While the prior art integrated operating environments allow easier access to multiple application programs, and therefore, the user can select the particular application programs which best suit his needs, the user can only communicate with the application program with the input devices for which it has been written. For example, spreadsheet 50 is a program which has been written for a combination of keyboard 32 and mouse 34. Although a touch driver 63 can be developed to communicate coherently with the integrated operating environment 56, by converting input signals generated by the touch sensor 63 into mouse '~
movements and mouse clicks, the capabilities of the touch . ~
input device 36 are limited by the repertoire of the mouse device. New categories of touch co~n~ such as handwriting recognition or gestures cannot be easily '::
2~ 7~1~
accommodated by this method, especially by an application program such as spreadsheet 50 which has no understanding of what the types of touch commands mean. Other new input devices are more drastically limited. Thus, although the system designer or user would like to include new user-friendly input devices into an individual workstation, he may find it impossible or severely limiting to use existing software.
The preferred embodiment of the invention comprises a set of computer programs for controlling the interaction between a user and a computer system as shown in FIG. 2.
The invention is primarily envisioned for use with a personal computer such as the IBM PS/2*; however, the principles of this invention can be extended to other individual workstations or to much larger data processing systems. The architectural bloc~ diagram of FIG. 2 includes a central processing unit (CPU) 20 connected by means of a system bus 22 to a read-only memory (ROM) 24 and a random access memory (RA~) 26. Also included in the computer system in FIG. 2 are a display 28 by which the computer presents information to the user, and plurality of input devices including a keyboard 30, and devices A
32 and B 34, which for the purposes of illustration in the following description are a second alphanumeric keyboard 32 and a mouse 34, but which could be a specialized keyboard such as a numeric keyboard, and a remote pointing device or a variety of other input devices. The computer system ~urther comprises a touch sensor 36 or touch input * Re~istered Trade Mark X~07~
device for use at the surface of the display 28, a voice sensor 38 and an image sensor 40. The I/O 42 can be connected to communication lines or other I/O equipment.
The random access memory (RAM) 26 includes application program A 50, application program B 5:1, application program C 42, and application program D 53.
Examples of typical application programs would include word processors, spreadsheets, graphic programs, electronic mail, and data base programs. The RAM 26 also stores an operating system 54, such as DOS or OS/2*, and an integrated operating environment program 56, such as WINDOWS or Presentation Manager, ~lich allows several application programs to be running concurrently. Other software includes driver ~ 60 and driver B 62, which interpret the electrical signals generated by the second keyboard 32 and the mouse 34 respectively for the operating environment 56. The touch driver 63, the gesture recognition unit 64 and character recognition unit 65 handle input signals from the touch sensor. Input signals from the voice sensor 38 and the image sensor 40 are processed by the voice recognition unit 66 and image recognition unit 68 respectively. Although depicted in EIG. 2 as entirely software, driver A 60, driver B 62, touch driver 63, gesture recognition unit 64, voice recognition unit 66 and image recognition unit 68 can also be a combination of hardware and software. Finally, also resident in RAM 26 are the set of software modules which make up the advanced user interface 100.
* Registered rrrade ~lark .
20074~L
. . , First Embodiment FIG. 3 shows a computer system designed in accordance with the present invention. Rather than being connected to the integrated operating environment 56 via a standard device driver at least one of the plurality input devices is connected to the advanced user interface 100, via a recognition unit. In FIG. 3, the voice sensor 38 and image sensor 40 are connected to the advanced user interface (AUI) 100 via their respective recognition units, voice recognition unit 65 and image recognition unit 68. A keyboard 32, and a mouse 34, remain connected to integrated operating environment 56 via their respective drivers driver A 60 and driver B 52, slnce many application programs recognize keyboard and mouse input.
Einally, in FIG. 3, the touch sensor 36 is connected to both the integrated operating environment 56 via the standard touch driver 63 and to the advanced user interface 100 via the gesture recognition unit 64 and the -character recognition unit 65. The touch driver 63, gesture recognition unit 64 and character recognition unit 65 translate signals from the touch sensor 36 to produce gesture or character l'tokens" respectively. The series of positions of a finger or stylus on the surface of the touch sensor 36 can be recognized as handwritten alphanumeric characters by the character recognition unit 65, or as a gesture such as a circle of right hand arrow by the gesture recognition unit 64. The voice recognition MA9-88-029 14 :;
2~
unit 66 digitizes the sound received by the voice sensor 38 and performs pattern matching against previously defined sound patterns to produce voice "tokens". Image recognition unit 6~ processes in~ormation from ~he ima~e sensor 40 and outputs image "tokens". A "token" i~ an input signal which represents a series of electrical signals collected from an input device. For example, the series of points collected when a user draws a circle on the touch sensor can be defined as the "circle token".
Coupling the touch sensor 36 to integrated operating environment 56 allows the user to take advantage of application programs which are specifically written for the touch sensor 36 or other existing application programs for which only primitive mouse input signals are needed.
Yet as the touch sensor 36 is also connected to AUI
through the gesture recognition unit 64 and the character recognition unit 65, application programs which do not recognize touch input can still accept gesture or character "tokens" due to the translation capa~ilities of the advanced user interface lOO.
In other embodiments, the keyboard 32 and mouse 34 to be connected to both the AUI lOO and integrated operating environment 56 as is the touch sensor 36. In still other embodiments, all input de~ices are attached directly to the AUI 100 alone. By using a keyboard AUI
link, a user could change the normal typewritten commands for an application program to those which he finds more convenient to input or memorize. As the AUI lOO is :' ~
X0~7at~11 written as an application program, it is able to use the communication and focuslng capabilities of the integrated operating environment.
Another preferred embodiment of the present invention uses a single pointing device driver (not pictured) to combine the function of driver B 62 and touch driver 63 by accepting input from both mouse 34 and touch sensor 36 as well as any other pointing device. This embodiment is particularly adapted to operate with the touch input device described in U. S. Patent Number 4,868,332, to E.
Greanias, et al., entitlad "Combined Finger Touch and Stylus Detection System for Use on the Viewing Surface of a Visual Display Device", filed June 26, 1986. The pointing device driver arbitrates between mouse input ~rom the mouse 34 and finger and stylus input ~rom the touch input device 36 and sends appropriate input messages to the integrated operating environment 56.
Referring back to FIG. 3, the AUI 100 is further divided into several computer code modules. The environment link module 101 contains algorlthms for the execution of the procedures of the AUI 100 and provides communication capabilities to the integrated operating environment 56. The Alternative Input Subsystem (AIS~ 103 provides an interface description which allows the designers of new user friendly input devices, e.g., voice sensor 38, to seamlessly interconnect with the Advanced User Interface 100. The AIS 103 translates the input signals sent by the input devices and their respective .' ~
411.
recognition units to input massages useable by the remainder of the Advanced User Inter~ace 100. A part of each input message, such as a prefix GS! to denote a gesture, will be devoted to identify the input device ~rom which the input signal is received. In khis embodiment, the AIS 103 also time stamps each input message so that the correct application program receives the appropriate commands.
The interface profile module 104 is comprised of sets of application profiles 105 and the user pro~iles 107, which are files which list input messages produced by the AIS 103 from the input signals received by input devices 36, 38, 40, mapped to keyboard, mouse or other commands which are usable by existing application programs, e.g., mouse clicks, keystroke messages, MACROs, utility programs, etc. The Advanced User Interface utility (AUI
utility) 109 is a set of utility programs such as a pop-up keyboard or an image magnifier utility which can be used with AUI 100. As shown, the AUI utility 109 can be written as a separate application program, or set of application programs, so that it can use the message passing capabilities of the integrated operating environment 56 to communicate with other application programs.
A few examples of utilities which might be found in the advanced user interface utilities lO9 include a "pop-up" keyboard, a "pop-up" numeric keypad, an image magnifying utility and an electronic mail routing slip.
The "pop-up" keyboard and "pop-up" keypad are functions ~A9-88-029 17 74~1 which are preferably designed to operate with a touch input device at the display surface. By issuing the proper command, e.g., a circle ge~ture or touching a keyboard icon, the keyboard or keypad will "pop-up" on the screen. A user can quickly enter columns of alphanumeric information directly on the screen by touching the keys displayed on the screen, thus eliminating tha need to divert their attention to manual keyboard. The space needed ~or the keyboard or keypad is quite small, and will not completely obscure the work area of the spreadsheet, word processor, etc., which is the active application program. The image magnifying utility will magnify a rectangle of fixed size around a point at which the ,.
appropriate gesture was made. The utility allows very accurate positioning o~ a cursor in the expanded image.
After stylus liftoff, the normal size display is restored, and the selected cursor coordinates are sent to the active application program.
The flow diagram in FIG. 4 depicts the operation of the computer system in FIG. 3 in response to a voice input signal. The active application program is the spreadsheet SO, and presented to the user on the display 28 is an array of selected data concurrently ~ith a menu bar of possible functions available to the user. Rather than striking a series of keys on the keyboard, the user elects to input data into the computer system using another input device, the voice sensor 38. In response to the user vocalizing the word "G~APH", at 111 the voice sensor 38 generates MA9-~8-029 18 electrical signals and sends them to the appropriate translation device, the voice recognition unit 66. In response to these electrical signals, the ~oice recognition unit is activated at 113 and sends an appropriate input signal, a voice input "token", to the Alternate Input Subsystem 103. At 115, the AIS 103 forwards an input message to the environment link 101 which contains information on the time the input signal was received and which input device sent the signal. In response to the input message, at 117 the environment link 101 queries the integrated operating environment 56 as to which application program is currently active. After determining the active application program, spreadsheet at 119, the environment link 101 refers to the application profile 105 of the spreadsheet 50 ~or the command which corresponds to the input message "GRAPH"
which will be r~cognized by the spreadsheet 50, i.e., the menu selection --graph--. The environment link lOl then at 121 checks the user profile 107 to determine whether there is a higher priority command which would override the command from the application profile 105. The environment link 101 determines which command has higher priority at 123. The environment link 101 at 125 then sends the higher priority command, in this example, the menu-selection command ~rom the application profile l05, to the integrated operating environment 56 which routes the command to the active application program spreadsheet 50. The spreadsheet 50 at 127 takes the appropriate :, .
2~
action in response to that command, in this case, graphing the currently selected data and changing the information presented to the user by the display 28 accordingly.
Interface Profiles To understand the power and flexibility of the advanced user interface 100, the application profiles 105 and the user profiles 107 of the interface profile 104 must be explained in greater detail. Both types of profile associate lists of input messages with a list of corresponding commands which are recognized by the appropriate application program, the environment link module 101, or by the integrated operating environment 56 itself. The command can be the name of a program, an instruction recognized by the inteyrated operating environment or a MACRO. A MACRO is a file which contains an arbitrarily long string of "events" and/or utilities.
An "event", for the purpose~ for this specification, is an input message which results from a user action such as a single keystroke, a mouse button press, a mouse button release, a mouse double click, single menu selection, a single gesture or other "token", or a single accelerator key selection. An example of a utility is the pop-up keyboard. A MACRO may also treat other previously defined MACROS as commands. Accelerator keys are often a feature of the integrated operating environment 56. They speci~y a keystroke sequence which emulates a menu selection. For example, an application may specify that the DELETE key .
41~
, is the same as selecting CUT from the EDIT menu and that SHIFT/INSERT is the same as selecting PASTE from the E~IT
menu. These two examples are, in fact, widely used accelerators.
Each application program may have its own application .
interface profile 105, which contains mappings of input messages and their corresponding commands for each input device connected to the alternate input subsystem 103.
For example, one application profile might contain input messages and commands as shown in FIG. 5. The sample application profile shows input messages mapped against the corresponding command together with comments for display to the user describing the user action needed to create the input message and comments describing the action the computer system will take upon receiving that input message when the given application is active.
The same input message fr~m the AIS 103 will generally invoke a different MACR0 on different application profiles 105. For axample, a certain application profile 105 may specify that a MACR0 named "Cut/Paste" should be executed in response to a Right-Arrow gesture, while a different application profile 105 may map the Right-Arrow gesture to the keystroke sequence "Form ~01 (ENTER)". When the gesture recognition unit 64 detects a right arrow gesture from the touch sensor 36, it calls environment link 101 which determines the application program which owns the currently active window. The environment link 101 reads the corresponding : ' ~'7~
application's pro~ile 105, merges tha-t information with the appropriate user's profile 107, choosing the command with higher priority, then issues the command to the active application program. The integrated operating environment 56 or to other AUI modules as appropriate.
The user profile 107 contains commands which are common to all or several of the application programs 50-53. A separate user profile 107 can be written for each system user. The advanced user interface can determine the active user during log-on procedures.
Entries in the user profile 107 can be used to override those in the application profiles 105. This situation might occur when a particular user prefers to input data in a different manner from that provided by the application profile 105 written for a group of system users. ~he user profile lOS can also be used to provide a global default action, i.e., if a particular input message was not found by PM-Link lOl in the appropriate application profile 105. Where a user profile 107 is used for a global default action, the applicatio~ profile 105 which has a MACR0 for a given input message can be given a higher priority than the default action.
While the preferred embodiments of the present invention make a distinction between the application profiles 105 and the user profiles 107, in other embodiments, it is possible to write separate application profiles 105 for each user. Where the interface profile module 104 was organized in this manner, if an application ~74gl~
profile 105 did not exist for a particular user, the environment link 101 would refer to a default application interface profile 105. The default application profile 105 would be provided with the application program, th0 input device or by the programmer configuring the compu-ter system. As currently envisioned, an application profile 105 for a given application program would generally be provided with the application itself. Alternatively, all application profiles 105 as well as a general user profile 107 could be provided by the system programmer. For an unsophisticated user who does not wish to explore the power of the advanced user interface 100, the default profiles will usually be adequate. However, many users will wish to tailor the advanced user interface 100 to their particular desires and specification. Therefore, it will be necessary to either write or modify an existing application or user profile.
The Advanced User Interface Control Panel, one of tha Advanced User Interface Utilities 109, allows the user to tailor many of the AUI functions to create a user unique profile in an extremely efficient and user friendly manner. Although the advanced user interface 100 contains fairly sophisticated computer code, through the use o~ the Control Panel the user need not understand its workings.
Among the functions offered by the AUI Control Panel is the ability to modify or create profiles. This is -~
accomplished by invoking the interface profile createjmodify module of the AUI Control Panel by selecting ~.
21)~7~
..
"PROFILE" from the Control Panel menu. The Control Panel ~irst queries the user the name of the profile to be modified. At this point, a menu of the appropriate r~ profile is displayed to tha user, including a list of descriptions of user actions, e.g., circle gesture, "cut"
voice command, etc., against a descrip-tion of -the action taken by the computer, i.e., the MACRO, in response to the user action. If there is no application or user profile - associated with the name entered, the utility creates a new profile and displays a blank menu. The user moves the cursor through the menu to add or modi~y the information in the pro~ile, and then invokes the record command function or the record MACRO function. For example, returning to the first item in FIG. 5, a single line of the menu for the spreadsheet application profile may read:
USER ACTION COMMAND
CIRCLE gesture Invoke the pop-up keyboard on touch sensor If the user wants to change the input message from the c~rcle gesture to a right hand arrow gesture, he moves the cursor to the user action ~ield and selects "USER
ACTION" from the menu. The user is prompted to perform the action ~y which he wishes to invoke the pop-up keyboard, and then is prompted to add a written description of the action to be displayed on the menu.
Similarly, if the user wishes to change the MACRO invoXed 2(~0~4~ ~
.. ; .
.
by the circle gesture, he moves the cursor to the MACRO
~ield and enters the name of the MACRO to be executed.
The user creates a new profile in a similar fashion as an existing profile is modified. First, moving the cursor to a blank user action field, selecting 'IUSER
ACTION" from the menu, then when finished, moving the cursor to the corresponding macro field, entering a MACRO
name or other command, and so forth until the profile is complete. Thus, it becomes a relatively straightforward procedure for a relatively unsophisticated user to construct the advanced user interface profiles for the application programs he has selected to allow him to input data into a computer system in the most advantageous manner.
A MACRO can be created independently from the creation of a user or application pro~ile. In Presentation Manager 56, for example, there are three cooperating utilities which allow the user to create a MACRO, modify a MACRO and execute a M~CRO.
The Macro Creation Utility (MCU~ is a Presentation Manager program which can be set up as a tutorial, a prompter, or a totally invisible utility. It can be called from the standard program loader or via a designated gesture or other input signal. Its job is to record the user's sequence of commands for future playback. The user merely starts the MCU, performs the desired sequence of operations, then stops the MCU. The MCU then prompts the user for required information such )7~
as the name of the MACRO~ Once the macro is named, the user can control the use of that MACRO by referring to it in any of his application profiles or in his user profile.
The Macro Modify Utility (MMU) is a Presentation Manager program which allows the user to single step through a previously created macro. MMU displays an English language description of each step in the macro before it executes. At any point in the playback process, the user can modify one or more commands in the macro.
The modification may be performed by editing the contents of the English language window (for experienced macro writers) or by turning on the "record" function to overwrite previously stored commands. The MMU also allows the user to easily test, debug and modify MACROs. It also lets the user build MACROs by modifying existing MACROs which may already do most of what the user needs.
The Macro Execution Utility (MEU) is a Presentation Manager program which plays back the macro at full speed.
The MEU could be called by the environment link 101 whenever the user issues the appropriate gesture. For example, if a left arrow gesture is recognized, the environment link 101 looks in the current User's Profile to determine if that gestura is "globally" inhibited. If not, the environment link 101 then determines the name of the active application and reads the corresponding Application Profile to determine what (if any) MACRO is linked to the left arrow gesture, for this application.
The environment link 101 then calls the MEU, if necessary.
2(~4~1 -;One example of a default user profile is described with reference to a touch input device substituting for a mouse pointing device. As many programs recognize mouse ~'input, a great many application programs could be operated using this default profile.
In this example, the advanced user interface can be set in one of a plurality of modes which de-termine what mouse equivalent commands are found in the user profile as a result of stylus or finger actions. When the AUI is in mode one no mouse button messages are generated by touchdown, but does generate mouse move commands while the finger or stylus are moving in the touch panel. Upon lift-off of the stylus, a buttondown, buttonup command sequence is sent to the application program. Mode one allows accurate positioning of the cursor before action ;is taken by the active application. When the advanced user interface is in mode two, no mouse button commands are generated by touchdown, mouse move message are generated by moving the stylus, and a mouse double click is generated upon lift-off. When the AUI is in mode three, a mouse button down command is generated on contact and a mouse button up command on lift-off o~ the stylus.
Mode three is useful for dragging screen objects (icons, etc), sizing windows and marking areas in a window's client area (e.g., marking text which is to be moved).
These modes may be selected manually by touching the touch panel repeatedly at an icon which indicates which mode the AUI is in, or automatically by the applicatlon profile.
MA9-88-02~ 27 ~0074~
Second embodiment FIG. 6 shows another embodiment of the Advanced User Interface, where Presentation Manager 156 is used as the integrated operating environment, and OS/2 154 is the operating system which runs the computer system. Because of the peculiarities of OS/2 154 and Presentation Manager 156, a slightly different arrangement of the Advanced User Interface 200 is necessary. OS/2 154 has a single queue module which timestamps all input and requires that all input be queuad within this module in the order in which it was transmitted to the computer system. Therefore, all of the alternate input devices must go through OS/2 154 before being handled by the environment link module PM-Link 201. In addition, the present implementation of the single queue feature allows only keyboard and mouse input, and will not handle input from a touch input device, or other alternate input devices.
These constraints lead to the advanced user interface design shown in FIG. 6. All input devices keyboard 130, device C.132, device D 134, device E 136 and device F 138, go through OS/2 154 to be added to the single queue of OS/2 154. The keyboard 130 goes directly through a standard device driver 140 to OS/2 154. Device C 132, device D 134, device E 135 and device F 138, go throu~h device-specific ; code 141, 142, 143 and 144, respectively, to the alternative input subsystem 203, before being passed to the single queue of OS/2 154. Since the sin~le queue will ;
;~007~
only handle keyboard or mouse input, the alternative input subsystem 203 converts the input signals from device C
132, device 134, device 136 and device 138 into mouse or keyboard input messages. In addition, since OS/2 154 will not handle touch, voice or image "tokens" as input messages, all such inputs are passed through as a series of mouse mov~ points or other "events". OS/2 154 then passes the input messages to Presentation Manager 156 which keeps track of the active application program.
In FIG. 6, PM-Link 201, the environmental link module of the Advanced User Interface 200, acts as a filter between Presentation Manager 156 and the application programs, Application E 150, Application F 151, Application G 152 and Application H 153. After receiving an input message from Presentation Mana~er 156 together with information about which is the active application program, the PM-Link 201 refers back to the alternate input subsystem 203 to determine whether the "keyboard"
or "mouse" message it received is in fact a keyboard or a mouse message, or rather a touch, voice, image, or other message. The PM-Link 201 then passes the true message to the appropriate application program. The remaining modules of the advanced user interface 200 are not called unless the active application program indicates that the true input message is not understood. If the active application indicates that the true input message is not understood, PM-~ink 201 then refers to the interface profiles module 204 to determine the correct command. If .
~ 20074~
the true input message indicates that it may be a gesture, character, voice or image "token", the PM-Link 201 sends the input message to the appropriate recognition unit to identify the token, and then searches the interface profiles 204 to determine the correct command.
The flow diagram in FIG. 7 depicts the operation of the computer system in FIG. 6 in response to a voice input signal. The active application program is the spreadsheet 150, and presented to the user on the display 28 is an array of selected data concurrently with a member of possibl~ functions available to the user. Rather than using the keyboard 130, the user elects to input data using another input device, the voice sensor 136. In response to the user vocalizing the word "GRAP~", the voice sensor 136 generates electrical signals and thereby invokes the appropriate device specific code 144 is invoked at 221. The device specific code at 223 then sends a series of input signals to the AIS 203 corresponding to the word "GRAPH". The AIS 203 translates the input signals as a series of keyboard or mouse input me3sages to be added to the single ~ueue of ~S/2 154 at 225. After timestamping the stream of input messages, OS/2 passes them to Presentation Manager 1S6 which keeps track of the focus of the user and passes the input messages to the active application program, the spreadsheet 150. The PM-Link 201 intercepts the message to the active application program at 227 and refers back to the AIS 203, to determine that the true input messages ., :
~(~07~
are a set of voice input messages at 229 and sends the true input messages, the voice input, to the spreadsheet 150 at 231.
After the PM-Link 201 transmits the true voice input message, the application program will respond with its own message at 233 and one of two things may occur. If the application does not understand the type of input message sent, it returns a "R0" message to the PM-Link 201 at 233, indicating that it does not understand the input message.
The PM-Link 201, in response to an R0 message, would normally go to the interface profiles 204 to determine the corresponding command in the appropriate application profiles 205 at 241 and user profile 207 at 243. However, since the input messages indicate that they are a voice message at 237, the PM-Link 201 first sends the input messages to the voice recognition unit 215 to receive the correct voice input token for the word l'GRAPH". After receivin~ this toXen at 239, the PM-Link 201 first looks in the application profile 205 of the spreadsheet 150 at 241 to find the corresponding command to the input message which will be recognized by the spreadsheet 150, i.e., the menu selection --graph--. Then the PM-Link 201 refers to the user profile 207 of the current user at 243. The PM-Link 201 then determines which profile has the higher priority command for the spreadsheet 150 at ~43, and then sends the higher priority command to spreadsheet 150 at 247. The spreadsheet then takes the appropriate action at 249, in this case, graphing the currently selected data , ~:
MA9-88-OZ9 31 ~ ~ ~
: ' :
' ~07~
and changing the information presented to the user by the display 28.
If, on the other hand, the spreadsheet lS0 can decipher, hand or is otherwise aware of voice inpuk, an IlRl'l message is sent to the PM-~ink 201 at 233. The PM-Link 201 would then take no further action for the current message. While the spreadsheet 150 may be aware of voice input, it may also elect to ignore the voice input in certain circumstances, or periods during the application program. Alternatively, the aware application 150 may understand some voice input, but not others, and send an R0 message back to the PM-Link 201 at 233 whenever it does not understand a particular voice inpu-t message.
For those voice input messages for which it received an "R0" message, the PM-Link 201 would consult the voice recognition unit 215 and the inter~ace profiles 204 as detailed above.
In the embodiment of FIG. 6, the character recognition 211 and the gesture recognition unit 213 are part of the Advanced User Interface 200. When PM-Link 201 receives an input message which might be a character or gesture, it sends the set of points to the character recognition unit 211. If the character recognition unit 211 recognizes the set of points as an alphanumeric character, that character is identified and sent back to PM-Link 201 for further processing in the active application program. If the set of points is not recognized as a character, the set of points is sent to MAg-88-02g 32 2~
the gesture recognition unit 213 to determine whether the set of points is recognized as a gesture. If it is recognized as a gesture, PM-Link 201 sends the gesture to the interface profiles 204 to find the set of instructions to send to the active application program from the appropriate application profile 205 or user profile 207.
If it is not recognized as either a character or a gesture, the PM-Link 201 decides that the input message was probably intended as a mouse message and generates a '~mouse buttondown, mouse buttonup" message at the last point in the set of points.
As an alternative to the default user profile described above, where various modes are used to determine what mouse equivalent commands are generated by user actions on the touch panel, the present invention can include a computer module within PM-Link 201 to differentiate between touch input which emulates a mouse buttondown event mouse move event or a mouse buttonup event, and that touch input which is to be considered a gesture or character. This is done by using a time delay to differentiate between the two types of touch input. -~
The Advanced User Interface 200 allows the user to touch the screen and move to the desired position before the application is notified of the event by placing all the points received from the touch sensor in a stroke buffer which either stylus liftoff or the stylus has delayed at a particular point for the set time delay. Once the user reaches the desired position and stops moving for the set :
;~007~
time delay, the application program will receive ~he event through PM-Link 201 as a mouse event. For example, i~ the user stops moving for Z00 milliseconds, a mouse b~ttondown - mouse buttonup event at the desired position is communicated to the application program. However, if the user starts to move again before the end of the set period of time, the event is not generated until the user once again stops and pauses for the period of time delays. The time delay is typically defined in milliseconds and may be varied depending upon the area of the screen with which the user is interacting. The time delay may be specified by the user in the user profile 207 or may be altered by the application program which is utilizing the touch input.
;' If, on the other hand, the user does not stop at a particular point for the speci~ied time delay period and instead lifts the stylus off the touch screen, tha Advanced User Interface 200 selects the set of input points as a candidate for character or gesture recognition. In other words, the time delay provides a window in time in which gestures can be made. Durin~ this time, the Advanced User Intsrface 200 will only allow gestures to be made. If the user waits for the time delay period before lifting off, the points collected will not be candidates for a gesture, but a mouse event. If the user lifts off before the time delay period expires, an attempt will be made to recognize the set of input points in the character recognition unit 211 or gesture ' ~
-- 2~07~
recognition unit 213 as a gesture or character. If it is not recognized, however, the normal mouse emulation se~uence will be generated: mouse buttondown, mouse buttonup at the last input point in -the set. If it is recognized, the PM-Link 201 refers to the interface profiles 204 to send the appropriate commands to the active application program.
The recognition of a circle gesture drawn by a user on the face of the touch sensor 134 is described below with reference to FIG. 8.
A user having determined that he wishes to invoke the pop-up keyboard utility in the AUI utilities module 209, draws a circle of the face of the touch sensor 134 and lifts the stylus off be~ore the end of the set time delay.
The touch sensor 134 generates a series of interrupts to the device specific code 142 at 261 which passes a set of input signals to the AIS 203 corresponding to the set of points in the circle at 263. The AIS 203 translates the input signals to mouse input messages to be handled by the single gueue in OS/2 154 at 265. The input messages are sent to,the active application program, the spreadsheet 150, by Presentation Manager 156, but are intercepted first by the PM-Link 201 at 267. The PM-Link 201 then ~ueries the AIS 203 and determines that the "mouse" input messages are actually touch input at 269.
When PM-Link 201 discovers the first touch input point, usually a touch down in the sensor 134 it refers to the user profile 207 for the time delay period selected . ~. . ..
;~0(1741~.
by the user at 271. At 273, the PM-Link 201 collects the touch input points in its stroXe buffer until the stylus liftoff event is detected. As the PM-Link 201 collects the touch input points, each time the user pauses on the touch sensor within the stroke, initiating a "period of stability" at 275, the PM-Link 201 starts timing the period of stability to determine whether the time delay period has expired at 277. If the user mo~es before the time delay period has expired, the PM-Link 201 continues collecting the touch input points in the stroke at 273.
If, however, the period o~ stability exceeds the time delay, a mouse buttondown, mouse buttonup command is generated at 281. In this example, the user does not pause before the stylus is lifted, and therefore, the stro~e is sent to the character recognition unit 211 and gesture recognition unit 213 for processing at ~79. If the touch input stroXe had not been recognized as either a gesture or character, a mouse buttondown, mouse buttonup command would be generated at 281.
After being recognized by the gesture reco~nition unit 213 as a circle gesture, the PM-Link 201 passes the circle gesture to the spreadsheet 150. The spreadsheet returns an ''R0" message at 285, indicating that the circle gesture was not understood. Since the circle gesture was not understood at 287, the PM-Link 201 refers to the application profile 205 for the spreadsheet 150 at ~89 and finds no corresponding command for the circle gesture.
The PM-Lin~ 201 then refers to the user profile 207 at 291 , .
.' ''.
74~:~
which contains the corresponding command "invoke the pop-up keyboard". As the command from the user profile 207 has the highest priority at 293, the PM-Link 201 sends this message to the AUI utilities module 209 at 295 and the pop-up keyboard is presented to the user on the display.
While the present invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that changes in form and detail may be made without departing from the spirit and scope of the invention. For example, while gesture and character recognition has discussed in terms of a touch input device, another pointing input device such as a mouse or a graphic tablet can be utilized to send gestures or characters to the Advanced User Interface. In addition, while specific mention has been made of particular application programs, integrated operating environments and input devices, the flexibility of the Advanced User Interface can accommodate both further advances as well as the existing implementations of these programs and devices. The embodiments presented are for purposes of example and illustration only and are not to be taken to limit the scope of the invention narrower than the scope of the appended claims.
The inter~ace profile module comprises a set of application profiles, one for each application program for which the user wishes to use an input device in a different manner from the way in which the application program was originally written. An entirely new input device may be used with the application program, or optionally, the user may change the input signals of the existing input device from those to which the application program originally responded. Each application profile contains a list of input messages mapped to corresponding commands useable by the application program. The interface profile module may also comprise a ~et of user profiles for each user who utili~es the computer system.
The user profile contains a list of input messa~es and their corresponding commands which can be common to several application programs.
The environment link module communicates with integrated operating environment and contains algorithms for the execution of the advanced user interface procedures. For example, it takes the input messages generated by the alternate input subsystem, queries the integrated operating environment as to which application program is active, matches the input messages to the corresponding commands in the appropriate application profile within the interface profile module, and initiates an action based on the set of instructions such as sending messages to the active application program via the operating environment. The advanced user interface may )7~
also include a set of utility programs such as a pop-up keyboard or an image magnifier utility.
Brief Description of the Drawings FIG. 1 is a block diagram showing the interaction of computer system software and hardware in response to actions of user using an integrated operating environment such as Presentation Manager.
FIG. 2 is an architectural block diagram of a computer system in accordance with the present invention, depicting the memory having read therein a series of application and operating system proqrams.
FIG. 3 is a block diagram showing the interaction of computer system software and hardware in response to actions of a user designed in accordance with a first embodim~nt of the present invention.
FIG. 4 is a flow diagram of the operation of the first embodiment of the present invention in response to a voice input signal.
FIG. 5 is a diagram of an application profile.
; FIG. 6 is a block diagram showing the interaction of computer system software and hardware in response to actions of a user designed in accordance with a second embodiment of the present invention.
FIG. 7 is a flow diagram of the operation of the second embodiment of the present invention in response to a voice input signal.
.
, z~
' --FIG. 8 is a flow diagram of the operation of a computer module which distinguishes between touch input meant to emulate a mouse pointing device and touch input meant to be a gesture or character.
Detailed Description of the Preferred Embodiment FIG. 1 shows the interaction of the user and the computer system utilizing application programs A, B, C and D 50-53 in a typical integrated operating environment such as Presentation Manager 56.
The user inp~ts data into the computer system via device A (keyboard) 32, device B (mouse) 34 or touch sensor 36. As the user inputs data, the selected device generates an interrupt to activate the appropriate driver.
In the case of the keyboard 32, driver A 60 is activated;
in the case of the mouse 34, driver B 62 is energized; and input to the touch sensor 36 activates the touch driver 63. The appropriate driver signals the integrated operating environment 56. The integrated operating environment 56 keeps track of which application program is active and sends all input data to that application program. Alternatively, the user can change application programs, and the integrated operating environment S6 will route the message to the new active application program.
In one preferred embodiment, application program A 50 is a spreadsheet pro~ram, application program B 51 is a word processor, application program C 52 is a drawing program, and application program D 53 is an electronic mail Xq~:)7~1 facility. Keeping track of the active program is a "focusing function", as the integrated operating environment 56 determines the "focus" o~ the user at a given time. For purposes of illustration, assume the user is inputting data via the keyboard 32 and the spreadsheet 50 is the active application program. The spreadsheet 50 then decides what action should be performed. Generally, this action will fall into one of three categories: the spreadsheet 50 will change the display 28, the spreadsheet 50 will communicate with other I/O devices such as a printer, or the spreadsheet 50 will communicate with other applications. All of the messages are handled by the integrated operating environment 56.
While the prior art integrated operating environments allow easier access to multiple application programs, and therefore, the user can select the particular application programs which best suit his needs, the user can only communicate with the application program with the input devices for which it has been written. For example, spreadsheet 50 is a program which has been written for a combination of keyboard 32 and mouse 34. Although a touch driver 63 can be developed to communicate coherently with the integrated operating environment 56, by converting input signals generated by the touch sensor 63 into mouse '~
movements and mouse clicks, the capabilities of the touch . ~
input device 36 are limited by the repertoire of the mouse device. New categories of touch co~n~ such as handwriting recognition or gestures cannot be easily '::
2~ 7~1~
accommodated by this method, especially by an application program such as spreadsheet 50 which has no understanding of what the types of touch commands mean. Other new input devices are more drastically limited. Thus, although the system designer or user would like to include new user-friendly input devices into an individual workstation, he may find it impossible or severely limiting to use existing software.
The preferred embodiment of the invention comprises a set of computer programs for controlling the interaction between a user and a computer system as shown in FIG. 2.
The invention is primarily envisioned for use with a personal computer such as the IBM PS/2*; however, the principles of this invention can be extended to other individual workstations or to much larger data processing systems. The architectural bloc~ diagram of FIG. 2 includes a central processing unit (CPU) 20 connected by means of a system bus 22 to a read-only memory (ROM) 24 and a random access memory (RA~) 26. Also included in the computer system in FIG. 2 are a display 28 by which the computer presents information to the user, and plurality of input devices including a keyboard 30, and devices A
32 and B 34, which for the purposes of illustration in the following description are a second alphanumeric keyboard 32 and a mouse 34, but which could be a specialized keyboard such as a numeric keyboard, and a remote pointing device or a variety of other input devices. The computer system ~urther comprises a touch sensor 36 or touch input * Re~istered Trade Mark X~07~
device for use at the surface of the display 28, a voice sensor 38 and an image sensor 40. The I/O 42 can be connected to communication lines or other I/O equipment.
The random access memory (RAM) 26 includes application program A 50, application program B 5:1, application program C 42, and application program D 53.
Examples of typical application programs would include word processors, spreadsheets, graphic programs, electronic mail, and data base programs. The RAM 26 also stores an operating system 54, such as DOS or OS/2*, and an integrated operating environment program 56, such as WINDOWS or Presentation Manager, ~lich allows several application programs to be running concurrently. Other software includes driver ~ 60 and driver B 62, which interpret the electrical signals generated by the second keyboard 32 and the mouse 34 respectively for the operating environment 56. The touch driver 63, the gesture recognition unit 64 and character recognition unit 65 handle input signals from the touch sensor. Input signals from the voice sensor 38 and the image sensor 40 are processed by the voice recognition unit 66 and image recognition unit 68 respectively. Although depicted in EIG. 2 as entirely software, driver A 60, driver B 62, touch driver 63, gesture recognition unit 64, voice recognition unit 66 and image recognition unit 68 can also be a combination of hardware and software. Finally, also resident in RAM 26 are the set of software modules which make up the advanced user interface 100.
* Registered rrrade ~lark .
20074~L
. . , First Embodiment FIG. 3 shows a computer system designed in accordance with the present invention. Rather than being connected to the integrated operating environment 56 via a standard device driver at least one of the plurality input devices is connected to the advanced user interface 100, via a recognition unit. In FIG. 3, the voice sensor 38 and image sensor 40 are connected to the advanced user interface (AUI) 100 via their respective recognition units, voice recognition unit 65 and image recognition unit 68. A keyboard 32, and a mouse 34, remain connected to integrated operating environment 56 via their respective drivers driver A 60 and driver B 52, slnce many application programs recognize keyboard and mouse input.
Einally, in FIG. 3, the touch sensor 36 is connected to both the integrated operating environment 56 via the standard touch driver 63 and to the advanced user interface 100 via the gesture recognition unit 64 and the -character recognition unit 65. The touch driver 63, gesture recognition unit 64 and character recognition unit 65 translate signals from the touch sensor 36 to produce gesture or character l'tokens" respectively. The series of positions of a finger or stylus on the surface of the touch sensor 36 can be recognized as handwritten alphanumeric characters by the character recognition unit 65, or as a gesture such as a circle of right hand arrow by the gesture recognition unit 64. The voice recognition MA9-88-029 14 :;
2~
unit 66 digitizes the sound received by the voice sensor 38 and performs pattern matching against previously defined sound patterns to produce voice "tokens". Image recognition unit 6~ processes in~ormation from ~he ima~e sensor 40 and outputs image "tokens". A "token" i~ an input signal which represents a series of electrical signals collected from an input device. For example, the series of points collected when a user draws a circle on the touch sensor can be defined as the "circle token".
Coupling the touch sensor 36 to integrated operating environment 56 allows the user to take advantage of application programs which are specifically written for the touch sensor 36 or other existing application programs for which only primitive mouse input signals are needed.
Yet as the touch sensor 36 is also connected to AUI
through the gesture recognition unit 64 and the character recognition unit 65, application programs which do not recognize touch input can still accept gesture or character "tokens" due to the translation capa~ilities of the advanced user interface lOO.
In other embodiments, the keyboard 32 and mouse 34 to be connected to both the AUI lOO and integrated operating environment 56 as is the touch sensor 36. In still other embodiments, all input de~ices are attached directly to the AUI 100 alone. By using a keyboard AUI
link, a user could change the normal typewritten commands for an application program to those which he finds more convenient to input or memorize. As the AUI lOO is :' ~
X0~7at~11 written as an application program, it is able to use the communication and focuslng capabilities of the integrated operating environment.
Another preferred embodiment of the present invention uses a single pointing device driver (not pictured) to combine the function of driver B 62 and touch driver 63 by accepting input from both mouse 34 and touch sensor 36 as well as any other pointing device. This embodiment is particularly adapted to operate with the touch input device described in U. S. Patent Number 4,868,332, to E.
Greanias, et al., entitlad "Combined Finger Touch and Stylus Detection System for Use on the Viewing Surface of a Visual Display Device", filed June 26, 1986. The pointing device driver arbitrates between mouse input ~rom the mouse 34 and finger and stylus input ~rom the touch input device 36 and sends appropriate input messages to the integrated operating environment 56.
Referring back to FIG. 3, the AUI 100 is further divided into several computer code modules. The environment link module 101 contains algorlthms for the execution of the procedures of the AUI 100 and provides communication capabilities to the integrated operating environment 56. The Alternative Input Subsystem (AIS~ 103 provides an interface description which allows the designers of new user friendly input devices, e.g., voice sensor 38, to seamlessly interconnect with the Advanced User Interface 100. The AIS 103 translates the input signals sent by the input devices and their respective .' ~
411.
recognition units to input massages useable by the remainder of the Advanced User Inter~ace 100. A part of each input message, such as a prefix GS! to denote a gesture, will be devoted to identify the input device ~rom which the input signal is received. In khis embodiment, the AIS 103 also time stamps each input message so that the correct application program receives the appropriate commands.
The interface profile module 104 is comprised of sets of application profiles 105 and the user pro~iles 107, which are files which list input messages produced by the AIS 103 from the input signals received by input devices 36, 38, 40, mapped to keyboard, mouse or other commands which are usable by existing application programs, e.g., mouse clicks, keystroke messages, MACROs, utility programs, etc. The Advanced User Interface utility (AUI
utility) 109 is a set of utility programs such as a pop-up keyboard or an image magnifier utility which can be used with AUI 100. As shown, the AUI utility 109 can be written as a separate application program, or set of application programs, so that it can use the message passing capabilities of the integrated operating environment 56 to communicate with other application programs.
A few examples of utilities which might be found in the advanced user interface utilities lO9 include a "pop-up" keyboard, a "pop-up" numeric keypad, an image magnifying utility and an electronic mail routing slip.
The "pop-up" keyboard and "pop-up" keypad are functions ~A9-88-029 17 74~1 which are preferably designed to operate with a touch input device at the display surface. By issuing the proper command, e.g., a circle ge~ture or touching a keyboard icon, the keyboard or keypad will "pop-up" on the screen. A user can quickly enter columns of alphanumeric information directly on the screen by touching the keys displayed on the screen, thus eliminating tha need to divert their attention to manual keyboard. The space needed ~or the keyboard or keypad is quite small, and will not completely obscure the work area of the spreadsheet, word processor, etc., which is the active application program. The image magnifying utility will magnify a rectangle of fixed size around a point at which the ,.
appropriate gesture was made. The utility allows very accurate positioning o~ a cursor in the expanded image.
After stylus liftoff, the normal size display is restored, and the selected cursor coordinates are sent to the active application program.
The flow diagram in FIG. 4 depicts the operation of the computer system in FIG. 3 in response to a voice input signal. The active application program is the spreadsheet SO, and presented to the user on the display 28 is an array of selected data concurrently ~ith a menu bar of possible functions available to the user. Rather than striking a series of keys on the keyboard, the user elects to input data into the computer system using another input device, the voice sensor 38. In response to the user vocalizing the word "G~APH", at 111 the voice sensor 38 generates MA9-~8-029 18 electrical signals and sends them to the appropriate translation device, the voice recognition unit 66. In response to these electrical signals, the ~oice recognition unit is activated at 113 and sends an appropriate input signal, a voice input "token", to the Alternate Input Subsystem 103. At 115, the AIS 103 forwards an input message to the environment link 101 which contains information on the time the input signal was received and which input device sent the signal. In response to the input message, at 117 the environment link 101 queries the integrated operating environment 56 as to which application program is currently active. After determining the active application program, spreadsheet at 119, the environment link 101 refers to the application profile 105 of the spreadsheet 50 ~or the command which corresponds to the input message "GRAPH"
which will be r~cognized by the spreadsheet 50, i.e., the menu selection --graph--. The environment link lOl then at 121 checks the user profile 107 to determine whether there is a higher priority command which would override the command from the application profile 105. The environment link 101 determines which command has higher priority at 123. The environment link 101 at 125 then sends the higher priority command, in this example, the menu-selection command ~rom the application profile l05, to the integrated operating environment 56 which routes the command to the active application program spreadsheet 50. The spreadsheet 50 at 127 takes the appropriate :, .
2~
action in response to that command, in this case, graphing the currently selected data and changing the information presented to the user by the display 28 accordingly.
Interface Profiles To understand the power and flexibility of the advanced user interface 100, the application profiles 105 and the user profiles 107 of the interface profile 104 must be explained in greater detail. Both types of profile associate lists of input messages with a list of corresponding commands which are recognized by the appropriate application program, the environment link module 101, or by the integrated operating environment 56 itself. The command can be the name of a program, an instruction recognized by the inteyrated operating environment or a MACRO. A MACRO is a file which contains an arbitrarily long string of "events" and/or utilities.
An "event", for the purpose~ for this specification, is an input message which results from a user action such as a single keystroke, a mouse button press, a mouse button release, a mouse double click, single menu selection, a single gesture or other "token", or a single accelerator key selection. An example of a utility is the pop-up keyboard. A MACRO may also treat other previously defined MACROS as commands. Accelerator keys are often a feature of the integrated operating environment 56. They speci~y a keystroke sequence which emulates a menu selection. For example, an application may specify that the DELETE key .
41~
, is the same as selecting CUT from the EDIT menu and that SHIFT/INSERT is the same as selecting PASTE from the E~IT
menu. These two examples are, in fact, widely used accelerators.
Each application program may have its own application .
interface profile 105, which contains mappings of input messages and their corresponding commands for each input device connected to the alternate input subsystem 103.
For example, one application profile might contain input messages and commands as shown in FIG. 5. The sample application profile shows input messages mapped against the corresponding command together with comments for display to the user describing the user action needed to create the input message and comments describing the action the computer system will take upon receiving that input message when the given application is active.
The same input message fr~m the AIS 103 will generally invoke a different MACR0 on different application profiles 105. For axample, a certain application profile 105 may specify that a MACR0 named "Cut/Paste" should be executed in response to a Right-Arrow gesture, while a different application profile 105 may map the Right-Arrow gesture to the keystroke sequence "Form ~01 (ENTER)". When the gesture recognition unit 64 detects a right arrow gesture from the touch sensor 36, it calls environment link 101 which determines the application program which owns the currently active window. The environment link 101 reads the corresponding : ' ~'7~
application's pro~ile 105, merges tha-t information with the appropriate user's profile 107, choosing the command with higher priority, then issues the command to the active application program. The integrated operating environment 56 or to other AUI modules as appropriate.
The user profile 107 contains commands which are common to all or several of the application programs 50-53. A separate user profile 107 can be written for each system user. The advanced user interface can determine the active user during log-on procedures.
Entries in the user profile 107 can be used to override those in the application profiles 105. This situation might occur when a particular user prefers to input data in a different manner from that provided by the application profile 105 written for a group of system users. ~he user profile lOS can also be used to provide a global default action, i.e., if a particular input message was not found by PM-Link lOl in the appropriate application profile 105. Where a user profile 107 is used for a global default action, the applicatio~ profile 105 which has a MACR0 for a given input message can be given a higher priority than the default action.
While the preferred embodiments of the present invention make a distinction between the application profiles 105 and the user profiles 107, in other embodiments, it is possible to write separate application profiles 105 for each user. Where the interface profile module 104 was organized in this manner, if an application ~74gl~
profile 105 did not exist for a particular user, the environment link 101 would refer to a default application interface profile 105. The default application profile 105 would be provided with the application program, th0 input device or by the programmer configuring the compu-ter system. As currently envisioned, an application profile 105 for a given application program would generally be provided with the application itself. Alternatively, all application profiles 105 as well as a general user profile 107 could be provided by the system programmer. For an unsophisticated user who does not wish to explore the power of the advanced user interface 100, the default profiles will usually be adequate. However, many users will wish to tailor the advanced user interface 100 to their particular desires and specification. Therefore, it will be necessary to either write or modify an existing application or user profile.
The Advanced User Interface Control Panel, one of tha Advanced User Interface Utilities 109, allows the user to tailor many of the AUI functions to create a user unique profile in an extremely efficient and user friendly manner. Although the advanced user interface 100 contains fairly sophisticated computer code, through the use o~ the Control Panel the user need not understand its workings.
Among the functions offered by the AUI Control Panel is the ability to modify or create profiles. This is -~
accomplished by invoking the interface profile createjmodify module of the AUI Control Panel by selecting ~.
21)~7~
..
"PROFILE" from the Control Panel menu. The Control Panel ~irst queries the user the name of the profile to be modified. At this point, a menu of the appropriate r~ profile is displayed to tha user, including a list of descriptions of user actions, e.g., circle gesture, "cut"
voice command, etc., against a descrip-tion of -the action taken by the computer, i.e., the MACRO, in response to the user action. If there is no application or user profile - associated with the name entered, the utility creates a new profile and displays a blank menu. The user moves the cursor through the menu to add or modi~y the information in the pro~ile, and then invokes the record command function or the record MACRO function. For example, returning to the first item in FIG. 5, a single line of the menu for the spreadsheet application profile may read:
USER ACTION COMMAND
CIRCLE gesture Invoke the pop-up keyboard on touch sensor If the user wants to change the input message from the c~rcle gesture to a right hand arrow gesture, he moves the cursor to the user action ~ield and selects "USER
ACTION" from the menu. The user is prompted to perform the action ~y which he wishes to invoke the pop-up keyboard, and then is prompted to add a written description of the action to be displayed on the menu.
Similarly, if the user wishes to change the MACRO invoXed 2(~0~4~ ~
.. ; .
.
by the circle gesture, he moves the cursor to the MACRO
~ield and enters the name of the MACRO to be executed.
The user creates a new profile in a similar fashion as an existing profile is modified. First, moving the cursor to a blank user action field, selecting 'IUSER
ACTION" from the menu, then when finished, moving the cursor to the corresponding macro field, entering a MACRO
name or other command, and so forth until the profile is complete. Thus, it becomes a relatively straightforward procedure for a relatively unsophisticated user to construct the advanced user interface profiles for the application programs he has selected to allow him to input data into a computer system in the most advantageous manner.
A MACRO can be created independently from the creation of a user or application pro~ile. In Presentation Manager 56, for example, there are three cooperating utilities which allow the user to create a MACRO, modify a MACRO and execute a M~CRO.
The Macro Creation Utility (MCU~ is a Presentation Manager program which can be set up as a tutorial, a prompter, or a totally invisible utility. It can be called from the standard program loader or via a designated gesture or other input signal. Its job is to record the user's sequence of commands for future playback. The user merely starts the MCU, performs the desired sequence of operations, then stops the MCU. The MCU then prompts the user for required information such )7~
as the name of the MACRO~ Once the macro is named, the user can control the use of that MACRO by referring to it in any of his application profiles or in his user profile.
The Macro Modify Utility (MMU) is a Presentation Manager program which allows the user to single step through a previously created macro. MMU displays an English language description of each step in the macro before it executes. At any point in the playback process, the user can modify one or more commands in the macro.
The modification may be performed by editing the contents of the English language window (for experienced macro writers) or by turning on the "record" function to overwrite previously stored commands. The MMU also allows the user to easily test, debug and modify MACROs. It also lets the user build MACROs by modifying existing MACROs which may already do most of what the user needs.
The Macro Execution Utility (MEU) is a Presentation Manager program which plays back the macro at full speed.
The MEU could be called by the environment link 101 whenever the user issues the appropriate gesture. For example, if a left arrow gesture is recognized, the environment link 101 looks in the current User's Profile to determine if that gestura is "globally" inhibited. If not, the environment link 101 then determines the name of the active application and reads the corresponding Application Profile to determine what (if any) MACRO is linked to the left arrow gesture, for this application.
The environment link 101 then calls the MEU, if necessary.
2(~4~1 -;One example of a default user profile is described with reference to a touch input device substituting for a mouse pointing device. As many programs recognize mouse ~'input, a great many application programs could be operated using this default profile.
In this example, the advanced user interface can be set in one of a plurality of modes which de-termine what mouse equivalent commands are found in the user profile as a result of stylus or finger actions. When the AUI is in mode one no mouse button messages are generated by touchdown, but does generate mouse move commands while the finger or stylus are moving in the touch panel. Upon lift-off of the stylus, a buttondown, buttonup command sequence is sent to the application program. Mode one allows accurate positioning of the cursor before action ;is taken by the active application. When the advanced user interface is in mode two, no mouse button commands are generated by touchdown, mouse move message are generated by moving the stylus, and a mouse double click is generated upon lift-off. When the AUI is in mode three, a mouse button down command is generated on contact and a mouse button up command on lift-off o~ the stylus.
Mode three is useful for dragging screen objects (icons, etc), sizing windows and marking areas in a window's client area (e.g., marking text which is to be moved).
These modes may be selected manually by touching the touch panel repeatedly at an icon which indicates which mode the AUI is in, or automatically by the applicatlon profile.
MA9-88-02~ 27 ~0074~
Second embodiment FIG. 6 shows another embodiment of the Advanced User Interface, where Presentation Manager 156 is used as the integrated operating environment, and OS/2 154 is the operating system which runs the computer system. Because of the peculiarities of OS/2 154 and Presentation Manager 156, a slightly different arrangement of the Advanced User Interface 200 is necessary. OS/2 154 has a single queue module which timestamps all input and requires that all input be queuad within this module in the order in which it was transmitted to the computer system. Therefore, all of the alternate input devices must go through OS/2 154 before being handled by the environment link module PM-Link 201. In addition, the present implementation of the single queue feature allows only keyboard and mouse input, and will not handle input from a touch input device, or other alternate input devices.
These constraints lead to the advanced user interface design shown in FIG. 6. All input devices keyboard 130, device C.132, device D 134, device E 136 and device F 138, go through OS/2 154 to be added to the single queue of OS/2 154. The keyboard 130 goes directly through a standard device driver 140 to OS/2 154. Device C 132, device D 134, device E 135 and device F 138, go throu~h device-specific ; code 141, 142, 143 and 144, respectively, to the alternative input subsystem 203, before being passed to the single queue of OS/2 154. Since the sin~le queue will ;
;~007~
only handle keyboard or mouse input, the alternative input subsystem 203 converts the input signals from device C
132, device 134, device 136 and device 138 into mouse or keyboard input messages. In addition, since OS/2 154 will not handle touch, voice or image "tokens" as input messages, all such inputs are passed through as a series of mouse mov~ points or other "events". OS/2 154 then passes the input messages to Presentation Manager 156 which keeps track of the active application program.
In FIG. 6, PM-Link 201, the environmental link module of the Advanced User Interface 200, acts as a filter between Presentation Manager 156 and the application programs, Application E 150, Application F 151, Application G 152 and Application H 153. After receiving an input message from Presentation Mana~er 156 together with information about which is the active application program, the PM-Link 201 refers back to the alternate input subsystem 203 to determine whether the "keyboard"
or "mouse" message it received is in fact a keyboard or a mouse message, or rather a touch, voice, image, or other message. The PM-Link 201 then passes the true message to the appropriate application program. The remaining modules of the advanced user interface 200 are not called unless the active application program indicates that the true input message is not understood. If the active application indicates that the true input message is not understood, PM-~ink 201 then refers to the interface profiles module 204 to determine the correct command. If .
~ 20074~
the true input message indicates that it may be a gesture, character, voice or image "token", the PM-Link 201 sends the input message to the appropriate recognition unit to identify the token, and then searches the interface profiles 204 to determine the correct command.
The flow diagram in FIG. 7 depicts the operation of the computer system in FIG. 6 in response to a voice input signal. The active application program is the spreadsheet 150, and presented to the user on the display 28 is an array of selected data concurrently with a member of possibl~ functions available to the user. Rather than using the keyboard 130, the user elects to input data using another input device, the voice sensor 136. In response to the user vocalizing the word "GRAP~", the voice sensor 136 generates electrical signals and thereby invokes the appropriate device specific code 144 is invoked at 221. The device specific code at 223 then sends a series of input signals to the AIS 203 corresponding to the word "GRAPH". The AIS 203 translates the input signals as a series of keyboard or mouse input me3sages to be added to the single ~ueue of ~S/2 154 at 225. After timestamping the stream of input messages, OS/2 passes them to Presentation Manager 1S6 which keeps track of the focus of the user and passes the input messages to the active application program, the spreadsheet 150. The PM-Link 201 intercepts the message to the active application program at 227 and refers back to the AIS 203, to determine that the true input messages ., :
~(~07~
are a set of voice input messages at 229 and sends the true input messages, the voice input, to the spreadsheet 150 at 231.
After the PM-Link 201 transmits the true voice input message, the application program will respond with its own message at 233 and one of two things may occur. If the application does not understand the type of input message sent, it returns a "R0" message to the PM-Link 201 at 233, indicating that it does not understand the input message.
The PM-Link 201, in response to an R0 message, would normally go to the interface profiles 204 to determine the corresponding command in the appropriate application profiles 205 at 241 and user profile 207 at 243. However, since the input messages indicate that they are a voice message at 237, the PM-Link 201 first sends the input messages to the voice recognition unit 215 to receive the correct voice input token for the word l'GRAPH". After receivin~ this toXen at 239, the PM-Link 201 first looks in the application profile 205 of the spreadsheet 150 at 241 to find the corresponding command to the input message which will be recognized by the spreadsheet 150, i.e., the menu selection --graph--. Then the PM-Link 201 refers to the user profile 207 of the current user at 243. The PM-Link 201 then determines which profile has the higher priority command for the spreadsheet 150 at ~43, and then sends the higher priority command to spreadsheet 150 at 247. The spreadsheet then takes the appropriate action at 249, in this case, graphing the currently selected data , ~:
MA9-88-OZ9 31 ~ ~ ~
: ' :
' ~07~
and changing the information presented to the user by the display 28.
If, on the other hand, the spreadsheet lS0 can decipher, hand or is otherwise aware of voice inpuk, an IlRl'l message is sent to the PM-~ink 201 at 233. The PM-Link 201 would then take no further action for the current message. While the spreadsheet 150 may be aware of voice input, it may also elect to ignore the voice input in certain circumstances, or periods during the application program. Alternatively, the aware application 150 may understand some voice input, but not others, and send an R0 message back to the PM-Link 201 at 233 whenever it does not understand a particular voice inpu-t message.
For those voice input messages for which it received an "R0" message, the PM-Link 201 would consult the voice recognition unit 215 and the inter~ace profiles 204 as detailed above.
In the embodiment of FIG. 6, the character recognition 211 and the gesture recognition unit 213 are part of the Advanced User Interface 200. When PM-Link 201 receives an input message which might be a character or gesture, it sends the set of points to the character recognition unit 211. If the character recognition unit 211 recognizes the set of points as an alphanumeric character, that character is identified and sent back to PM-Link 201 for further processing in the active application program. If the set of points is not recognized as a character, the set of points is sent to MAg-88-02g 32 2~
the gesture recognition unit 213 to determine whether the set of points is recognized as a gesture. If it is recognized as a gesture, PM-Link 201 sends the gesture to the interface profiles 204 to find the set of instructions to send to the active application program from the appropriate application profile 205 or user profile 207.
If it is not recognized as either a character or a gesture, the PM-Link 201 decides that the input message was probably intended as a mouse message and generates a '~mouse buttondown, mouse buttonup" message at the last point in the set of points.
As an alternative to the default user profile described above, where various modes are used to determine what mouse equivalent commands are generated by user actions on the touch panel, the present invention can include a computer module within PM-Link 201 to differentiate between touch input which emulates a mouse buttondown event mouse move event or a mouse buttonup event, and that touch input which is to be considered a gesture or character. This is done by using a time delay to differentiate between the two types of touch input. -~
The Advanced User Interface 200 allows the user to touch the screen and move to the desired position before the application is notified of the event by placing all the points received from the touch sensor in a stroke buffer which either stylus liftoff or the stylus has delayed at a particular point for the set time delay. Once the user reaches the desired position and stops moving for the set :
;~007~
time delay, the application program will receive ~he event through PM-Link 201 as a mouse event. For example, i~ the user stops moving for Z00 milliseconds, a mouse b~ttondown - mouse buttonup event at the desired position is communicated to the application program. However, if the user starts to move again before the end of the set period of time, the event is not generated until the user once again stops and pauses for the period of time delays. The time delay is typically defined in milliseconds and may be varied depending upon the area of the screen with which the user is interacting. The time delay may be specified by the user in the user profile 207 or may be altered by the application program which is utilizing the touch input.
;' If, on the other hand, the user does not stop at a particular point for the speci~ied time delay period and instead lifts the stylus off the touch screen, tha Advanced User Interface 200 selects the set of input points as a candidate for character or gesture recognition. In other words, the time delay provides a window in time in which gestures can be made. Durin~ this time, the Advanced User Intsrface 200 will only allow gestures to be made. If the user waits for the time delay period before lifting off, the points collected will not be candidates for a gesture, but a mouse event. If the user lifts off before the time delay period expires, an attempt will be made to recognize the set of input points in the character recognition unit 211 or gesture ' ~
-- 2~07~
recognition unit 213 as a gesture or character. If it is not recognized, however, the normal mouse emulation se~uence will be generated: mouse buttondown, mouse buttonup at the last input point in -the set. If it is recognized, the PM-Link 201 refers to the interface profiles 204 to send the appropriate commands to the active application program.
The recognition of a circle gesture drawn by a user on the face of the touch sensor 134 is described below with reference to FIG. 8.
A user having determined that he wishes to invoke the pop-up keyboard utility in the AUI utilities module 209, draws a circle of the face of the touch sensor 134 and lifts the stylus off be~ore the end of the set time delay.
The touch sensor 134 generates a series of interrupts to the device specific code 142 at 261 which passes a set of input signals to the AIS 203 corresponding to the set of points in the circle at 263. The AIS 203 translates the input signals to mouse input messages to be handled by the single gueue in OS/2 154 at 265. The input messages are sent to,the active application program, the spreadsheet 150, by Presentation Manager 156, but are intercepted first by the PM-Link 201 at 267. The PM-Link 201 then ~ueries the AIS 203 and determines that the "mouse" input messages are actually touch input at 269.
When PM-Link 201 discovers the first touch input point, usually a touch down in the sensor 134 it refers to the user profile 207 for the time delay period selected . ~. . ..
;~0(1741~.
by the user at 271. At 273, the PM-Link 201 collects the touch input points in its stroXe buffer until the stylus liftoff event is detected. As the PM-Link 201 collects the touch input points, each time the user pauses on the touch sensor within the stroke, initiating a "period of stability" at 275, the PM-Link 201 starts timing the period of stability to determine whether the time delay period has expired at 277. If the user mo~es before the time delay period has expired, the PM-Link 201 continues collecting the touch input points in the stroke at 273.
If, however, the period o~ stability exceeds the time delay, a mouse buttondown, mouse buttonup command is generated at 281. In this example, the user does not pause before the stylus is lifted, and therefore, the stro~e is sent to the character recognition unit 211 and gesture recognition unit 213 for processing at ~79. If the touch input stroXe had not been recognized as either a gesture or character, a mouse buttondown, mouse buttonup command would be generated at 281.
After being recognized by the gesture reco~nition unit 213 as a circle gesture, the PM-Link 201 passes the circle gesture to the spreadsheet 150. The spreadsheet returns an ''R0" message at 285, indicating that the circle gesture was not understood. Since the circle gesture was not understood at 287, the PM-Link 201 refers to the application profile 205 for the spreadsheet 150 at ~89 and finds no corresponding command for the circle gesture.
The PM-Lin~ 201 then refers to the user profile 207 at 291 , .
.' ''.
74~:~
which contains the corresponding command "invoke the pop-up keyboard". As the command from the user profile 207 has the highest priority at 293, the PM-Link 201 sends this message to the AUI utilities module 209 at 295 and the pop-up keyboard is presented to the user on the display.
While the present invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that changes in form and detail may be made without departing from the spirit and scope of the invention. For example, while gesture and character recognition has discussed in terms of a touch input device, another pointing input device such as a mouse or a graphic tablet can be utilized to send gestures or characters to the Advanced User Interface. In addition, while specific mention has been made of particular application programs, integrated operating environments and input devices, the flexibility of the Advanced User Interface can accommodate both further advances as well as the existing implementations of these programs and devices. The embodiments presented are for purposes of example and illustration only and are not to be taken to limit the scope of the invention narrower than the scope of the appended claims.
Claims (39)
1. An advanced user interface, for use with an integrated operating environment and an operating system capable of running a plurality of application programs simultaneously on a computer system, only one of which is active at a given time, the computer system having a central processor, a random access memory, a display and at least one input device which transmits input signals to the advanced user interface in response to actions performed by a user comprising:
interface profiles which contain mappings of input messages representative of the input signals against corresponding commands usable by at least one of the plurality of application programs;
and an environment link module coupled to the interface profiles, the integrated operating environment and the operating system, the environment link module receiving the input messages, determining which of the plurality of application programs was active at the time the input signals were transmitted by the input device, matching the input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device, and sending the corresponding commands to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the active application program.
interface profiles which contain mappings of input messages representative of the input signals against corresponding commands usable by at least one of the plurality of application programs;
and an environment link module coupled to the interface profiles, the integrated operating environment and the operating system, the environment link module receiving the input messages, determining which of the plurality of application programs was active at the time the input signals were transmitted by the input device, matching the input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device, and sending the corresponding commands to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the active application program.
2. The advanced user interface as recited in claim 1 which further comprises an alternative input subsystem coupled to the input device and the environment link which translates the input signals to the input messages.
3. The advanced user interface as recited in claim 1, wherein the interface profiles comprise a plurality of application programs each of which corresponds to one of the plurality of application programs, the application profiles containing mappings of input messages against corresponding commands at least some of which are usable by its corresponding application program.
4. The advanced user interface as recited in claim 3, wherein the interface profiles further comprise a user profile which contains mappings of input messages against corresponding commands, the user profile usable with more than one of the plurality of application programs.
5. The advanced user interface as recited in claim 4, wherein at least one of the corresponding instructions contained in the user profile overrides at least one of the corresponding commands contained in the plurality of application profiles.
6. The advanced user interface as recited in claim 4, wherein at least one of the corresponding commands contained in the user profile is a default instruction which is sent only when the application profile of the active application program has no corresponding commands to the input message.
7. The advanced user interface as recited in claim 4, wherein the interface profiles further comprise a user profile for each user authorized to utilize the computer system.
8. The advanced user interface as recited in claim 2, wherein a plurality of input devices are coupled to the alternative input subsystem module.
9. The advanced user interface as recited in claim 8, wherein the interface profiles contain mappings of input messages from a different input device from the input devices for which at least one of the plurality of application programs were originally written.
10. The advanced user interface as recited in claim 9 wherein the interface profiles contain mappings of touch input messages against corresponding mouse commands.
11. The advanced user interface as recited in claim 10 wherein the interface profiles also contain mappings of touch input gesture tokens representing a series of points drawn by a user on an input surface detected by a touch input device against corresponding commands.
12. An advanced user interface, for use with an integrated operating environment and an operating system capable of running a plurality of application programs simultaneously on a computer system, only one of which is active at a given time, the computer system having a central processor, a random access memory, a display and at least one input device which transmits input signals to the advanced user interface in response to actions performed by a user, comprising:
an alternative input subsystem module coupled to the input device which translates the input signals to input messages compatible with the integrated operating environment;
interface profiles which contain mappings of the input messages against corresponding commands usable by at least one of the plurality of application programs, and, an environment link module coupled to the alternative input subsystem, the interface profiles and the integrated operating environment, the environment link module receiving the input messages, intercepting the compatible input messages before receipt by the active application program, querying the alternative input subsystem as to whether the compatible input messages are true input messages, determining which of the plurality of application programs was active at the time the input signals were transmitted by the input device, matching the true input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device and sending the corresponding commands to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the application program.
an alternative input subsystem module coupled to the input device which translates the input signals to input messages compatible with the integrated operating environment;
interface profiles which contain mappings of the input messages against corresponding commands usable by at least one of the plurality of application programs, and, an environment link module coupled to the alternative input subsystem, the interface profiles and the integrated operating environment, the environment link module receiving the input messages, intercepting the compatible input messages before receipt by the active application program, querying the alternative input subsystem as to whether the compatible input messages are true input messages, determining which of the plurality of application programs was active at the time the input signals were transmitted by the input device, matching the true input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device and sending the corresponding commands to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the application program.
13. The advanced user interface as recited in claim 12, wherein the environmental link first sends the true input messages to the application which was active at the time the input messages were transmitted by the input device, and responsive to a message indicating that the active application did not understand the true input message, matching the true input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device and sending the corresponding commands to an appropriate code module.
14. A computer system having a central processor and a random access memory having an advanced user interface, an integrated operating environment and an operating system resident in the memory capable of running a plurality of application programs simultaneously, only one of which is active at a given time, comprising:
a plurality of input devices which transmit input signals to the advanced user interface in response to actions performed by the user;
translating means coupled to the plurality of input devices to translate the input signals to input messages;
interface profiles which contain mappings of the input messages against corresponding commands usable by at least one of the plurality of application programs;
an environment link module coupled to the translating means input subsystem, the interface profiles and the integrated operating environment, the environment link module receiving the input messages, determining which of the application programs was active at the time the input signals were transmitted by the input devices, matching the input messages against the corresponding commands contained in the interface profiles for the application program which was active at the time the input signals were transmitted by one of the plurality of input devices, and sending the corresponding commands to an appropriate code module; and, a display that presents information concerning the status, results, choices and other items associated with the plurality of application programs;
the computer system performing a given action in response to user action different from those user actions for which the active application program was originally written without change to the active application program.
a plurality of input devices which transmit input signals to the advanced user interface in response to actions performed by the user;
translating means coupled to the plurality of input devices to translate the input signals to input messages;
interface profiles which contain mappings of the input messages against corresponding commands usable by at least one of the plurality of application programs;
an environment link module coupled to the translating means input subsystem, the interface profiles and the integrated operating environment, the environment link module receiving the input messages, determining which of the application programs was active at the time the input signals were transmitted by the input devices, matching the input messages against the corresponding commands contained in the interface profiles for the application program which was active at the time the input signals were transmitted by one of the plurality of input devices, and sending the corresponding commands to an appropriate code module; and, a display that presents information concerning the status, results, choices and other items associated with the plurality of application programs;
the computer system performing a given action in response to user action different from those user actions for which the active application program was originally written without change to the active application program.
15. The computer system as recited in claim 14, which further comprises a second plurality of input devices coupled to the integrated operating environment through standard device drivers.
16. The computer system as recited in claim 15, wherein at least one of the second plurality of input devices is also coupled to the translating means.
17. The computer system as recited in claim 14, which further comprises a gesture recognition unit which interprets the input signals transmitted by a touch input device in response to a user drawing symbols on a surface detected by the touch input device.
18. The computer system as recited in claim 14, which further comprises a character recognition unit which interprets the input signals transmitted by a touch input device in response to a user writing alphanumeric characters on a surface detected by the touch input device.
19. The computer system as recited in claim 14, wherein input messages of a touch input device are mapped against corresponding mouse commands for at least one of the plurality of application programs.
20. The computer system as recited in claim 14, wherein the interface profiles comprise a plurality of application profiles each of which is for one of the plurality of application programs containing mappings of input messages against corresponding commands at least some of which are usable by its corresponding application program.
21. The computer system as recited in claim 20, wherein the interface profiles further comprise a user profile which contains mappings of input messages against corresponding commands, the user profile usable with more than one of the plurality of application programs.
22. The computer system as recited in claim 21, wherein at least one of the corresponding commands contained in the user profile overrides at least one of the corresponding commands contained in the plurality of application profiles.
23. The computer system as recited in claim 21, wherein at least one of the corresponding commands contained in the user profile is a default command which is sent only when the application profile of the active application program has no corresponding command to the input message.
24. The computer system as recited in claim 21, wherein the interface profiles further comprise a user profile for each user authorized to utilize the computer system.
25. The computer system as recited in claim 14, which further comprises a touch input device disposed over the viewing surface of the display device, and wherein the advanced user interface further comprises a popup keyboard module which displays an image of a keyboard on the display device and in response to a selection of a key in the keyboard transmits a character string to the appropriate computer module in the random access memory.
26. The computer system as recited in claim 14, wherein the advanced user interface further comprises an image magnifying utility which magnifies a selected portion of an image displayed on the display device so that a more precise pointing operation can be accomplished.
27. The computer system as recited in claim 14, wherein the interface profiles contain mappings of input messages from a different input device form the input devices for which at least one of the plurality of application programs was originally written.
28. The computer system as recited in claim 14, wherein the advanced user interface further comprises an advanced user interface control panel to allow a user to create and modify the advanced user interface in a user friendly manner.
29. A computer system having a central processor and a random access memory having an operating system, an advanced user interface and an integrated operating environment and an operating system resident in the memory capable of running a plurality of application programs simultaneously, only one of which is active at a given time, comprising:
a plurality of input devices which transmit input signals to the advanced user interface in response to actions performed by the user;
an alternative input subsystem module coupled to the input device which translates the input signals to input messages compatible with the integrated operating environment and the operating system;
interface profiles which contain mappings of the input messages against corresponding commands usable by at least one of the plurality of application programs; and, an environment link module coupled to the alternative input subsystem, the interface profiles and the integrated operating environment, the environment link module receiving the input messages, intercepting the compatible input messages before receipt by the active application program, querying the alternative input subsystem as to whether the compatible input messages are true input messages, determining which of the plurality of application programs was active at the time the input signals were transmitted by the input device, matching the true input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device and sending the corresponding commands to an appropriate code module; and a display that presents information concerning the status, results, choices and other items associated with the plurality of application programs;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the application program.
a plurality of input devices which transmit input signals to the advanced user interface in response to actions performed by the user;
an alternative input subsystem module coupled to the input device which translates the input signals to input messages compatible with the integrated operating environment and the operating system;
interface profiles which contain mappings of the input messages against corresponding commands usable by at least one of the plurality of application programs; and, an environment link module coupled to the alternative input subsystem, the interface profiles and the integrated operating environment, the environment link module receiving the input messages, intercepting the compatible input messages before receipt by the active application program, querying the alternative input subsystem as to whether the compatible input messages are true input messages, determining which of the plurality of application programs was active at the time the input signals were transmitted by the input device, matching the true input messages against the corresponding commands contained in the interface profiles according to the application program which was active at the time the input signals were transmitted by the input device and sending the corresponding commands to an appropriate code module; and a display that presents information concerning the status, results, choices and other items associated with the plurality of application programs;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the application program.
30. The computer system as recited in claim 29, wherein the environmental link sends the true input messages to the application which was active at the time the input messages were transmitted by the input device and only responsive to a message indicating that the active application did not understand the true input message, matching the true input messages against the corresponding commands contained in the interface profiles.
31. A method of operating a computer system with an integrated operating environment and an operating system capable of running a plurality of application programs simultaneously only one of which is active at a given time by the use of an advanced user interface, the computer system having a central processor, a random access memory, a display and at least one input device which transmits input signals to the advanced user interface comprising the steps of:
determining which one of the application programs was active at the time the input signals were transmitted by the input device;
finding a corresponding command to the input message representative of the input signals in a set of interface profiles for the active application program, the interface profiles comprising mappings of the input messages against corresponding commands; and, sending the corresponding command to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the active application program.
determining which one of the application programs was active at the time the input signals were transmitted by the input device;
finding a corresponding command to the input message representative of the input signals in a set of interface profiles for the active application program, the interface profiles comprising mappings of the input messages against corresponding commands; and, sending the corresponding command to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the active application program.
32. The method of operating a computer system as recited in claim 31 which further comprises the steps of:
receiving the input signals transmitted from the input device in response to actions performed by a user; and, translating the input signals into input messages.
receiving the input signals transmitted from the input device in response to actions performed by a user; and, translating the input signals into input messages.
33. The method of operating a computer system as recited in claim 31 wherein the input device is a touch input device, and the interface profiles contain mappings of touch input messages against corresponding mouse commands.
34. The method of operating a computer system as recited in claim 33, wherein the interface profiles also contain mappings of touch input gesture tokens against corresponding commands, the touch input gesture tokens representing a series of points drawn by a user on an input surface detected by a touch input device.
35. The method of operating a computer system as recited in claim 31, wherein the finding step comprises the steps of:
finding a corresponding command to the input messages in an application profile for the active application;
finding a corresponding command to the input messages in a user profile for the user operating the computer system; and, determining which corresponding command has higher priority.
finding a corresponding command to the input messages in an application profile for the active application;
finding a corresponding command to the input messages in a user profile for the user operating the computer system; and, determining which corresponding command has higher priority.
36. The method of operating a computer system as recited in claim 31, wherein the corresponding command is sent to the active application program.
37. The method of operating a computer system as recited in claim 31, wherein the corresponding command is sent to a utilities module of the advanced user interface.
38. A method of operating a computer system with an integrated operating environment and an operating system capable of running a plurality of application programs simultaneously only one of which is active at a given time by the use of an advanced user interface, comprising the steps of:
receiving input signals transmitted from at least one input device in response to actions performed by a user;
translating the input signals into an input message compatible with the integrated operating environment;
intercepting the compatible input message before receipt by an active application program;
determining whether the compatible input message is a true input message;
determining which one of the application programs was active at the time the input signals were transmitted;
finding a corresponding command to the true input message in a set of interface profiles for the active application program, the interface profiles comparing mappings of the true input messages against corresponding commands; and, sending the corresponding command to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the active application program.
receiving input signals transmitted from at least one input device in response to actions performed by a user;
translating the input signals into an input message compatible with the integrated operating environment;
intercepting the compatible input message before receipt by an active application program;
determining whether the compatible input message is a true input message;
determining which one of the application programs was active at the time the input signals were transmitted;
finding a corresponding command to the true input message in a set of interface profiles for the active application program, the interface profiles comparing mappings of the true input messages against corresponding commands; and, sending the corresponding command to an appropriate code module;
the computer system performing a given action in response to user actions different from those user actions for which the active application program was originally written without change to the active application program.
39. The method of operating a computer system as recited in claim 38 which further comprises the step of sending the true input message to the application which was active at the time the input signals were received and wherein only responsive to a message that the active application did not understand the true input message performing the finding and sending steps.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US344,879 | 1989-04-28 | ||
US07/344,879 US5157384A (en) | 1989-04-28 | 1989-04-28 | Advanced user interface |
Publications (2)
Publication Number | Publication Date |
---|---|
CA2007411A1 CA2007411A1 (en) | 1990-10-28 |
CA2007411C true CA2007411C (en) | 1998-04-21 |
Family
ID=23352455
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CA002007411A Expired - Fee Related CA2007411C (en) | 1989-04-28 | 1990-01-09 | Advanced user interface |
Country Status (5)
Country | Link |
---|---|
US (1) | US5157384A (en) |
EP (1) | EP0394614B1 (en) |
JP (1) | JPH0320812A (en) |
CA (1) | CA2007411C (en) |
DE (1) | DE69026417T2 (en) |
Families Citing this family (230)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5252951A (en) * | 1989-04-28 | 1993-10-12 | International Business Machines Corporation | Graphical user interface with gesture recognition in a multiapplication environment |
JPH03163623A (en) * | 1989-06-23 | 1991-07-15 | Articulate Syst Inc | Voice control computor interface |
JPH0778782B2 (en) * | 1989-07-19 | 1995-08-23 | インターナショナル・ビジネス・マシーンズ・コーポレーシヨン | Interactive computer system and apparatus and method for adapting use environment thereof |
US5276816A (en) * | 1990-12-31 | 1994-01-04 | International Business Machines Corporation | Icon object interface system and method |
US5497319A (en) * | 1990-12-31 | 1996-03-05 | Trans-Link International Corp. | Machine translation and telecommunications system |
US5606702A (en) * | 1991-03-30 | 1997-02-25 | International Business Machines Corporation | Method for specifying user interfaces and programming system running a multiple user interface computer |
JPH0520274A (en) * | 1991-07-10 | 1993-01-29 | Canon Inc | Electronic information equipment |
JP2695542B2 (en) * | 1991-07-12 | 1997-12-24 | 財団法人ニューメディア開発協会 | Information processing device that uses and acquires user information |
US5301326A (en) * | 1991-09-24 | 1994-04-05 | Microsoft Corporation | Method and system for controlling the execution of an application program |
US5825865A (en) * | 1991-10-04 | 1998-10-20 | Motorola, Inc. | Temporary message routing and destination selection |
JPH0736142B2 (en) * | 1991-10-10 | 1995-04-19 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Method and information processing apparatus for recognizing movement stop of movement instruction means |
CA2076466C (en) * | 1991-10-21 | 2002-01-15 | Alan R. Tannenbaum | Method for buffering high bandwidth data from an input device |
US5848187A (en) * | 1991-11-18 | 1998-12-08 | Compaq Computer Corporation | Method and apparatus for entering and manipulating spreadsheet cell data |
US8352400B2 (en) | 1991-12-23 | 2013-01-08 | Hoffberg Steven M | Adaptive pattern recognition based controller apparatus and method and human-factored interface therefore |
JP3286339B2 (en) * | 1992-03-25 | 2002-05-27 | 株式会社リコー | Window screen control device |
US5317695A (en) * | 1992-04-03 | 1994-05-31 | International Business Machines Corporation | Method and system for permitting communication between a data processing system and input pointing devices of multiple types |
US5404524A (en) * | 1992-04-03 | 1995-04-04 | International Business Machines Corporation | System for identifying attached input pointing devices, loading associated software routines, and interacting with anyone input pointing device while disabling the others |
US5818800A (en) * | 1992-04-06 | 1998-10-06 | Barker; Bruce J. | Voice recording device having portable and local modes of operation |
US5508821A (en) * | 1992-04-09 | 1996-04-16 | Matsushita Electric Industrial Co., Ltd. | Image scanner and image forming apparatus with an interface for connection with an external computer |
CA2089784C (en) * | 1992-04-15 | 1996-12-24 | William Joseph Anderson | Apparatus and method for disambiguating an input stream generated by a stylus-based user interface |
EP0566293B1 (en) * | 1992-04-15 | 2003-07-16 | Xerox Corporation | Graphical drawing and editing systems and methods therefor |
US5861583A (en) | 1992-06-08 | 1999-01-19 | Synaptics, Incorporated | Object position detector |
US6028271A (en) | 1992-06-08 | 2000-02-22 | Synaptics, Inc. | Object position detector with edge motion feature and gesture recognition |
US5880411A (en) | 1992-06-08 | 1999-03-09 | Synaptics, Incorporated | Object position detector with edge motion feature and gesture recognition |
DE69324067T2 (en) | 1992-06-08 | 1999-07-15 | Synaptics Inc | Object position detector |
US5942733A (en) * | 1992-06-08 | 1999-08-24 | Synaptics, Inc. | Stylus input capacitive touchpad sensor |
US5889236A (en) * | 1992-06-08 | 1999-03-30 | Synaptics Incorporated | Pressure sensitive scrollbar feature |
US6239389B1 (en) | 1992-06-08 | 2001-05-29 | Synaptics, Inc. | Object position detection system and method |
FR2696258B1 (en) * | 1992-09-25 | 1994-10-28 | Sextant Avionique | Device for managing a human-machine interaction system. |
FR2696574B1 (en) * | 1992-10-06 | 1994-11-18 | Sextant Avionique | Method and device for analyzing a message supplied by means of interaction with a human-machine dialogue system. |
EP0593386A3 (en) * | 1992-10-16 | 1996-07-31 | Ibm | Method and apparatus for accessing touch screen desktop objects via fingerprint recognition |
US5442376A (en) * | 1992-10-26 | 1995-08-15 | International Business Machines Corporation | Handling multiple command recognition inputs in a multi-tasking graphical environment |
US5561811A (en) * | 1992-11-10 | 1996-10-01 | Xerox Corporation | Method and apparatus for per-user customization of applications shared by a plurality of users on a single display |
EP0606811A3 (en) * | 1992-12-28 | 1995-06-21 | Ibm | Employing a preferred object handler. |
US5502803A (en) * | 1993-01-18 | 1996-03-26 | Sharp Kabushiki Kaisha | Information processing apparatus having a gesture editing function |
US5890122A (en) * | 1993-02-08 | 1999-03-30 | Microsoft Corporation | Voice-controlled computer simulateously displaying application menu and list of available commands |
CA2157910A1 (en) * | 1993-03-10 | 1994-09-15 | Bruce Barker | Data entry device |
CA2115210C (en) * | 1993-04-21 | 1997-09-23 | Joseph C. Andreshak | Interactive computer system recognizing spoken commands |
US5638501A (en) | 1993-05-10 | 1997-06-10 | Apple Computer, Inc. | Method and apparatus for displaying an overlay image |
US5614926A (en) * | 1993-05-17 | 1997-03-25 | Sharp Kabushiki Kaisha | Word processor with a handwriting text processing function |
WO1995003574A1 (en) * | 1993-07-19 | 1995-02-02 | Taligent, Inc. | Dynamic linking system |
CA2124505C (en) * | 1993-07-21 | 2000-01-04 | William A. S. Buxton | User interface having simultaneously movable tools and cursor |
US5581670A (en) * | 1993-07-21 | 1996-12-03 | Xerox Corporation | User interface having movable sheet with click-through tools |
CA2124624C (en) * | 1993-07-21 | 1999-07-13 | Eric A. Bier | User interface having click-through tools that can be composed with other tools |
JPH0792993A (en) * | 1993-09-20 | 1995-04-07 | Fujitsu Ltd | Speech recognizing device |
US5583761A (en) * | 1993-10-13 | 1996-12-10 | Kt International, Inc. | Method for automatic displaying program presentations in different languages |
JP2813728B2 (en) * | 1993-11-01 | 1998-10-22 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Personal communication device with zoom / pan function |
US5375084A (en) * | 1993-11-08 | 1994-12-20 | International Business Machines Corporation | Selectable interface between memory controller and memory simms |
JP2710547B2 (en) * | 1994-02-15 | 1998-02-10 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Graphical user interface |
US5542069A (en) * | 1994-03-02 | 1996-07-30 | Sun Microsystems, Inc. | Method and apparatus for simulating input events in a windowed environment |
US5613137A (en) * | 1994-03-18 | 1997-03-18 | International Business Machines Corporation | Computer system with touchpad support in operating system |
US5681220A (en) * | 1994-03-18 | 1997-10-28 | International Business Machines Corporation | Keyboard touchpad combination in a bivalve enclosure |
EP0756725B1 (en) * | 1994-04-21 | 1999-07-28 | BRITISH TELECOMMUNICATIONS public limited company | Interface device and method |
US6496872B1 (en) * | 1994-05-16 | 2002-12-17 | Apple Computer, Inc. | Computer system for automatically instantiating tasks designated by a user |
US5748980A (en) * | 1994-05-27 | 1998-05-05 | Microsoft Corporation | System for configuring a computer system |
US5787246A (en) * | 1994-05-27 | 1998-07-28 | Microsoft Corporation | System for configuring devices for a computer system |
US5655148A (en) | 1994-05-27 | 1997-08-05 | Microsoft Corporation | Method for automatically configuring devices including a network adapter without manual intervention and without prior configuration information |
US6763454B2 (en) | 1994-05-27 | 2004-07-13 | Microsoft Corp. | System for allocating resources in a computer system |
US5570113A (en) * | 1994-06-29 | 1996-10-29 | International Business Machines Corporation | Computer based pen system and method for automatically cancelling unwanted gestures and preventing anomalous signals as inputs to such system |
US5867106A (en) * | 1994-09-02 | 1999-02-02 | Packard Bell Nec | Password switch to override remote control |
US6292181B1 (en) | 1994-09-02 | 2001-09-18 | Nec Corporation | Structure and method for controlling a host computer using a remote hand-held interface device |
US6209034B1 (en) | 1994-09-02 | 2001-03-27 | Nec Corporation | Remote keyboard macros activated by hot icons |
US5974558A (en) * | 1994-09-02 | 1999-10-26 | Packard Bell Nec | Resume on pen contact |
US6092117A (en) * | 1994-09-02 | 2000-07-18 | Packard Bell Nec | System and method for automatically reconnecting a wireless interface device to a host computer |
US6137473A (en) * | 1994-09-02 | 2000-10-24 | Nec Corporation | System and method for switching control between a host computer and a remote interface device |
US6262719B1 (en) | 1994-09-02 | 2001-07-17 | Packard Bell Nec, Inc. | Mouse emulation with a passive pen |
WO1996009579A1 (en) * | 1994-09-22 | 1996-03-28 | Izak Van Cruyningen | Popup menus with directional gestures |
JP3399665B2 (en) * | 1994-11-04 | 2003-04-21 | シャープ株式会社 | Information processing device |
US5630159A (en) * | 1994-12-29 | 1997-05-13 | Motorola, Inc. | Method and apparatus for personal attribute selection having delay management method and apparatus for preference establishment when preferences in a donor device are unavailable |
US5761420A (en) * | 1995-02-28 | 1998-06-02 | Johnson; William | Method and apparatus for telephone proof of documents |
US5706456A (en) * | 1995-04-18 | 1998-01-06 | Unisys Corporation | Application specific graphical user interface (GUI) that is window programmable and capable of operating above a windows operating system GUI |
US5802388A (en) * | 1995-05-04 | 1998-09-01 | Ibm Corporation | System and method for correction and confirmation dialog for hand printed character input to a data processing system |
EP0752646B1 (en) * | 1995-07-07 | 2002-03-27 | Sun Microsystems, Inc. | Data access implementation of device driver interface |
US6668244B1 (en) * | 1995-07-21 | 2003-12-23 | Quartet Technology, Inc. | Method and means of voice control of a computer, including its mouse and keyboard |
US5748191A (en) * | 1995-07-31 | 1998-05-05 | Microsoft Corporation | Method and system for creating voice commands using an automatically maintained log interactions performed by a user |
US5857172A (en) * | 1995-07-31 | 1999-01-05 | Microsoft Corporation | Activation control of a speech recognizer through use of a pointing device |
US5864815A (en) * | 1995-07-31 | 1999-01-26 | Microsoft Corporation | Method and system for displaying speech recognition status information in a visual notification area |
US5761641A (en) * | 1995-07-31 | 1998-06-02 | Microsoft Corporation | Method and system for creating voice commands for inserting previously entered information |
US5790889A (en) * | 1995-08-11 | 1998-08-04 | Dell U.S.A., L.P. | Method of selecting pointing device in a computer comprising responsive to a reconfiguration event, indentifying all connected pointing devices and selecting one of the devices |
US5787285A (en) * | 1995-08-15 | 1998-07-28 | International Business Machines Corporation | Apparatus and method for optimizing applications for multiple operational environments or modes |
US6289312B1 (en) * | 1995-10-02 | 2001-09-11 | Digital Equipment Corporation | Speech interface for computer application programs |
US5805777A (en) * | 1995-10-11 | 1998-09-08 | Eastman Kodak Company | Extended printer control interface |
US6664982B1 (en) | 1995-10-16 | 2003-12-16 | Nec Corporation | Multi-user on-screen keyboard |
US6005533A (en) * | 1995-10-16 | 1999-12-21 | Packard Bell Nec | Remote occlusion region |
US6108727A (en) * | 1995-10-16 | 2000-08-22 | Packard Bell Nec | System having wireless interface device for storing compressed predetermined program files received from a remote host and communicating with the remote host via wireless link |
US6126327A (en) * | 1995-10-16 | 2000-10-03 | Packard Bell Nec | Radio flash update |
US6018806A (en) * | 1995-10-16 | 2000-01-25 | Packard Bell Nec | Method and system for rebooting a computer having corrupted memory using an external jumper |
US5990875A (en) * | 1995-10-16 | 1999-11-23 | Packard Bell Nec | Double pen up event |
US6724372B1 (en) | 1995-10-16 | 2004-04-20 | Nec Corporation | Ink trails on a wireless remote interface tablet and wireless remote ink field object |
US6148344A (en) * | 1995-10-16 | 2000-11-14 | Nec Corporation | System and method for enabling an IPX driver to accommodate multiple LAN adapters |
US7512671B1 (en) * | 1995-10-16 | 2009-03-31 | Nec Corporation | Computer system for enabling a wireless interface device to selectively establish a communication link with a user selectable remote computer |
US6279153B1 (en) | 1995-10-16 | 2001-08-21 | Nec Corporation | Multi-user flash ROM update |
US5996082A (en) * | 1995-10-16 | 1999-11-30 | Packard Bell Nec | System and method for delaying a wake-up signal |
JP3434105B2 (en) * | 1995-11-28 | 2003-08-04 | 富士通株式会社 | Macro program management device |
US6300936B1 (en) | 1997-11-14 | 2001-10-09 | Immersion Corporation | Force feedback system including multi-tasking graphical host environment and interface device |
US5823782A (en) * | 1995-12-29 | 1998-10-20 | Tinkers & Chance | Character recognition educational system |
US5838972A (en) * | 1996-02-09 | 1998-11-17 | Sun Microsystems, Inc. | Method and apparatus for dynamically loading an input run-time module and an output run-time module |
USRE38286E1 (en) | 1996-02-15 | 2003-10-28 | Leapfrog Enterprises, Inc. | Surface position location system and method |
USRE39881E1 (en) | 1996-02-15 | 2007-10-16 | Leapfrog Enterprises, Inc. | Surface position location system and method |
US5898422A (en) * | 1996-05-31 | 1999-04-27 | International Business Machines Corporation | Method and system for recognition of pointers |
US5841994A (en) * | 1996-06-14 | 1998-11-24 | Texas Instruments Incorporated | Portable computer with multiple zoom port interface |
GB2315139A (en) * | 1996-07-06 | 1998-01-21 | Ibm | Selecting operations in an object-oriented user interface |
US5867153A (en) * | 1996-10-30 | 1999-02-02 | Transaction Technology, Inc. | Method and system for automatically harmonizing access to a software application program via different access devices |
US5873064A (en) * | 1996-11-08 | 1999-02-16 | International Business Machines Corporation | Multi-action voice macro method |
US6654955B1 (en) * | 1996-12-19 | 2003-11-25 | International Business Machines Corporation | Adding speech recognition libraries to an existing program at runtime |
US6268857B1 (en) | 1997-08-29 | 2001-07-31 | Xerox Corporation | Computer user interface using a physical manipulatory grammar |
US6243075B1 (en) | 1997-08-29 | 2001-06-05 | Xerox Corporation | Graspable device manipulation for controlling a computer display |
US6243074B1 (en) | 1997-08-29 | 2001-06-05 | Xerox Corporation | Handedness detection for a physical manipulatory grammar |
US6297838B1 (en) | 1997-08-29 | 2001-10-02 | Xerox Corporation | Spinning as a morpheme for a physical manipulatory grammar |
US6297805B1 (en) | 1997-08-29 | 2001-10-02 | Xerox Corporation | Multiple interacting computers interfaceable through a physical manipulatory grammar |
US6678744B2 (en) * | 1997-10-09 | 2004-01-13 | Ericsson Inc. | Application wrapper methods and systems |
US8020095B2 (en) | 1997-11-14 | 2011-09-13 | Immersion Corporation | Force feedback system including multi-tasking graphical host environment |
US6160540A (en) * | 1998-01-12 | 2000-12-12 | Xerox Company | Zoomorphic computer user interface |
US9239673B2 (en) | 1998-01-26 | 2016-01-19 | Apple Inc. | Gesturing with a multipoint sensing device |
US8479122B2 (en) | 2004-07-30 | 2013-07-02 | Apple Inc. | Gestures for touch sensitive input devices |
US7614008B2 (en) | 2004-07-30 | 2009-11-03 | Apple Inc. | Operation of a computer with touch screen interface |
US20070177804A1 (en) * | 2006-01-30 | 2007-08-02 | Apple Computer, Inc. | Multi-touch gesture dictionary |
US9292111B2 (en) * | 1998-01-26 | 2016-03-22 | Apple Inc. | Gesturing with a multipoint sensing device |
US6509913B2 (en) | 1998-04-30 | 2003-01-21 | Openwave Systems Inc. | Configurable man-machine interface |
US6438523B1 (en) | 1998-05-20 | 2002-08-20 | John A. Oberteuffer | Processing handwritten and hand-drawn input and speech input |
US6263375B1 (en) * | 1998-08-31 | 2001-07-17 | International Business Machines Corp. | Method for creating dictation macros |
AU6040999A (en) * | 1998-09-14 | 2000-04-03 | Microsoft Corporation | Input device with forward/backward control |
US7002552B1 (en) | 1999-04-06 | 2006-02-21 | Microsoft Corporation | Computer input device with digit support and natural position actuators |
US8275617B1 (en) | 1998-12-17 | 2012-09-25 | Nuance Communications, Inc. | Speech command input recognition system for interactive computer display with interpretation of ancillary relevant speech query terms into commands |
US6192343B1 (en) | 1998-12-17 | 2001-02-20 | International Business Machines Corporation | Speech command input recognition system for interactive computer display with term weighting means used in interpreting potential commands from relevant speech terms |
US7206747B1 (en) | 1998-12-16 | 2007-04-17 | International Business Machines Corporation | Speech command input recognition system for interactive computer display with means for concurrent and modeless distinguishing between speech commands and speech queries for locating commands |
US6937984B1 (en) | 1998-12-17 | 2005-08-30 | International Business Machines Corporation | Speech command input recognition system for interactive computer display with speech controlled display of recognized commands |
US6233560B1 (en) | 1998-12-16 | 2001-05-15 | International Business Machines Corporation | Method and apparatus for presenting proximal feedback in voice command systems |
US6963937B1 (en) | 1998-12-17 | 2005-11-08 | International Business Machines Corporation | Method and apparatus for providing configurability and customization of adaptive user-input filtration |
US6570555B1 (en) * | 1998-12-30 | 2003-05-27 | Fuji Xerox Co., Ltd. | Method and apparatus for embodied conversational characters with multimodal input/output in an interface device |
US6336149B1 (en) * | 1999-01-26 | 2002-01-01 | Glenn W. Preston | Macro recording and playback device independent of operating system or application software and method of using same |
US7966078B2 (en) | 1999-02-01 | 2011-06-21 | Steven Hoffberg | Network media appliance system and method |
US6965368B1 (en) * | 1999-04-06 | 2005-11-15 | Microsoft Corporation | Game control device having genre data |
US6727884B1 (en) * | 1999-04-06 | 2004-04-27 | Microsoft Corporation | System and method for mapping input device controls to software actions |
US7116310B1 (en) * | 1999-04-06 | 2006-10-03 | Microsoft Corporation | Application programming interface that maps input device controls to software actions |
US6711620B1 (en) | 1999-04-14 | 2004-03-23 | Matsushita Electric Industrial Co. | Event control device and digital broadcasting system |
JP4528384B2 (en) * | 1999-07-08 | 2010-08-18 | マイクロソフト コーポレーション | Display device for graphical user interface for voice input |
US6598169B1 (en) * | 1999-07-26 | 2003-07-22 | Microsoft Corporation | System and method for accessing information made available by a kernel mode driver |
US6724403B1 (en) | 1999-10-29 | 2004-04-20 | Surfcast, Inc. | System and method for simultaneous display of multiple information sources |
US7987431B2 (en) | 1999-10-29 | 2011-07-26 | Surfcast, Inc. | System and method for simultaneous display of multiple information sources |
US7028264B2 (en) * | 1999-10-29 | 2006-04-11 | Surfcast, Inc. | System and method for simultaneous display of multiple information sources |
US7016835B2 (en) * | 1999-10-29 | 2006-03-21 | International Business Machines Corporation | Speech and signal digitization by using recognition metrics to select from multiple techniques |
GB9930851D0 (en) | 1999-12-24 | 2000-02-16 | Koninkl Philips Electronics Nv | Apparatus control system and method |
US6642942B1 (en) * | 2000-03-07 | 2003-11-04 | Intel Corporation | Method and system for configuring among call processing applications in a call processing system |
US7058888B1 (en) * | 2000-04-25 | 2006-06-06 | Microsoft Corporation | Multi-modal text editing correction |
US6661405B1 (en) | 2000-04-27 | 2003-12-09 | Leapfrog Enterprises, Inc. | Electrographic position location apparatus and method |
US6750978B1 (en) * | 2000-04-27 | 2004-06-15 | Leapfrog Enterprises, Inc. | Print media information system with a portable print media receiving unit assembly |
US6668156B2 (en) | 2000-04-27 | 2003-12-23 | Leapfrog Enterprises, Inc. | Print media receiving unit including platform and print media |
AU2001230044A1 (en) * | 2000-11-27 | 2002-06-03 | Nokia Corporation | Method and system providing a mediation function |
DE60117676T2 (en) * | 2000-12-29 | 2006-11-16 | Stmicroelectronics S.R.L., Agrate Brianza | A method for easily extending the functionality of a portable electronic device and associated portable electronic device |
US7916124B1 (en) | 2001-06-20 | 2011-03-29 | Leapfrog Enterprises, Inc. | Interactive apparatus using print media |
US6608618B2 (en) | 2001-06-20 | 2003-08-19 | Leapfrog Enterprises, Inc. | Interactive apparatus using print media |
EP1286261A1 (en) * | 2001-08-13 | 2003-02-26 | Alcatel | Method, control interface module, intermediate modules and so equipped network management system for the operation of a control interface module |
EP1321852A1 (en) * | 2001-10-03 | 2003-06-25 | C O2W3 Sas | Modular logic platform |
US9189467B1 (en) | 2001-11-07 | 2015-11-17 | Apple Inc. | Method and apparatus for annotating an electronic document |
MXPA04007716A (en) * | 2002-02-06 | 2004-12-06 | Leapfrog Entpr Inc | Write on interactive apparatus and method. |
EP1388786A1 (en) * | 2002-03-15 | 2004-02-11 | Alcatel | User interface for interaction with a computer |
US7490329B2 (en) | 2002-04-22 | 2009-02-10 | Thomson Licensing | Remote key manager |
US7099893B2 (en) * | 2002-06-28 | 2006-08-29 | Sap Aktiengesellschaft | Recording application user actions for application testing |
US8682636B2 (en) * | 2002-08-30 | 2014-03-25 | Sap Ag | Non-client-specific testing of applications |
US20040063078A1 (en) * | 2002-09-30 | 2004-04-01 | Marcus Brian I. | Electronic educational toy appliance |
US20040229195A1 (en) * | 2003-03-18 | 2004-11-18 | Leapfrog Enterprises, Inc. | Scanning apparatus |
US8745541B2 (en) * | 2003-03-25 | 2014-06-03 | Microsoft Corporation | Architecture for controlling a computer using hand gestures |
US7068262B2 (en) * | 2003-06-09 | 2006-06-27 | Leapfrog Enterprises, Inc. | Writing stylus for electrographic position location apparatus |
EP1654632A1 (en) * | 2003-08-14 | 2006-05-10 | NTT DoCoMo INC. | Direct data input |
US20050055647A1 (en) * | 2003-09-05 | 2005-03-10 | Chung-Hui Chen | System and method for switching software functions |
MXPA06007945A (en) * | 2004-01-13 | 2007-01-26 | Univ Toledo | Noninvasive birefringence compensated sensing polarimeter. |
US7207885B2 (en) | 2004-01-15 | 2007-04-24 | Espeed, Inc. | System and method for using a game controller device for electronic trading |
US8469808B2 (en) | 2004-01-15 | 2013-06-25 | Bgc Partners, Inc. | System and method for managing a game controller device for electronic trading |
US8170945B2 (en) | 2004-01-15 | 2012-05-01 | Bgc Partners, Inc. | System and method for providing security to a game controller device for electronic trading |
US7458029B2 (en) * | 2004-01-15 | 2008-11-25 | Microsoft Corporation | System and process for controlling a shared display given inputs from multiple users using multiple input modalities |
US7707039B2 (en) | 2004-02-15 | 2010-04-27 | Exbiblio B.V. | Automatic modification of web pages |
US8442331B2 (en) | 2004-02-15 | 2013-05-14 | Google Inc. | Capturing text from rendered documents using supplemental information |
US10635723B2 (en) | 2004-02-15 | 2020-04-28 | Google Llc | Search engines and systems with handheld document data capture devices |
US7812860B2 (en) | 2004-04-01 | 2010-10-12 | Exbiblio B.V. | Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device |
US7345681B2 (en) * | 2004-02-17 | 2008-03-18 | Microsoft Corporation | Pen data capture and injection |
US20060125805A1 (en) * | 2004-03-17 | 2006-06-15 | James Marggraff | Method and system for conducting a transaction using recognized text |
US20060067576A1 (en) * | 2004-03-17 | 2006-03-30 | James Marggraff | Providing a user interface having interactive elements on a writable surface |
US20060098900A1 (en) | 2004-09-27 | 2006-05-11 | King Martin T | Secure data gathering from rendered documents |
US20060081714A1 (en) | 2004-08-23 | 2006-04-20 | King Martin T | Portable scanning device |
US7990556B2 (en) | 2004-12-03 | 2011-08-02 | Google Inc. | Association of a portable scanner with input/output and storage devices |
US9116890B2 (en) | 2004-04-01 | 2015-08-25 | Google Inc. | Triggering actions in response to optically or acoustically capturing keywords from a rendered document |
US7894670B2 (en) | 2004-04-01 | 2011-02-22 | Exbiblio B.V. | Triggering actions in response to optically or acoustically capturing keywords from a rendered document |
US8146156B2 (en) | 2004-04-01 | 2012-03-27 | Google Inc. | Archive of text captures from rendered documents |
US9143638B2 (en) | 2004-04-01 | 2015-09-22 | Google Inc. | Data capture from rendered documents using handheld device |
US8489624B2 (en) | 2004-05-17 | 2013-07-16 | Google, Inc. | Processing techniques for text capture from a rendered document |
US8620083B2 (en) | 2004-12-03 | 2013-12-31 | Google Inc. | Method and system for character recognition |
US8874504B2 (en) | 2004-12-03 | 2014-10-28 | Google Inc. | Processing techniques for visual capture data from a rendered document |
US8346620B2 (en) | 2004-07-19 | 2013-01-01 | Google Inc. | Automatic modification of web pages |
US7653883B2 (en) | 2004-07-30 | 2010-01-26 | Apple Inc. | Proximity detector in handheld device |
US8381135B2 (en) | 2004-07-30 | 2013-02-19 | Apple Inc. | Proximity detector in handheld device |
US7167675B2 (en) * | 2004-10-15 | 2007-01-23 | Leapfrog Enterprises, Inc. | Magnetic switch and apparatus including magnetic switch |
US7922099B1 (en) | 2005-07-29 | 2011-04-12 | Leapfrog Enterprises, Inc. | System and method for associating content with an image bearing surface |
US7883420B2 (en) | 2005-09-12 | 2011-02-08 | Mattel, Inc. | Video game systems |
US7936339B2 (en) * | 2005-11-01 | 2011-05-03 | Leapfrog Enterprises, Inc. | Method and system for invoking computer functionality by interaction with dynamically generated interface regions of a writing surface |
EP1865410A1 (en) * | 2006-06-09 | 2007-12-12 | Delta Electronics, Inc. | Human-machine interface system with device bridge and method for designing and operating the same |
US8261967B1 (en) | 2006-07-19 | 2012-09-11 | Leapfrog Enterprises, Inc. | Techniques for interactively coupling electronic content with printed media |
EP2067119A2 (en) | 2006-09-08 | 2009-06-10 | Exbiblio B.V. | Optical scanners, such as hand-held optical scanners |
US8130194B1 (en) * | 2006-12-18 | 2012-03-06 | Imi Innovations, Inc. | Non-mouse devices that function via mouse-like messages |
US20080228618A1 (en) | 2007-03-15 | 2008-09-18 | Noviello Joseph C | System And Method For Providing An Operator Interface For Displaying Market Data, Trader Options, And Trader Input |
US7631124B2 (en) * | 2007-04-06 | 2009-12-08 | Microsoft Corporation | Application-specific mapping of input device elements |
US20090046064A1 (en) * | 2007-08-17 | 2009-02-19 | Microsoft Corporation | Pointing device for control of a graphical display or application |
US8638363B2 (en) | 2009-02-18 | 2014-01-28 | Google Inc. | Automatically capturing information, such as capturing information using a document-aware device |
US20090161575A1 (en) * | 2007-12-19 | 2009-06-25 | Research In Motion Limited | Long Term Evolution User Equipment Multi-Packet Data Network Connectivity Control |
US20090187847A1 (en) * | 2008-01-18 | 2009-07-23 | Palm, Inc. | Operating System Providing Consistent Operations Across Multiple Input Devices |
FR2933211B1 (en) * | 2008-06-26 | 2010-11-26 | Airbus France | INTERACTION DEVICE WITH A DISPLAY SYSTEM, IN PARTICULAR FOR AN AVIONIC DISPLAY SYSTEM |
US20100095250A1 (en) * | 2008-10-15 | 2010-04-15 | Raytheon Company | Facilitating Interaction With An Application |
KR101185093B1 (en) * | 2008-12-19 | 2012-09-21 | 한국전자통신연구원 | Project management device and method for architecture modeling tool of application software on AUTOSAR |
WO2010105246A2 (en) | 2009-03-12 | 2010-09-16 | Exbiblio B.V. | Accessing resources based on capturing information from a rendered document |
US8447066B2 (en) | 2009-03-12 | 2013-05-21 | Google Inc. | Performing actions based on capturing information from rendered documents, such as documents under copyright |
US8421761B2 (en) * | 2009-08-26 | 2013-04-16 | General Electric Company | Imaging multi-modality touch pad interface systems, methods, articles of manufacture, and apparatus |
US9081799B2 (en) | 2009-12-04 | 2015-07-14 | Google Inc. | Using gestalt information to identify locations in printed information |
US9323784B2 (en) | 2009-12-09 | 2016-04-26 | Google Inc. | Image search using text-based elements within the contents of images |
US9092128B2 (en) | 2010-05-21 | 2015-07-28 | Apple Inc. | Method and apparatus for managing visual information |
US8151279B1 (en) * | 2011-03-28 | 2012-04-03 | Google Inc. | Uniform event handling across multiple computing devices |
US8392935B2 (en) | 2011-03-28 | 2013-03-05 | Google Inc. | Uniform event handling across multiple computing devices |
US20130139113A1 (en) * | 2011-11-30 | 2013-05-30 | Microsoft Corporation | Quick action for performing frequent tasks on a mobile device |
KR101322952B1 (en) * | 2012-02-24 | 2013-10-29 | 주식회사 팬택 | Apparatus and method that manage processing of motion realization in portable terminal |
US9274622B2 (en) * | 2012-09-11 | 2016-03-01 | Microsoft Technology Licensing, Llc | Device specific data in a unified pointer message |
CN102999340B (en) * | 2012-11-28 | 2016-08-31 | 百度在线网络技术(北京)有限公司 | The edit methods of a kind of interface of input method and device |
DE102014200782A1 (en) * | 2014-01-17 | 2015-07-23 | Bayerische Motoren Werke Aktiengesellschaft | Operating a vehicle according to the desire of a vehicle occupant |
EP3149558A4 (en) * | 2014-05-29 | 2017-12-20 | Hewlett-Packard Development Company, L.P. | User account switching interface |
US10078382B2 (en) * | 2014-09-04 | 2018-09-18 | Home Box Office, Inc. | Unified input and invoke handling |
US10044591B2 (en) | 2014-09-04 | 2018-08-07 | Home Box Office, Inc. | Two-way remote communication system for testing a client device program |
JP6138737B2 (en) * | 2014-10-02 | 2017-05-31 | 株式会社アスタリスク | Mobile terminal and keyboard application program |
US10168895B2 (en) * | 2015-08-04 | 2019-01-01 | International Business Machines Corporation | Input control on a touch-sensitive surface |
US10636074B1 (en) * | 2015-09-18 | 2020-04-28 | Amazon Technologies, Inc. | Determining and executing application functionality based on text analysis |
US20200050540A1 (en) * | 2018-08-10 | 2020-02-13 | International Business Machines Corporation | Interactive automation test |
TWI709071B (en) * | 2018-12-27 | 2020-11-01 | 英屬開曼群島商敦泰電子有限公司 | Touch pad driving system and method |
CN113094188A (en) * | 2021-03-30 | 2021-07-09 | 网易(杭州)网络有限公司 | System message processing method and device |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4451895A (en) * | 1980-07-17 | 1984-05-29 | Telesis Corporation Of Delaware, Inc. | Interactive computer aided design system |
JPS5968037A (en) * | 1982-09-21 | 1984-04-17 | Canon Inc | Input device |
DE3243679A1 (en) * | 1982-11-25 | 1984-05-30 | Preh Elektro Feinmechanik | TERMINAL OF A DATA PROCESSING SYSTEM |
US4726065A (en) * | 1984-01-26 | 1988-02-16 | Horst Froessl | Image manipulation by speech signals |
US4587630A (en) * | 1984-02-15 | 1986-05-06 | Hewlett-Packard Company | Intelligent programmable touchscreen system |
US4689761A (en) * | 1984-08-14 | 1987-08-25 | Metaphor Computer Systems | Multiple independent input peripherals |
US4972496A (en) * | 1986-07-25 | 1990-11-20 | Grid Systems Corporation | Handwritten keyboardless entry computer system |
JPS63172325A (en) * | 1987-01-10 | 1988-07-16 | Pioneer Electronic Corp | Touch panel controller |
US4903012A (en) * | 1987-01-20 | 1990-02-20 | Alps Electric Co., Ltd. | Coordinate system input device providing registration calibration and a mouse function |
JPS63298426A (en) * | 1987-05-29 | 1988-12-06 | Hitachi Ltd | Menu selecting and processing system |
US4896290A (en) * | 1987-08-24 | 1990-01-23 | Wang Laboratories, Inc. | Method for routing events from key strokes in a multi-processing computer systems |
US4908612A (en) * | 1987-09-29 | 1990-03-13 | Penguin Products, Inc. | Computer input-output device |
US4931950A (en) * | 1988-07-25 | 1990-06-05 | Electric Power Research Institute | Multimedia interface and method for computer system |
-
1989
- 1989-04-28 US US07/344,879 patent/US5157384A/en not_active Expired - Fee Related
-
1990
- 1990-01-09 CA CA002007411A patent/CA2007411C/en not_active Expired - Fee Related
- 1990-01-23 EP EP90101292A patent/EP0394614B1/en not_active Expired - Lifetime
- 1990-01-23 DE DE69026417T patent/DE69026417T2/en not_active Expired - Lifetime
- 1990-04-27 JP JP2110657A patent/JPH0320812A/en active Granted
Also Published As
Publication number | Publication date |
---|---|
DE69026417T2 (en) | 1996-11-14 |
EP0394614A2 (en) | 1990-10-31 |
US5157384A (en) | 1992-10-20 |
JPH0580009B2 (en) | 1993-11-05 |
EP0394614A3 (en) | 1992-09-02 |
CA2007411A1 (en) | 1990-10-28 |
DE69026417D1 (en) | 1996-05-15 |
EP0394614B1 (en) | 1996-04-10 |
JPH0320812A (en) | 1991-01-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2007411C (en) | Advanced user interface | |
US5252951A (en) | Graphical user interface with gesture recognition in a multiapplication environment | |
EP0440812B1 (en) | System and apparatus for handwritten input information | |
US5936614A (en) | User defined keyboard entry system | |
US5148155A (en) | Computer with tablet input to standard programs | |
EP0557284B1 (en) | Computer with separate display plane and user interface processor | |
US7904823B2 (en) | Transparent windows methods and apparatus therefor | |
US5805167A (en) | Popup menus with directional gestures | |
AU657499B2 (en) | User interface having simulated devices | |
US20060119588A1 (en) | Apparatus and method of processing information input using a touchpad | |
US20030214531A1 (en) | Ink input mechanisms | |
JPH0778782B2 (en) | Interactive computer system and apparatus and method for adapting use environment thereof | |
JP2000207089A (en) | Method and device for displaying hypertext document | |
US20020190946A1 (en) | Pointing method | |
Mankoff et al. | OOPS: a toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces | |
JP2009527041A (en) | System and method for entering data into a computing system | |
US5361310A (en) | Handwriting entry processing system using keyboard and mouse interfaces requiring no modification of an application program | |
WO2021242651A2 (en) | A new composite computer keyboard | |
JPH09152951A (en) | Handwriting data transfer method and data transfer system | |
WO1998007087A1 (en) | Method and apparatus for efficient text entry and editing | |
JPS61255463A (en) | Information processing device and information processing system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
EEER | Examination request | ||
MKLA | Lapsed |