CA2210601C - Intelligent user assistance facility - Google Patents
Intelligent user assistance facility Download PDFInfo
- Publication number
- CA2210601C CA2210601C CA002210601A CA2210601A CA2210601C CA 2210601 C CA2210601 C CA 2210601C CA 002210601 A CA002210601 A CA 002210601A CA 2210601 A CA2210601 A CA 2210601A CA 2210601 C CA2210601 C CA 2210601C
- Authority
- CA
- Canada
- Prior art keywords
- user
- events
- event
- modeled
- assistance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/453—Help systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99943—Generating database or data structure, e.g. via user interface
Abstract
A general event composing and monitoring system that allows high-level events to be created from combinations of low-level events. An event specification tool allows for rapid development of a general event processor that creates high- level events from combinations of user actions. The event system, in combination with a reasoning system, is able to monitor and perform inference about several classes of events for a variety of purposes. The various classes of events include the current context, the state of key data structures in a program, general sequences of user inputs, including actions with a mouse-controlled cursor while interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information. Additionally, a method is provided for building an intelligent user interface system by constructing a reasoning model to compute the probability of alternative user's intentions, goals, or informational needs through analysis of information about a user's actions, program state, and words. The intelligent user interface system monitors user interaction with a software application and applies probabilistic reasoning to sense that the user may need assistance in using a particular feature or to accomplish a specific task. The intelligent user interface also accepts a free-text query from the user asking for help and combines the inference analysis of user actions and program state with an inference analysis of the free-text query. The inference system accesses a rich, updatable user profile system to continually check for competencies and changes assistance that is given based on user competence.
Description
Description INTELLIGENT USER ASSISTANCE FACILITY
Technical Field This invention relates to expert systems in data processing. In particular, this invention relates to a system and method for automated machine reasoning to provide assistance to software users and to optimize the functionality of computer systems and software by performing inference about users needs and preferences in the operation of software systems or applications.
Backuround of the Invention Although significant advances in graphical use: interfaces have made software applications more accessible and productive for personal computer users, the 1 ~ increased ease of use has fostered a demand to include more and more sophisticated features. When first exposed to a complicated software application, a new user can be confused by the overwhelming number and complexity of features accessible through the menus and toolbars of the user interface. It is not uncommon for a user unfamiliar with the software features to resort to menu surfing, rapidly svritching from menu item to menu item, in the hope of discovering how to effect the desired feature.
Even when users know one or more ways to perform a task in a software application, they may not realize an efficient approach. Also, user's may know how to perform tasks, yet instead of performing the task themselves, they would rather have a system proactively determine their needs and perform or offer to perform operations, such as launching applications that will soon be needed, exiting applications, and prefetching files or information from distant servers to make them available more efficiently when they are requested.
Determining in an automated manner the best actions to perform or the best information to provide users while they work with software requires the development of computational methods that operate in conjunction with software
Technical Field This invention relates to expert systems in data processing. In particular, this invention relates to a system and method for automated machine reasoning to provide assistance to software users and to optimize the functionality of computer systems and software by performing inference about users needs and preferences in the operation of software systems or applications.
Backuround of the Invention Although significant advances in graphical use: interfaces have made software applications more accessible and productive for personal computer users, the 1 ~ increased ease of use has fostered a demand to include more and more sophisticated features. When first exposed to a complicated software application, a new user can be confused by the overwhelming number and complexity of features accessible through the menus and toolbars of the user interface. It is not uncommon for a user unfamiliar with the software features to resort to menu surfing, rapidly svritching from menu item to menu item, in the hope of discovering how to effect the desired feature.
Even when users know one or more ways to perform a task in a software application, they may not realize an efficient approach. Also, user's may know how to perform tasks, yet instead of performing the task themselves, they would rather have a system proactively determine their needs and perform or offer to perform operations, such as launching applications that will soon be needed, exiting applications, and prefetching files or information from distant servers to make them available more efficiently when they are requested.
Determining in an automated manner the best actions to perform or the best information to provide users while they work with software requires the development of computational methods that operate in conjunction with software
2 programs and that have the ability to identify a user's needs, intentions, or goals from aspects of a user's background and actions. Such methods may be derived from the construction and use of models that explicitly links the users' needs to their -backgrounds and actions.
There have been several studies of the use of models for reasoning about the intentions of people performing a variety of tasks, and for making available advice or assistance based on these models. Researchers studying the use of computers in education have attempted to construct programs that use models of users to determine the source of misunderstanding and the best way to tutor a student. These models can IO look at the answers users give to questions and other challenges. The Strategic Computing Initiative of the 1980s focused in part on methods for doing reasoning about a pilot's intentions from a diverse set of inputs, and for providing them with the best assistance with piloting and warfighting with fighter jets. Within the many such "pilot associates" projects, engineers attempted to design systems that could reason about the intentions of users. Typically, rule-based or logical approaches were used for determining or prioritizing options.
There has also been some work in the use of probabilistic models in the modeling of user's needs for such tasks as monitoring complex, time-critical applications. Some of the earliest discussion of probabilistic dependency models, with applications to enhancing the display of information, were in projects experimenting with the modeling of users for controlling the information displayed to aircraft pilots.
Probabilistic inference was explored as a means for controlling the information being displayed to pilots. Other work included using decision-theoretic methods to reason about the most important information to display to flight engineers at NASA
Mission Control Center. The methods consider context, user expertise level, and telemetry from the Space Shuttle to make these decisions, and take advantage of time-criticality and probabilistic models of user beliefs and intentions. Other related work has explored the best kind of information and explanations to display to pilots given various tradeoffs in display. In other work, researchers have explored the use of probabilistic models in
There have been several studies of the use of models for reasoning about the intentions of people performing a variety of tasks, and for making available advice or assistance based on these models. Researchers studying the use of computers in education have attempted to construct programs that use models of users to determine the source of misunderstanding and the best way to tutor a student. These models can IO look at the answers users give to questions and other challenges. The Strategic Computing Initiative of the 1980s focused in part on methods for doing reasoning about a pilot's intentions from a diverse set of inputs, and for providing them with the best assistance with piloting and warfighting with fighter jets. Within the many such "pilot associates" projects, engineers attempted to design systems that could reason about the intentions of users. Typically, rule-based or logical approaches were used for determining or prioritizing options.
There has also been some work in the use of probabilistic models in the modeling of user's needs for such tasks as monitoring complex, time-critical applications. Some of the earliest discussion of probabilistic dependency models, with applications to enhancing the display of information, were in projects experimenting with the modeling of users for controlling the information displayed to aircraft pilots.
Probabilistic inference was explored as a means for controlling the information being displayed to pilots. Other work included using decision-theoretic methods to reason about the most important information to display to flight engineers at NASA
Mission Control Center. The methods consider context, user expertise level, and telemetry from the Space Shuttle to make these decisions, and take advantage of time-criticality and probabilistic models of user beliefs and intentions. Other related work has explored the best kind of information and explanations to display to pilots given various tradeoffs in display. In other work, researchers have explored the use of probabilistic models in
3 assisting computer users working with command line input systems of a computer operating system.
There have been several projects focusing on the use of sets of logical rules and heuristics based on logical rules far providing users with assistance based on the context and activity. For instance, the common context-sensitive help that is made available in applications written for the ivlicrosoft Windows operating system is accessed by selecting a graphical object and clicking a standard help command (e.g., a function key on the keyboard). Help information is then provided which describes the functionality of the graphical object such a toolbar. Another example is the TipWizard system in Microsoft Excel products which employs a large set of logical rules that continue to scan for sequences of user events and provide assistance to users on performing tasks more efficiently. In work at Apple Computer described by U.S.
Patent No. 5,390,281 to Luciw et al., sets of rules we used for providing assistance to users on tasks by employing the heuristic of comparing the number of conditions for a rule to fire with the specific number of conditions that are seen in a particular situation and for interpreting this fraction as a strength associated with the action indicated by the rule.
There has also been work on more sophisticated probabilistic models that consider the uncertainty in user goals and needs for software assistance.
The Answer Wizard feature of Microsoft Office products on the use of probabilistic models 21) for interpreting the words in a usez's free-text query with a focus on assisting users with using computer software applications. In this work, the uncertain relationship between sets of related words that might be seen in a query and various kinds of assistance are modeled with probability. This work has been described in U.S. Patent Application No.
08/400,797 (now issued as U.S. Patent No. 5,ti94,559).
2a However, none of these former efforts have employed sophisticated methods for reasoning under uncertainty about a user's needs for assistance by considering multiple pieces c~f evidence, including information about a user's background and a user's activities that are autonomously sensed by the program, and, then, have offered to provide rei~:vant assistance to a user based on this reasoning. Nor 3CI have these systems considered nnor~itorina ~.iser interaction with a software program so
There have been several projects focusing on the use of sets of logical rules and heuristics based on logical rules far providing users with assistance based on the context and activity. For instance, the common context-sensitive help that is made available in applications written for the ivlicrosoft Windows operating system is accessed by selecting a graphical object and clicking a standard help command (e.g., a function key on the keyboard). Help information is then provided which describes the functionality of the graphical object such a toolbar. Another example is the TipWizard system in Microsoft Excel products which employs a large set of logical rules that continue to scan for sequences of user events and provide assistance to users on performing tasks more efficiently. In work at Apple Computer described by U.S.
Patent No. 5,390,281 to Luciw et al., sets of rules we used for providing assistance to users on tasks by employing the heuristic of comparing the number of conditions for a rule to fire with the specific number of conditions that are seen in a particular situation and for interpreting this fraction as a strength associated with the action indicated by the rule.
There has also been work on more sophisticated probabilistic models that consider the uncertainty in user goals and needs for software assistance.
The Answer Wizard feature of Microsoft Office products on the use of probabilistic models 21) for interpreting the words in a usez's free-text query with a focus on assisting users with using computer software applications. In this work, the uncertain relationship between sets of related words that might be seen in a query and various kinds of assistance are modeled with probability. This work has been described in U.S. Patent Application No.
08/400,797 (now issued as U.S. Patent No. 5,ti94,559).
2a However, none of these former efforts have employed sophisticated methods for reasoning under uncertainty about a user's needs for assistance by considering multiple pieces c~f evidence, including information about a user's background and a user's activities that are autonomously sensed by the program, and, then, have offered to provide rei~:vant assistance to a user based on this reasoning. Nor 3CI have these systems considered nnor~itorina ~.iser interaction with a software program so
4 that, when a user does inquire about help directly with free-text queries, the system combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context.
What is desirable is an intelligent user assistance facility that autonomously senses that the user may need assistance in using a particular feature or to accomplish a specific task, and that offers to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities. What is also desirable is an intelligent user assistance facility that combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. This invention solves the problem of autonomously sensing that a user may need assistance in using a particular feature or to accomplish a specific task, and offering to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities.
This invention also solves the problem of combining the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. In addition, the invention is able to monitor and perform inference about several classes of events, including the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information.
Summary of Invention The present invention provides a method of building an intelligent user assistance facility for a software program to assist a user in operation of the software program. Functionality of the software program is identified for providing intelligent user assistance. A reasoning model is constructed for the identified program functionality for the task of providing intelligent user assistance. The reasoning model computes the probability of alternative user's intentions, goals, or informational needs through analysis of information about a user's actions, program state, and words.
Modeled events for the identified program functionality are defined for use by the reasoning model to provide intelligent user assistance. Atomic user interface interactions are identified that sigaify the occurrence of modeled events. .
What is desirable is an intelligent user assistance facility that autonomously senses that the user may need assistance in using a particular feature or to accomplish a specific task, and that offers to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities. What is also desirable is an intelligent user assistance facility that combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. This invention solves the problem of autonomously sensing that a user may need assistance in using a particular feature or to accomplish a specific task, and offering to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities.
This invention also solves the problem of combining the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. In addition, the invention is able to monitor and perform inference about several classes of events, including the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information.
Summary of Invention The present invention provides a method of building an intelligent user assistance facility for a software program to assist a user in operation of the software program. Functionality of the software program is identified for providing intelligent user assistance. A reasoning model is constructed for the identified program functionality for the task of providing intelligent user assistance. The reasoning model computes the probability of alternative user's intentions, goals, or informational needs through analysis of information about a user's actions, program state, and words.
Modeled events for the identified program functionality are defined for use by the reasoning model to provide intelligent user assistance. Atomic user interface interactions are identified that sigaify the occurrence of modeled events. .
5 The present invention also includes a general event composing and monitoring system that allows high-level events to be created from combinations of low-level events. The event system, in combination with a reasoning system, is able to monitor and perform inference about several classes of events for a variety of purposes.
The various classes of events include the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information. The present invention additionally provides a system and method to monitor user interaction with a software program and to apply probabilistic reasoning to sense that the user may need assistance in using a particular feature or to accomplish a specific task. The system continues to examine a user's actions and, from this information, generates probabilities about user needs and goals.
In an exemplary embodiment, the likelihoods that various text-based help topics or demonstrations of various software tasks are relevant are computed and the assistance is made available to users. Probabilities are computed about relevant help topics as well as about the probability that a user would desire help. In this embodiment, the invention computes the probabilities that a user needs alternate forms of help, and uses this information to prioritize a list of help topics, when help is requested. When help has not been requested, the system uses its inference to 2~ determine when a user might desire assistance, and comes forward autonomously, depending on the computed importance of coming forward and the level at which a user sets a threshold. Whenever a user asks directly for help with free-text queries, this embodiment combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context.
The various classes of events include the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information. The present invention additionally provides a system and method to monitor user interaction with a software program and to apply probabilistic reasoning to sense that the user may need assistance in using a particular feature or to accomplish a specific task. The system continues to examine a user's actions and, from this information, generates probabilities about user needs and goals.
In an exemplary embodiment, the likelihoods that various text-based help topics or demonstrations of various software tasks are relevant are computed and the assistance is made available to users. Probabilities are computed about relevant help topics as well as about the probability that a user would desire help. In this embodiment, the invention computes the probabilities that a user needs alternate forms of help, and uses this information to prioritize a list of help topics, when help is requested. When help has not been requested, the system uses its inference to 2~ determine when a user might desire assistance, and comes forward autonomously, depending on the computed importance of coming forward and the level at which a user sets a threshold. Whenever a user asks directly for help with free-text queries, this embodiment combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context.
6 The invention includes a rich, updatable user profiling system. In the help-topic embodiment, the inference system accesses the user profile system to continually check for competencies and changes assistance that is given based on user competence.
The present invention also includes a novel inference system with an annotated Bayesian network to include special temporal reasoning procedures.
Temporal reasoning mechanisms are used to consider the changes in relevance of events with current needs and goals of users with time passing after an event has occurred.
The inference engine includes a new specific approximation procedure which uses a single explicit Bayesian network knowledge base, but changes the likelihood information in the network based on the distance in the past that an observation was made.
The system also allows users to control the thresholds that must be reached before users are offered assistance, allowing them to tailor the system's 1 S behavior to their own personal preference about being distracted with assistance.
Finally, the methods have been integrated with multiple graphical user interfaces, including a social-user interface in the form of an animated character.
Brief Description of the Drawing Figure 1 is a block diagram of a computer system.
Figure 2 is a flowchart for building an intelligent user assistance facility for a software program.
Figure 3 is a diagram illustrating a simple two variable Bayesian network.
Figure 4 is a block diagram of an Intelligent User Assistance Facility.
Figure 5 is a block diagram of an instrurnented application program.
Figure 6 is a block diagram of an instrumented spreadsheet application program for use with the Intelligent User Assistance Facility.
Figure 7 is a block diagram of an Event Processor.
The present invention also includes a novel inference system with an annotated Bayesian network to include special temporal reasoning procedures.
Temporal reasoning mechanisms are used to consider the changes in relevance of events with current needs and goals of users with time passing after an event has occurred.
The inference engine includes a new specific approximation procedure which uses a single explicit Bayesian network knowledge base, but changes the likelihood information in the network based on the distance in the past that an observation was made.
The system also allows users to control the thresholds that must be reached before users are offered assistance, allowing them to tailor the system's 1 S behavior to their own personal preference about being distracted with assistance.
Finally, the methods have been integrated with multiple graphical user interfaces, including a social-user interface in the form of an animated character.
Brief Description of the Drawing Figure 1 is a block diagram of a computer system.
Figure 2 is a flowchart for building an intelligent user assistance facility for a software program.
Figure 3 is a diagram illustrating a simple two variable Bayesian network.
Figure 4 is a block diagram of an Intelligent User Assistance Facility.
Figure 5 is a block diagram of an instrurnented application program.
Figure 6 is a block diagram of an instrumented spreadsheet application program for use with the Intelligent User Assistance Facility.
Figure 7 is a block diagram of an Event Processor.
7 Figure 8 is a block diagram of an Event System Specification Tool for creating a modeled event composer.
Figure 9 is a block diagram of a User Profile System.
Figure 10 is a flowchart of the Event Status Update Routine of the User Profile System of Figure 9.
Figure 11 is a flowchart of the User Profile Access Routine of the User Profile System of Figure 9.
Figure 12 is a flowchart of the Customized Tutorial Routine of the User Profile System of Figure 9.
Figure 13 is a block diagram of the Inference System of the present invention.
Figure 14 is a Bayesian network diagram illustrating the overall influence relationship for any software program using the IUAF of the present invention to provide user assistance.
Figure 15 is a diagram illustrating a portion of a Bayesian network representing some probabilistic dependencies over time.
Figure 16 is a flowchart of the value-of information procedure according to the principles of the invention for acquiring user information.
Figures 17 - 22 illustrate screen outputs of the IUAF made in accordance with the principles of the present invention in an exemplary embodiment of a spreadsheet application.
Figure 23 illustrates screen output of the IUAF according to the principles of the present invention in an exemplary embodiment of a spreadsheet application for displaying a summary of a custom-tailored tutorial.
Figure 24 is a portion of a Bayesian network for ILTAF made in accordance with the principles of the present invention.
Figure 25 is a portion of a Bayesian network for a spreadsheet application illustrating modeled and profiled events for charting data of a spreadsheet.
Figure 9 is a block diagram of a User Profile System.
Figure 10 is a flowchart of the Event Status Update Routine of the User Profile System of Figure 9.
Figure 11 is a flowchart of the User Profile Access Routine of the User Profile System of Figure 9.
Figure 12 is a flowchart of the Customized Tutorial Routine of the User Profile System of Figure 9.
Figure 13 is a block diagram of the Inference System of the present invention.
Figure 14 is a Bayesian network diagram illustrating the overall influence relationship for any software program using the IUAF of the present invention to provide user assistance.
Figure 15 is a diagram illustrating a portion of a Bayesian network representing some probabilistic dependencies over time.
Figure 16 is a flowchart of the value-of information procedure according to the principles of the invention for acquiring user information.
Figures 17 - 22 illustrate screen outputs of the IUAF made in accordance with the principles of the present invention in an exemplary embodiment of a spreadsheet application.
Figure 23 illustrates screen output of the IUAF according to the principles of the present invention in an exemplary embodiment of a spreadsheet application for displaying a summary of a custom-tailored tutorial.
Figure 24 is a portion of a Bayesian network for ILTAF made in accordance with the principles of the present invention.
Figure 25 is a portion of a Bayesian network for a spreadsheet application illustrating modeled and profiled events for charting data of a spreadsheet.
8 Figure 26 is a Bayesi2n network annotated with temporal dynamics information made in accordance with the principles of the present invention in an exemplary embodiment of a spreadsheet application.
Figures 27 is a portion of a Bayesian network illustrating a model in ' , S accordance with the principles of the present invention for assisting with exiting in a multiple application setting.
Figures 28 - 30 is a portion of a Bayesian network illustrating reasoning for assisting with exiting in a multiple application setting.
Figure 31 is a diagram illustrating the control modes of the IUAF
Controller.
Figure 32 is a method according to the present invention for combining inference analysis of user actions and program state with inference analysis of free-text query.
Figure 33 is a flowchart of the autonomous assistance procedure of the present invention.
Figure 34~is an illustration of a social interface providing intelligent user assistance for a spreadsheet program.
Figure 35 is a block diagram of the IUAF with speech recognition system showing use of inference results to enhance speech recognition.
Detailed Descn_ption of the Invention The present invention provides a general event composing and monitoring system that allows high-level events to be created from combinations of low-level events. The event system, in combination with a reasoning system, is able to monitor and perform inference about several classes of events for a variety of purposes.
The various classes of events include the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries foi assistance, visual information about users, such as gaze and gesture information, and speech information. The present invention also employs a novel event specification
Figures 27 is a portion of a Bayesian network illustrating a model in ' , S accordance with the principles of the present invention for assisting with exiting in a multiple application setting.
Figures 28 - 30 is a portion of a Bayesian network illustrating reasoning for assisting with exiting in a multiple application setting.
Figure 31 is a diagram illustrating the control modes of the IUAF
Controller.
Figure 32 is a method according to the present invention for combining inference analysis of user actions and program state with inference analysis of free-text query.
Figure 33 is a flowchart of the autonomous assistance procedure of the present invention.
Figure 34~is an illustration of a social interface providing intelligent user assistance for a spreadsheet program.
Figure 35 is a block diagram of the IUAF with speech recognition system showing use of inference results to enhance speech recognition.
Detailed Descn_ption of the Invention The present invention provides a general event composing and monitoring system that allows high-level events to be created from combinations of low-level events. The event system, in combination with a reasoning system, is able to monitor and perform inference about several classes of events for a variety of purposes.
The various classes of events include the state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in the normal course of interacting with a graphical user interface, words typed in free-text queries foi assistance, visual information about users, such as gaze and gesture information, and speech information. The present invention also employs a novel event specification
9 tool that allows for rapid development of a general event processor that creates high-level events from combinations of user actions.
The present invention also includes a novel inference system with an annotated Bayesian network to include special temporal reasoning procedures.
The inference engine includes a new specific approximation procedure which uses a single explicit Bayesian network knowledge base, but changes the likelihood information in the network based on the distance in the past that an observation was made.
The knowledge base includes special annotations with temporal dynamics information and the inference engine includes special procedures to handle changes in relevance with the passing of time.
The present invention also provides an intelligent user assistance facility and method for use of probabilistic and logical reasoning to make decisions about the best way to provide assistance to a computer software user. The present invention additionally provides a method of building an intelligent user assistance facility by 1 ~ constructing a reasoning model to compute the probability of a user's intentions, goals, or informational needs through analysis of information about a user's actions, program state, and words. An exemplary embodiment of the present invention monitors user interaction with a software application and applies probabilistic reasoning to sense that the user may need assistance in using a particular feature or to accomplish a specific task. This embodiment of the present invention additionally accepts a free-text query from the user asking for help and combines the inference analysis of user actions and program state with an inference analysis of the free-text query. This embodiment of the present invention also incorporates a novel user profile system that stores information about the competence of users that may be established initially by user input or dialogue with the user and that is updated with the results of ongoing user activity.
This persistent user profile provides user competency information during inference analysis and further enables customization of tutorial information for the user after an application session concludes.
Furthermore, this embodiment includes automated assistance reasoning to determine when a user might want help, and comes forward autonomously, IO
depending on the computed importance of coming forward and the level at which a user sets a threshold. When a user does inquire about help directly with free-text queries, the system also combines the analysis of the words in the free-text query with the ongoing analysis of user actions.
S Figure I is a block diagram of a computer system 10 that is suitable for practicing an exemplary embodiment of the present invention. Those skilled in the art will appreciate that the computer of system 10 depicted in Figure 1 is intended to be merely illustrative and that the present invention may be practiced with other computer system configurations, including distributed systems and multiprocessor systems, and handheld devices. The computer system 10 includes a processor 12 and an input/output system 14. The input devices may be, for example, a camera I5, a keyboard 16, a mouse 18, a microphone 20, a pointing device or other input device. The output devices may be, for example, a display screen 22, a printer 24, a speaker 26 or other output device. The computer system IO may include a speech interface 27 that interfaces a speech system with the microphone 20 and speaker 26. The computer system 10 may include a visual recognition and backing interface 29 that interfaces a visual recognition and tracking system with the camera I ~ or other specialized head gear with an infrared reflector or 3D ultrasound sensors. The computer system 10 may also include a network interface 28 that interfaces the computer with a network 30 that may be either a local area network or a wide area network such as the Internet. The computer system additionally includes a system memory 32 and persistent storage 34 that contain application programs and the intelligent user assistance facility. The persistent storage may be a diskette, CDROM, hard-drive, or firmware. The computer system 10 includes an operating system 36 and several application programs such as a spreadsheet application program 38, a word processing application program 40, or other application program 42. A system bus 44 interconnects the processor 12 , system memory 32, persistent storage 34, aad inputloutput system 14. Those skilled in the art will appreciate that the intelligent user assistance facility may be integrated directly into the application programs or may be a stand alone facility that is part of a system library or operating system.
Overview of the Intelligent User Assistance Facilitv~IUAF~
Although software usability studies can pinpoint user interface problems for accessing complex features and can simplify and make the user interface more intuitive for accessing the features, improvements to the user interface cannot hide the multitude of complex features or simplify the functionality of a complex feature for the inexperienced user without reducing the number of features exposed through the interface and/or exposing only simplified versions of the full feature.
One by-product of software program usability studies can be the identification of user behavior when a user experiences difficulties in using basic functionality of a particular software program. This invention provides a process for making an intelligent user assistance facility for a particular software program by observing and understanding user behavior when a user experiences such difficulties or when a user might benefit by providing automated actions Figure 2 is a flowchart of the method for making an intelligent user assistance facility for a software program. The first step 50 of the process is to identify the basic functionality of a particular software program in which users experience difficulties, or could benefit from automated assistance, and the user behavior exhibited by the user when the user is experiencing those difficulties or desire for assistance. One way of accomplishing this step is to observe and record user behavior when a user experiences difficulties during usability studies for that particular software program.
For example, during a usability study for a spreadsheet program, a user interacting with a spreadsheet may be asked to update a graphical chart (e.g., a bar chart) of data on a spreadsheet. The user may have no experience in changing or modifying bar charts once they have been created. The user subsequently exhibits the following behavior:
First the user selects the bar chart by placing the mouse pointer on the chart and double clicking the mouse button. Then the user pauses to dwell on the chart for some period of time while introspecting or searching for the next step. Observation of this activity could serve as an indication that the user is experiencing difficulty in using the chart feature of the spreadsheet.
Once the diffculties with using basic functionality are identified, the next step 52 is to build a model for the task of providing intelligent assistance for the basic functionality identified. One type of model is an network of interdependent variables such as a Bayesian network that can represent probabilistic relationships between observable evidence, intermediate variables, and hypotheses of interest. A
Bayesian network is a directed acyclic graph in which the nodes represent random variables and arcs between nodes represent probabilistic dependencies among those variables. A Bayesian network represents a joint-probability distribution for a set of variables. Computational procedures, referred to as Bayesian inference algorithms have been developed to operate on Bayesian networks. These algorithms update the probability distributions over states of variables in a Bayesian network given changes that are made in values of states. An example of state changes are new values assigned to states of variables representing observable states of the world, following the observation of these states in the world. A description of Bayesian networks can be found in "Probabilistic Reasoning In Intelligent Systems", by Judea Pearl, lMorgan Kaufrnann Publishing Inc., San Mateo, 1988.
Some of the nodes are variables that represent evidence that may be observed. Other nodes represent unobservable or difficult to observe hypotheses of interest that may be used in automated decision making. A simple two-variable Bayesian network is displayed in Figure 3. The network represents the dependency between a user being in a state of needing assistance 68 and the user activity of multiple menu exploration 69 or "menu surfing" without making progress. Menu surfing can be sensed, but it cannot be easily observed whether a user needs assistance. In modeling user difficulties with complex software features, evidence nodes may represent user behavior exhibited while performing actions toward certain goals or experiencing difficulty with a software program feature, and hypothesis nodes may represent the goals of the user or the particular assistance needed by the user to complete a task using that feature. For example, while creating a chart the user pauses after selecting the create chart command. An evidence node variable would represent the fact that the user paused after selecting a command. An hypothesis node variable may represent the probability of the hypothesis that the user needs assistance in creating a chart given the fact that the user paused after selecting the create chart command. Using a BayPsian network is one way of building such a model of uncertain relationships between user actions and hypotheses. Those skilled in the art will recognize that other models can be , used such as decision trees or rule-based reasoning.
For any of these models, the next step 54 is to define evidence variables and corresponding modeled events that can discriminate among alternate di~culties and needs for assistance that a person may have in using the basic functionality of the software program, and that can be sensed during the user's interaction with software. A
key task required in building the Bayesian network model for the intelligent assistance facility is defining the evidence variables that will be observed, and noting the probabilistic relationships of these variables to variables representing hypotheses of interest. These events are those that are useful in detecting and discriminating among alternate difficulties and needs a person may have in using complex software features in general and, in particular, specific complex software features of a software application.
Classes of observation that are valuable for identifying when and what kind of assistance users need include evidence of search (e.g., exploration of multiple menus), of introspection, (e.g., sudden pause or slowing of command stream), of focus of attention (e.g., selecting and pausing on objects), of undesired effects of action (e.g., command/undo, dialogue opened and canceled without action), of ine~cient command sequences, the structural and semantic content of file being worked on, and goal-specific sequences of actions. Also, program state and program context, user competencies, and the history of information reviewed by the user are useful.
In developing the Bayesian network and specifying these observable variables, it is important to consider the nature and definition of the intelligent user assistance facility which will be harnessed to detect user actions.
Consideration must be given to the basic events that can be made available from the software system or application. Modeled events detected as occurring by the system are designed to correspond with each observable variable. The modeled events must be defined in terms of the basic, or atomic events that are generated by the software program.
At run-time, the modeled events corresponding to the observable evidence variables that are modeled in the Bayesian network are provided to the inference system which sets the evidence variables to specific values. The intelligent user assistance facility has the ability to consider the recent occurrence of multiple modeled events in parallel and to consider the time these events occurred, and to use this monitored information to update the state of observed variables in the Bayesian network. The Bayesian network with its associated dynamically changing settings with regard to the status and timing of observed variables is analyzed in the inference system which generates updated likelihoods on different hypotheses of interest. These probabilities are used in making decisions about action and display of information to users.
Upon completing the task of defining evidence variables in a Bayesian network and their associated modeled events, the next step 56 is to identify atomic events that indicate the occurrence of each modeled event. Modeled events can be defined by the occurrence of a single atomic event, associated with a user action or state of the program or may represent a sequence of one or more atomic events. For example, the two atomic events of (I) a user selecting a bar chart and then (2) pausing to dwell on the chart for more than 10 seconds can be composed into a single modeled event that corresponds to an evidence variable in a Bayesian network. Both of these events may be combined to form a modeled event, Sel Chart Pause, that indicates an occurrence of a user having difficulty updating a chart with data on a spreadsheet.
Variables in the Bayesian network may be binary and have just two states (e.g., absent and present), or may have multiple states. In cases where there are multiple values, the variable is set to one of the states corresponding to the observation.
Binary observational variables are described, but the techniques also apply to variables with greater numbers of values. When the modeled event is observed and is passed to the inference system, the evidence variable corresponding to this event is set to "present" at the time noted for when the event occurred. Whenever probabilistic inference is performed on the Bayesian network, the status of all variables that could be set by modeled events are considered. As a result of inference we may see changes in l~
hypothesis variables of interest. For example, given the occurrence of Sel Chart Pause, and other observations that update the probability that -a user is having di~culty with chart-related tasks, the probability assigned to the hypothesis that a user is having di~culty updating the chart with data on a spreadsheet may increase.
Another example of a modeled event is the action of a user selecting the Undo command following any one of a set of chart-related commands. The occurrence of one of any atomic or modeled events in a set followed by an undo command would be composed as a Chart Cmd Undo modeled event. When this modeled event sets the corresponding evidence node in the Bayesian net, we might see an increase in the probability assigned to the hypothesis that the user may be having difficulty updating chart with data on a spreadsheet Continuing the example above, atomic events for the modeled event, Sel Chart Pause, are (1) a user selects a bar chart, followed by (2) some minimal prespecified period of time (e.g., 10 seconds) in which there is a dwell, indicated a cessation of user activity.
Atomic events may serve directly as modeled events. For example, DeI Chart is an atomic event representing the deletion of a chart that is also modeled event, corresponding to the observation variable in the Bayesian network.
Modeled events also include contextual information about the state of data structures in an application. For example, Chart View, means that a chart is present in the current display. This will increase the probability that charting assistance is relevant to the user.
Words appearing in a free-text query are also atomic events. Higher-level, modeled events may be defined as the appearance of one of a predefined set of words in a user's query. If a usez's query is, "How do I change the colors in this chart?,"
the modeled event, that one of a set of words (e.g., in this case, a set containing as elements the words "modify," "cony ert," and "update's has been noted in the query, is passed to the inference system. Other important words in the query, such as "color" and "chart", are also detected as modeled events when they are noted to be in sets of previously modeled sets of similar words. These modeled events are analyzed in the same way as "convert."
The class of modeled events that are persistent across multiple uses of a software program are defined as profile information. These modeled events include events that indicate user competency or lack of competency and are stored in the user profile database. Modeled events stored in the user profile provide additional -information about the user, such as the user's previous experience with particular application features or accomplishment of specific tasks, or review of tutorial material in the past. These events may correspond to variables represented in the Bayesian network and used by the inference system to adjust the evaluation of the hypotheses.
Other types of atomic events may include information about the presence or absence of user in front of the computer, as gleaned by proximity detectors, by visual features of the user, including information about the focus of the gaze of the user, expressions of the user, or gestures of the user, as identified by a video camera and vision recognition and tracking system, and speech utterances of a user as detected and processed by a speech understanding system.
After atomic events and their appropriate combination are identified for each of the modeled events, the next step 58 is to construct an event monitor for the software program that will survey user interface actions to extract the atomic events for input to inference analysis. The event monitor constructed is integrated with the software application to instrument that software application at step 60 in making an intelligent user assistance facility. Those skilled in the art will appreciate that the event monitor may be integrated within the application software, or may be a stand alone component that is part of a system library or operating system.
To complete the intelligent user assistance facility, an event processor is to compose modeled events from atomic user-interface actions (step 62). Next, a knowledge base is constructed from the model (step 64). The last step 66 is to construct an inference engine from the model which uses the knowledge base to determine suggested user assistance for the occurrence of modeled events. A Bayesian network with inference procedures serves both as a knowledge base and inference engine.
Figure 4 is a functional block diagram of the components of an embodiment of an Intelligent User Assistance Facility constructed by using the method described above. The Intelligent User Assistance Facility includes a software program 72 that has been instnlmented to monitor actions or events initiated by a user who is interacting with the software program. Periodically, these actions or events are scrutinized by an event processor 74 to discover whether any of them, either ~
.
individually or in combination, represent observable variables in the Bayesian network.
Modeled events are analyzed by an inference system 76 to form and evaluate multiple hypotheses of what assistance the user may need. These events can lead to increases or decreases in the probability that a user is experiencing a problem in using a particular application feature or in accomplishing a specific task.
The inference system 76 also accesses the user profile system 78 to check for competencies and changes assistance that is given based on user competence.
User's background, successful completion of key tasks indicative of competence in particular areas, and previous help reviewed by the user are some of the variables stored in the user profile system 78.
The IUAF controller 79 initiates a cycle of inference analysis by the inference system 76 based upon the setting of its control mode. An inference cycle may be initiated whenever a user asks for help, or whenever one or more of any number of special events have occurred, or may simply occur on a regular clocked cycle, or for any combination of these modes. When the control mode is set to trigger an inference cycle upon occurrence of a special event or the control mode is set to occur on a regular clocked cycle, the inference system 76 will autonomously offer help to the user whenever the computed importance of coming forward exceeds the autonomous assistance threshold set by the user. When the control mode is set to initiate an inference cycle whenever the user asks for help, the inference analysis of user actions and program context is combined with an analysis of the words in a free-text query whenever the user submits a free-text query.
Those skilled in the art will appreciate that the functional blocks illustrated in the diagram may be implemented as separate components or several or all of the functional blocks may be implemented within a single component. For example, a separate library may be implemented that contains the reasoning system functionality and the event processor functionality. In such an implementation, the reasoning system may query the instrumented program periodically for all of the atomic events since the last query or may simply ask for a specific number of the last occurring events, and then the reasoning system may access the modeled event database for that particular software program to determining whether any of the atomic events are modeled events for input to inference analysis. Those skilled in the art will also appreciate that the inference analysis may be implemented to include only the ongoing analysis of user actions and program context except when a user asks directly for help with a free-text query. In that case, the inference analysis may be implemented to include only the analysis of the words in the free-text query whenever a user asks directly for help with a free-text query. Those skilled in the art will also appreciate that a separate analyses could be undertaken for different classes of modeled events, such as for words in a query and for actions in the interface, and the results of these separate analyses could be combined with the ability to control the weight given to the results of the different analyses.
Instrumented Program Any software program or library may be an instrumented program, including operating system programs and application programs. As an illustration of an instrumented program, an application program will be described. A typical software application contains a user interface and application procedures. As a user interacts with a software application using input devices such as depressing a key of the keyboard or clicking the button of a mouse, the operating system translates those user actions into events and sends them to the application program. The user interface component of the application program processes those events and forwards them to the appropriate application procedures for responding to those events. The event monitor constructed for the Intelligent user assistance facility for a particular application may be integrated within the application software as illustrated by Figure 5. In this configuration, the event monitor 84 watches for events passing from the user interface 82 component to application procedures 86 and forwards them to the event processor through a call interface to the event processor. It also may be implemented as a stand alone component that is part of a system or application library which makes periodic calls upon the application program to send it all events since the last period. or it may simply request that the application program send only a specific number of the last occurring events. Alternatively, it may be incorporated as part of the operating system in which case the event monitor screens events passing from the operating system to the application program and forwards them to the event processor.
Figure 6 is a functional block diagram of an instrumented spreadsheet application program as in the help-topic embodiment. As a user interacts with a spreadsheet application program using input devices, the operating system translates those user actions into events and sends them to the spreadsheet application program.
For example, the user may place the pointer of the mouse on a menu item and may then click on the button of mouse. The operating system will translate that user action into an event that the user has clicked on an application menu item and send the event to the spreadsheet application. The user interface 92 component of the instrumented spreadsheet application program 90 receives that event and forwards it to the appropriate spreadsheet application procedures 96 for processing that event.
The spreadsheet events monitor 94, watching for events passing from the user interface 92 component to application procedures 96, will see that event and forward a copy to the event processor.
Event Processor The event processor is a general monitoring and event composing system that allows high-level events to be created from combinations of low-level events. The event processor does not need to include sophisticated reasoning capability.
However, those skilled in the art will also appreciate that a variety of deterministic and probabilistic event preprocessing methods can be employed to work with arbitrary abstractions or clusters of events and to transform events into numeric information with a variety of mathematical functions, including those that might be directly interpreted as likelihoods.
Since the Event Processor produces a set of logical events, any reasoning engine, rule-based or probabilistic, can utilize its output. Figure 7 is a block diagram illustrating the components of the event processor. Atomic events or user input device interactions are sent by an events source 100 such as an instrumented program to the 5 event processor. The instrumented program may use a call interface that specifies the event information to the event processor. Upon receiving these events, the event processor may time-stamp each event and store them in local storage 102, such as system memory, in one of several kinds of data structures for storing information, including a database of records or finite circular queue ordered by the time stamp, or
The present invention also includes a novel inference system with an annotated Bayesian network to include special temporal reasoning procedures.
The inference engine includes a new specific approximation procedure which uses a single explicit Bayesian network knowledge base, but changes the likelihood information in the network based on the distance in the past that an observation was made.
The knowledge base includes special annotations with temporal dynamics information and the inference engine includes special procedures to handle changes in relevance with the passing of time.
The present invention also provides an intelligent user assistance facility and method for use of probabilistic and logical reasoning to make decisions about the best way to provide assistance to a computer software user. The present invention additionally provides a method of building an intelligent user assistance facility by 1 ~ constructing a reasoning model to compute the probability of a user's intentions, goals, or informational needs through analysis of information about a user's actions, program state, and words. An exemplary embodiment of the present invention monitors user interaction with a software application and applies probabilistic reasoning to sense that the user may need assistance in using a particular feature or to accomplish a specific task. This embodiment of the present invention additionally accepts a free-text query from the user asking for help and combines the inference analysis of user actions and program state with an inference analysis of the free-text query. This embodiment of the present invention also incorporates a novel user profile system that stores information about the competence of users that may be established initially by user input or dialogue with the user and that is updated with the results of ongoing user activity.
This persistent user profile provides user competency information during inference analysis and further enables customization of tutorial information for the user after an application session concludes.
Furthermore, this embodiment includes automated assistance reasoning to determine when a user might want help, and comes forward autonomously, IO
depending on the computed importance of coming forward and the level at which a user sets a threshold. When a user does inquire about help directly with free-text queries, the system also combines the analysis of the words in the free-text query with the ongoing analysis of user actions.
S Figure I is a block diagram of a computer system 10 that is suitable for practicing an exemplary embodiment of the present invention. Those skilled in the art will appreciate that the computer of system 10 depicted in Figure 1 is intended to be merely illustrative and that the present invention may be practiced with other computer system configurations, including distributed systems and multiprocessor systems, and handheld devices. The computer system 10 includes a processor 12 and an input/output system 14. The input devices may be, for example, a camera I5, a keyboard 16, a mouse 18, a microphone 20, a pointing device or other input device. The output devices may be, for example, a display screen 22, a printer 24, a speaker 26 or other output device. The computer system IO may include a speech interface 27 that interfaces a speech system with the microphone 20 and speaker 26. The computer system 10 may include a visual recognition and backing interface 29 that interfaces a visual recognition and tracking system with the camera I ~ or other specialized head gear with an infrared reflector or 3D ultrasound sensors. The computer system 10 may also include a network interface 28 that interfaces the computer with a network 30 that may be either a local area network or a wide area network such as the Internet. The computer system additionally includes a system memory 32 and persistent storage 34 that contain application programs and the intelligent user assistance facility. The persistent storage may be a diskette, CDROM, hard-drive, or firmware. The computer system 10 includes an operating system 36 and several application programs such as a spreadsheet application program 38, a word processing application program 40, or other application program 42. A system bus 44 interconnects the processor 12 , system memory 32, persistent storage 34, aad inputloutput system 14. Those skilled in the art will appreciate that the intelligent user assistance facility may be integrated directly into the application programs or may be a stand alone facility that is part of a system library or operating system.
Overview of the Intelligent User Assistance Facilitv~IUAF~
Although software usability studies can pinpoint user interface problems for accessing complex features and can simplify and make the user interface more intuitive for accessing the features, improvements to the user interface cannot hide the multitude of complex features or simplify the functionality of a complex feature for the inexperienced user without reducing the number of features exposed through the interface and/or exposing only simplified versions of the full feature.
One by-product of software program usability studies can be the identification of user behavior when a user experiences difficulties in using basic functionality of a particular software program. This invention provides a process for making an intelligent user assistance facility for a particular software program by observing and understanding user behavior when a user experiences such difficulties or when a user might benefit by providing automated actions Figure 2 is a flowchart of the method for making an intelligent user assistance facility for a software program. The first step 50 of the process is to identify the basic functionality of a particular software program in which users experience difficulties, or could benefit from automated assistance, and the user behavior exhibited by the user when the user is experiencing those difficulties or desire for assistance. One way of accomplishing this step is to observe and record user behavior when a user experiences difficulties during usability studies for that particular software program.
For example, during a usability study for a spreadsheet program, a user interacting with a spreadsheet may be asked to update a graphical chart (e.g., a bar chart) of data on a spreadsheet. The user may have no experience in changing or modifying bar charts once they have been created. The user subsequently exhibits the following behavior:
First the user selects the bar chart by placing the mouse pointer on the chart and double clicking the mouse button. Then the user pauses to dwell on the chart for some period of time while introspecting or searching for the next step. Observation of this activity could serve as an indication that the user is experiencing difficulty in using the chart feature of the spreadsheet.
Once the diffculties with using basic functionality are identified, the next step 52 is to build a model for the task of providing intelligent assistance for the basic functionality identified. One type of model is an network of interdependent variables such as a Bayesian network that can represent probabilistic relationships between observable evidence, intermediate variables, and hypotheses of interest. A
Bayesian network is a directed acyclic graph in which the nodes represent random variables and arcs between nodes represent probabilistic dependencies among those variables. A Bayesian network represents a joint-probability distribution for a set of variables. Computational procedures, referred to as Bayesian inference algorithms have been developed to operate on Bayesian networks. These algorithms update the probability distributions over states of variables in a Bayesian network given changes that are made in values of states. An example of state changes are new values assigned to states of variables representing observable states of the world, following the observation of these states in the world. A description of Bayesian networks can be found in "Probabilistic Reasoning In Intelligent Systems", by Judea Pearl, lMorgan Kaufrnann Publishing Inc., San Mateo, 1988.
Some of the nodes are variables that represent evidence that may be observed. Other nodes represent unobservable or difficult to observe hypotheses of interest that may be used in automated decision making. A simple two-variable Bayesian network is displayed in Figure 3. The network represents the dependency between a user being in a state of needing assistance 68 and the user activity of multiple menu exploration 69 or "menu surfing" without making progress. Menu surfing can be sensed, but it cannot be easily observed whether a user needs assistance. In modeling user difficulties with complex software features, evidence nodes may represent user behavior exhibited while performing actions toward certain goals or experiencing difficulty with a software program feature, and hypothesis nodes may represent the goals of the user or the particular assistance needed by the user to complete a task using that feature. For example, while creating a chart the user pauses after selecting the create chart command. An evidence node variable would represent the fact that the user paused after selecting a command. An hypothesis node variable may represent the probability of the hypothesis that the user needs assistance in creating a chart given the fact that the user paused after selecting the create chart command. Using a BayPsian network is one way of building such a model of uncertain relationships between user actions and hypotheses. Those skilled in the art will recognize that other models can be , used such as decision trees or rule-based reasoning.
For any of these models, the next step 54 is to define evidence variables and corresponding modeled events that can discriminate among alternate di~culties and needs for assistance that a person may have in using the basic functionality of the software program, and that can be sensed during the user's interaction with software. A
key task required in building the Bayesian network model for the intelligent assistance facility is defining the evidence variables that will be observed, and noting the probabilistic relationships of these variables to variables representing hypotheses of interest. These events are those that are useful in detecting and discriminating among alternate difficulties and needs a person may have in using complex software features in general and, in particular, specific complex software features of a software application.
Classes of observation that are valuable for identifying when and what kind of assistance users need include evidence of search (e.g., exploration of multiple menus), of introspection, (e.g., sudden pause or slowing of command stream), of focus of attention (e.g., selecting and pausing on objects), of undesired effects of action (e.g., command/undo, dialogue opened and canceled without action), of ine~cient command sequences, the structural and semantic content of file being worked on, and goal-specific sequences of actions. Also, program state and program context, user competencies, and the history of information reviewed by the user are useful.
In developing the Bayesian network and specifying these observable variables, it is important to consider the nature and definition of the intelligent user assistance facility which will be harnessed to detect user actions.
Consideration must be given to the basic events that can be made available from the software system or application. Modeled events detected as occurring by the system are designed to correspond with each observable variable. The modeled events must be defined in terms of the basic, or atomic events that are generated by the software program.
At run-time, the modeled events corresponding to the observable evidence variables that are modeled in the Bayesian network are provided to the inference system which sets the evidence variables to specific values. The intelligent user assistance facility has the ability to consider the recent occurrence of multiple modeled events in parallel and to consider the time these events occurred, and to use this monitored information to update the state of observed variables in the Bayesian network. The Bayesian network with its associated dynamically changing settings with regard to the status and timing of observed variables is analyzed in the inference system which generates updated likelihoods on different hypotheses of interest. These probabilities are used in making decisions about action and display of information to users.
Upon completing the task of defining evidence variables in a Bayesian network and their associated modeled events, the next step 56 is to identify atomic events that indicate the occurrence of each modeled event. Modeled events can be defined by the occurrence of a single atomic event, associated with a user action or state of the program or may represent a sequence of one or more atomic events. For example, the two atomic events of (I) a user selecting a bar chart and then (2) pausing to dwell on the chart for more than 10 seconds can be composed into a single modeled event that corresponds to an evidence variable in a Bayesian network. Both of these events may be combined to form a modeled event, Sel Chart Pause, that indicates an occurrence of a user having difficulty updating a chart with data on a spreadsheet.
Variables in the Bayesian network may be binary and have just two states (e.g., absent and present), or may have multiple states. In cases where there are multiple values, the variable is set to one of the states corresponding to the observation.
Binary observational variables are described, but the techniques also apply to variables with greater numbers of values. When the modeled event is observed and is passed to the inference system, the evidence variable corresponding to this event is set to "present" at the time noted for when the event occurred. Whenever probabilistic inference is performed on the Bayesian network, the status of all variables that could be set by modeled events are considered. As a result of inference we may see changes in l~
hypothesis variables of interest. For example, given the occurrence of Sel Chart Pause, and other observations that update the probability that -a user is having di~culty with chart-related tasks, the probability assigned to the hypothesis that a user is having di~culty updating the chart with data on a spreadsheet may increase.
Another example of a modeled event is the action of a user selecting the Undo command following any one of a set of chart-related commands. The occurrence of one of any atomic or modeled events in a set followed by an undo command would be composed as a Chart Cmd Undo modeled event. When this modeled event sets the corresponding evidence node in the Bayesian net, we might see an increase in the probability assigned to the hypothesis that the user may be having difficulty updating chart with data on a spreadsheet Continuing the example above, atomic events for the modeled event, Sel Chart Pause, are (1) a user selects a bar chart, followed by (2) some minimal prespecified period of time (e.g., 10 seconds) in which there is a dwell, indicated a cessation of user activity.
Atomic events may serve directly as modeled events. For example, DeI Chart is an atomic event representing the deletion of a chart that is also modeled event, corresponding to the observation variable in the Bayesian network.
Modeled events also include contextual information about the state of data structures in an application. For example, Chart View, means that a chart is present in the current display. This will increase the probability that charting assistance is relevant to the user.
Words appearing in a free-text query are also atomic events. Higher-level, modeled events may be defined as the appearance of one of a predefined set of words in a user's query. If a usez's query is, "How do I change the colors in this chart?,"
the modeled event, that one of a set of words (e.g., in this case, a set containing as elements the words "modify," "cony ert," and "update's has been noted in the query, is passed to the inference system. Other important words in the query, such as "color" and "chart", are also detected as modeled events when they are noted to be in sets of previously modeled sets of similar words. These modeled events are analyzed in the same way as "convert."
The class of modeled events that are persistent across multiple uses of a software program are defined as profile information. These modeled events include events that indicate user competency or lack of competency and are stored in the user profile database. Modeled events stored in the user profile provide additional -information about the user, such as the user's previous experience with particular application features or accomplishment of specific tasks, or review of tutorial material in the past. These events may correspond to variables represented in the Bayesian network and used by the inference system to adjust the evaluation of the hypotheses.
Other types of atomic events may include information about the presence or absence of user in front of the computer, as gleaned by proximity detectors, by visual features of the user, including information about the focus of the gaze of the user, expressions of the user, or gestures of the user, as identified by a video camera and vision recognition and tracking system, and speech utterances of a user as detected and processed by a speech understanding system.
After atomic events and their appropriate combination are identified for each of the modeled events, the next step 58 is to construct an event monitor for the software program that will survey user interface actions to extract the atomic events for input to inference analysis. The event monitor constructed is integrated with the software application to instrument that software application at step 60 in making an intelligent user assistance facility. Those skilled in the art will appreciate that the event monitor may be integrated within the application software, or may be a stand alone component that is part of a system library or operating system.
To complete the intelligent user assistance facility, an event processor is to compose modeled events from atomic user-interface actions (step 62). Next, a knowledge base is constructed from the model (step 64). The last step 66 is to construct an inference engine from the model which uses the knowledge base to determine suggested user assistance for the occurrence of modeled events. A Bayesian network with inference procedures serves both as a knowledge base and inference engine.
Figure 4 is a functional block diagram of the components of an embodiment of an Intelligent User Assistance Facility constructed by using the method described above. The Intelligent User Assistance Facility includes a software program 72 that has been instnlmented to monitor actions or events initiated by a user who is interacting with the software program. Periodically, these actions or events are scrutinized by an event processor 74 to discover whether any of them, either ~
.
individually or in combination, represent observable variables in the Bayesian network.
Modeled events are analyzed by an inference system 76 to form and evaluate multiple hypotheses of what assistance the user may need. These events can lead to increases or decreases in the probability that a user is experiencing a problem in using a particular application feature or in accomplishing a specific task.
The inference system 76 also accesses the user profile system 78 to check for competencies and changes assistance that is given based on user competence.
User's background, successful completion of key tasks indicative of competence in particular areas, and previous help reviewed by the user are some of the variables stored in the user profile system 78.
The IUAF controller 79 initiates a cycle of inference analysis by the inference system 76 based upon the setting of its control mode. An inference cycle may be initiated whenever a user asks for help, or whenever one or more of any number of special events have occurred, or may simply occur on a regular clocked cycle, or for any combination of these modes. When the control mode is set to trigger an inference cycle upon occurrence of a special event or the control mode is set to occur on a regular clocked cycle, the inference system 76 will autonomously offer help to the user whenever the computed importance of coming forward exceeds the autonomous assistance threshold set by the user. When the control mode is set to initiate an inference cycle whenever the user asks for help, the inference analysis of user actions and program context is combined with an analysis of the words in a free-text query whenever the user submits a free-text query.
Those skilled in the art will appreciate that the functional blocks illustrated in the diagram may be implemented as separate components or several or all of the functional blocks may be implemented within a single component. For example, a separate library may be implemented that contains the reasoning system functionality and the event processor functionality. In such an implementation, the reasoning system may query the instrumented program periodically for all of the atomic events since the last query or may simply ask for a specific number of the last occurring events, and then the reasoning system may access the modeled event database for that particular software program to determining whether any of the atomic events are modeled events for input to inference analysis. Those skilled in the art will also appreciate that the inference analysis may be implemented to include only the ongoing analysis of user actions and program context except when a user asks directly for help with a free-text query. In that case, the inference analysis may be implemented to include only the analysis of the words in the free-text query whenever a user asks directly for help with a free-text query. Those skilled in the art will also appreciate that a separate analyses could be undertaken for different classes of modeled events, such as for words in a query and for actions in the interface, and the results of these separate analyses could be combined with the ability to control the weight given to the results of the different analyses.
Instrumented Program Any software program or library may be an instrumented program, including operating system programs and application programs. As an illustration of an instrumented program, an application program will be described. A typical software application contains a user interface and application procedures. As a user interacts with a software application using input devices such as depressing a key of the keyboard or clicking the button of a mouse, the operating system translates those user actions into events and sends them to the application program. The user interface component of the application program processes those events and forwards them to the appropriate application procedures for responding to those events. The event monitor constructed for the Intelligent user assistance facility for a particular application may be integrated within the application software as illustrated by Figure 5. In this configuration, the event monitor 84 watches for events passing from the user interface 82 component to application procedures 86 and forwards them to the event processor through a call interface to the event processor. It also may be implemented as a stand alone component that is part of a system or application library which makes periodic calls upon the application program to send it all events since the last period. or it may simply request that the application program send only a specific number of the last occurring events. Alternatively, it may be incorporated as part of the operating system in which case the event monitor screens events passing from the operating system to the application program and forwards them to the event processor.
Figure 6 is a functional block diagram of an instrumented spreadsheet application program as in the help-topic embodiment. As a user interacts with a spreadsheet application program using input devices, the operating system translates those user actions into events and sends them to the spreadsheet application program.
For example, the user may place the pointer of the mouse on a menu item and may then click on the button of mouse. The operating system will translate that user action into an event that the user has clicked on an application menu item and send the event to the spreadsheet application. The user interface 92 component of the instrumented spreadsheet application program 90 receives that event and forwards it to the appropriate spreadsheet application procedures 96 for processing that event.
The spreadsheet events monitor 94, watching for events passing from the user interface 92 component to application procedures 96, will see that event and forward a copy to the event processor.
Event Processor The event processor is a general monitoring and event composing system that allows high-level events to be created from combinations of low-level events. The event processor does not need to include sophisticated reasoning capability.
However, those skilled in the art will also appreciate that a variety of deterministic and probabilistic event preprocessing methods can be employed to work with arbitrary abstractions or clusters of events and to transform events into numeric information with a variety of mathematical functions, including those that might be directly interpreted as likelihoods.
Since the Event Processor produces a set of logical events, any reasoning engine, rule-based or probabilistic, can utilize its output. Figure 7 is a block diagram illustrating the components of the event processor. Atomic events or user input device interactions are sent by an events source 100 such as an instrumented program to the 5 event processor. The instrumented program may use a call interface that specifies the event information to the event processor. Upon receiving these events, the event processor may time-stamp each event and store them in local storage 102, such as system memory, in one of several kinds of data structures for storing information, including a database of records or finite circular queue ordered by the time stamp, or
10 first in, first out order.
A modeled event composer 104 periodically surveys the atomic events in local storage 102 in search of one or more atomic events that may be composed into a modeled event. This typically would be initiated by the inference system 74 at the beginning of an inference cycle; however, the modeled event composer 104 could 15 perform composition of events between inference cycles such as whenever a new atomic event is received, or when special trigger events are received. The modeled event composer 104 then forms modeled events from atomic events according to the event composition rules defined in the event specification. The modeled event composer 104 stores each modeled event that the composer 104 generates from atomic 20 events in the modeled event database 108.
Those skilled in the art will recognize that there could be implemented a modeled event database for each individual program employing the intelligent user assistance facility. For example, there could be a modeled database for each instrumented application program resident on a computer system. Those skilled in the art will also recognize that there may additionally be a separate general modeled database that contains common system or application events such as activating a program or exciting a program. In such a case, the inference system may access both the general modeled event database and the active program's modeled event database during a cycle of inference.
In one example of a modeled event database, there is a single record for every modeled event for that database. When a program with the intelligent user interface is activated, pre-existing persistent events stored in the user profile may be used to initialize the modeled event database. As modeled events are formed by the composer, the record for that event is updated in the modeled event database.
An example of a record format for modeled events in this modeled event database is:
Fields Name Type Description EventName string Name of modeled event Occurred Boolean Flag which indicates that the modeled event occurred within its horizon.
Atomic Boolean . Flag which indicates that the modeled event is derived from a single atomic event.
StartTime real Lowest start time on any atomic events from which this modeled event is composed.
EndTime real Highest end time on any atomic events from which this modeled event is composed.
Value real This field holds the result of the evaluation of an expression LastTime real The time of last occurrence of the modeled event Count real Total number of occurrences over all sessions LastTimeThisSession real Time of last occurrence this session CountThisSession real Number of occurrences this session Persistent Boolean Flag which indicates that this is a persistent modeled event.
Of the fields used to count the number of occurrences and the rate of their occurrence, LastTime and Count are stored in the user profile database for all persistent modeled events. Upon initialization of the program, these fields get loaded from the user profile database into their corresponding modeled events in the modeled event database.
Modeled events are only relevant for a comparatively short time. This period of time during which a modeled event is relevant may be referred to as its horizon. Different events may have different horizons, a si.Yty second horizon is a suitable length of time for the horizon of most events generated by user actions. The duration of the horizon determines the length of time that a modeled event is considered in the inference system 76.
. Each modeled event may be assigned a default system or application horizon or it may be assigned a specific horizon. Modeled events that have resided in the modeled event database 104 beyond their horizon time are discarded whenever the modeled event database 104 is accessed or updated. However, before they are discarded from the modeled event database 104, persistent modeled events are recorded in the user profile. The inference engine 76 and user profiler, in general, act as event sinks and access the modeled event database 104 to extract modeled event records.
However, there is an exception. At program initialization time, pre-existing persistent events stored in the user profile may be used to initialize the modeled event database 104 and, as a consequence, modify the program's behavior.
Event Svstem Specification Tool The event specification tool allows for rapid creation of the modeled event composer and modeled event database for specific application programs during creation of an intelligent user assistance facility for that software program.
A language was developed for constructing modeled events (e.g., menu surfing) from atomic events made available by a program, for instance an application (e.g., multiple menus visited without action within 10 seconds, position of cursor on spreadsheet, etc.).
The language allows atomic events to be used as modeled events directly, as well as for atomic events to be formed into higher-level modeled events. Also, modeled events can be further combined to create other modeled events. More specifically, the language allows for Boolean combinations of atomic events and for combining the Boolean events into modeled events with operators that capture notions of temporality. The event -specification tool can be linked directly with a Bayesian network modeling tool so that the modeled events and their definitions can be generated while building and refining the Bayesian user model.
Figure 7 is a block diagram of an event system specification tool 110 used for generating code for the modeled event composer 104 and modeled event database definitions 118. Each modeled event definition 112 created during the step of defining modeled events is used to specify the events in the event language 114. The event specification 114 is then input to an event language translator 116 which translates the event language statements of the specification into high-level language code, such as C, to create the modeled event composer 104 component. The event language interpreter also creates the modeled event database definition. As part of that process, the event language interpreter 1 I6 may also create the modeled event database 106 directly from the modeled event database definitions 118 in systems employing a database structure to store information on the status of modeled events.
The event language includes Boolean and set operators, with the addition of temporal operations including dwell and sequence. Modeled events are defined in a separate text module written in the event specification language. However, they can also be constructed with the use of special definition forms. The text definitions are created as named statement blocks for each modeled event. These definitions contain the labels of atomic events and indicate operations on one or more atomic events.
The general form of a modeled event declaration in the event definition language is:
declare ModeledEventName <e:cpression list }
An example of the modeled event declaration for the Dwell on Chart modeled event is:
declare Dwell on Chart { _ _ (and (Select Chart, Dwell(S,s)) The interpreter translates the syntax statement of the event definition language to the modeled event name, Dwell on Chart and updates the modeled event database by adding a record with that event name and initializes all record fields. A
modeled event may be a complex synthesis of several other events, including other modeled events, or it may be simply a re,~aming of an atomic event. The event synthesis language supports infix expressions which entail logical combinations of atomic events, modeled events, arithmetic, and logical operators. As each elementary operation is performed, the intermediate result is treated as an event and has similar properties.
The event language has logical, set-theoretic operators, temporal, and arithmetic operators. Operators allow the definition of specific events as well as events defined as generalizations based on sets of events. For example, higher-level events can be built via abstraction or generalization with operators like Element:
Element({xi,...xo}): Any event drawn from the elements of the set of events {x,,...xa }
occurs. Such sets can be defined as specific classes of events.
Temporal operators allow us to define events in terms of sequences of events over time. Temporal intervals may be measured in terms of the number of user actions, the amount of time, or other measures of duration that can be defined as functions of user actions and time. For example, it can be useful to define modeled events in terms of Scaled seconds, a measure of duration where seconds are scaled by the rapidity at which a user is working, so as to adapt the temporal constants to users that work at faster or slower rates than an average user. When specifying temporal operators, we indicate the dimension of the measure of temporal interval by specifying the temporal dimension d with a value (e.g., c for commands, s for seconds, and ss for scaled seconds). Some useful temporal operators are:
Rate(x,t,d): At least x atomic events occur in duration t of dimension d.
Ensemble N({x,,...~,},t,d) At least N events from the set of events{xl,...xo} occurs within interval t of dimension d.
All({x,,...xt,},t,d): All events in a specified set of events {xl,...xn} occur within duration t of dimension d.
10 Sequence({xl,...xn},t,d): All of the events specified in a set of events {xl,...xn} occur within duration t of dimension d.
TightSeq({xl,...xo},t): All of the events specified in a set of events {x,,...xo} occur within occur within duration t of dimension d.
Dwell(t,d): No user action for t seconds or scaled seconds of dimension 15 d.
In addition, parameters can be specified that define the persistence or dynamics of probabilistic relationships between modeled events and other variables in a Bayesian network with increasing amounts of time since the modeled event has occurred. In the general case, we can provide any temporal function that describes how 20 the probabilistic relationships of events to other variables in a Bayesian network change as the time following the occurrence of an event increases.
More specifically, the language allows the specification of how p(E; ,to ~H~, tp), the probability of event E; occurring at time to in the past conditioned on the truth of hypotheses H~ at the present moment, tp changes with the increasing amount of 25 time tp = t since the event last occurred.
Useful distinctions for representing the temporal dynamics include horizon and decay:
Horizon: The interval of time beginning at the time t=to that an event becomes true, that the probabilistic relationship persists as p(E; , to ~ H~, to) without change. The probability remains unchanged as long as difference between the present moment, tp, and the time, tp, the event was most recently seen is less than the horizon.
Dynamic: The time-dependent functions that describe p(E; , to ~ HJ, t~ and p(E; , to ~not(HI), tp) after a horizon is reached. The dynamics can be summarized with a description of the change in the likelihood ratio defined by the ratio of these probabilities. The functions indicate how the initial probabilities p(E; , ~ ~
H~, t~ and p(E; , to ~not(li~), t~ change as the time the event became true becomes increasingly distant from the present time, tp. It can be asserted that these probabilities converge at some time to p(not(Ey H~, ~ and p(not(E~~not(H~), tp), and the initial likelihood ratio, when tp = t~, converges to the ratio of these latter probabilities, and, therefore, functions can be assessed that decay the probabilities or likelihood ratio from the initial likelihood ratio when tp = to to the likelihood ratio associated with absence of the event.
The probabilities can be computed as a function that takes as arguments the initial probabilities (at t 0), the probabilities associated with the finding being absent, and the time of the present moment, tp, and ensures consistency among probabilities at any moment. That is, P~~ ~ ~ ~ H;~ ~)=f CPS; ~ to ~ H;~ ~)~ P(no~~l H;~ ~)~ ~]
and p(E;, to ~ not(H~), tp~-f [p(E; , to ~ not(H~, t~, P(not(E~I(not(H~), tp), tp]
Operators being used to define modeled events may affect the duration information of the resulting modeled event. Conjunction results in storing the latest occurrence time of any subordinate events. Disjunction results in storing the times associated with the first true subordinate event.
The Value field of the generated intermediate result of the modeled event is set according to the result of the expression. The result of all arithmetic comparison operations is either one (Boolean TRUE) or zero (Boolean FALSE). If the argument of an arithmetic operation is an event, the argument is evaluated as one if the event has occurred and zero if the event has not occurred. The time stamps associated with simple numbers are set to the span of the current cycle; in other words, the start time is the time of the last cycle and the end time is the current time.
Since the syntax allows for recursion, care should be taken that infinite recursion not occur. These statements are entered into a text file known as-the event specification module.
The event specification module 114 is then processed by the event -definition language translator 116. The event language translator converts the event specification language 114 into C++ code which may be compiled to machine language and linked directly to the event processor system library, IUAF library, operating system library, or other implementation library. The results of translating a high-level event specification is the modeled event composer 104 which creates modeled events.
When the modeled event composer 104 is invoked during a cycle of inference, it builds the set of currently active modeled events and then returns to the caller. The inference code then performs appropriate actions indicated by the event set.
User Profile Svstem The inference system 76 accesses the user profile system 78 to check for competencies and changes assistance that is given based on user competence.
User's background, successful completion of key, tasks indicative of competence in particular areas, and previous help reviewed by the user are variables that can be stored in a persistent file and updated with time. Such persistent information about user background, experience, and competence is referred to as "profile information."
Figure 9 is a block diagram of the user profile system 78. The user profile system 78 includes a user profile database 120, a user profile access routine I22, an event status update routine 124, a background and competency dialog routine 126, and a customized tutorial routine 128.
The user profile database 120 contains records identical in format to the records in the modeled event database 106 so that there is a common format between the modeled event database records and the user profile database records. This common format makes it more convenient both for updating the user profile database with modeled events and for the inference engine to interpret these records when it access these databases for inference analysis. The user profile database records are stored persistently so that they create an historical record of particular userrs competency with specific software applications. This database is maintained on a by-user-by-application basis. An example of persistently stored modeled events that are indicative of user competency are the completion or non-completion of certain tasks, successful or unsuccessful use of particular features, and assistance received or help information reviewed in the past.
Figure 10 is a flowchart illustrating how the user profile database 120 is updated with event status of a user interacting with a specific application program: The event status update routine 124 in step 130 retrieves the first modeled event record in the modeled event database and checks in step 132 whether the record is marked as a persistent record. If it is marked as a persistent record, then the record is added to the user profile database in step 134. If it is not marked as a persistent record, then it is not added to the user profile database. Each subsequent record in the modeled event record database is retrieved in turn (steps 136 and 138), checked whether the record is marked as a persistent record (step 132), and added to the user profile database if it is marked as a persistent record (step 134).
The availability of a standard stored user profile containing information about "persistent events" could follow a user around over a local network or over the Internet, custom-tailoring the user's software wherever the user may be. While a software application is active, a new user may log on to the system and use the active software application. Figure 11 is a flowchart illustrating how a user profile is accessed whenever a new user signs on to the system or application. When a user logs on to the system or application (step 140), the user profile access routine 122 searches the local user profile database for that user's profile (step 141 ). If the profile is found, the path to that user profile is passed to the inference system 76 (step 142). If that user's profile is not found, because that user may be using the application at a remote location, then the user profile access routine searches the available networks for any user profile databases that contain the user profile (step 143). If the user's profile databases is not found er the user's profile is not found in any user profile databases accessed, then the user profile access routine asks the user for permission to enter a user competency dialog with the user (step 144) and creates a user profile for that user (step 145). The path for that user profile is passed to the inference system 76 (step 142). If the dialog is not-completed, the path to default profile information is passed to the inference system 76 (step 146) and used.
In addition to maintaining a persistent user profile for use during real-time inference, a continuing background analysis of repetitive patterns of a user's needs for assistance during one or more sessions that exceed a threshold, may be used to customize a tutorial of help information that may be offered to the user at the end of a user's session with the that program. Figure 22 is a flowchart of the routine for generating a customized tutorial. During a user's session with a specific application program, a histogram of help topics exceeding a relevant probability threshold is recorded by the inference system 76 (step 150). When the user asks to quit the session with the application program, the user profile saves the histogram in the user's profile (step 152) and prepares a customized tutorial of help information related to the help topics recorded in the histogram (step 154). The user profile system notifies the user upon exiting the application program that a customized tutorial was prepared for online review or for printing as a manual (step 156).
Inference Svstem When a cycle of inference analysis is initiated, the inference system 76 accesses the modeled event database 106 to extract modeled event records for each of the modeled events that has occurred since the last cycle of inference. A
functional block diagram of the inference system is displayed in Figure 13. The inference system 76 includes one or more knowledge bases 160 and an inference engine 165.
Knowledge bases 160 include information relating variables that represent observable states of the world, such as user actions and words, to variables that represent hypotheses of interest about a user's goals and needs for assistance that may not be observed directly or that are costly to observe. For example, the goals of users can not necessarily be directly inspected, but a sequence of computer commands can be recorded. A user can be actively queried about goals, but this may be quite distracting to users.
A deterministic knowledge base consisting of an interrelated set of logical rules may Link observations to hidden hypotheses. However, it is often more appropriate to process the uncertain relationships as probabilities between observations and the likelihood of hypotheses about a user's needs. For example, in attempting to -5 understand and to predict the behavior of a complex system such as human physiology it is typically not possible to completely model with deterministic relationships all components of the system, and to then have access to a deterministic model for performing diagnosis or designing therapy based on a set of symptoms. For diagnosis and decision making about complex systems, we are often forced to reason under 10 uncertainty, and to explicitly address the incompleteness in our understanding.
Probability provides us with a means of diagnosis and forecasting about the behavior of complex systems given knowledge about a set of relationships among observational variables and hidden variables we may identify as being important. We can use probabilistic methods to represent and reason about weak and strong uncertain 15 dependencies among observations, such as symptoms of a patient and variables such as diseases in a patient. A good example of complex system is a user attempting to perform a task while interacting with a computer software application or system. It is very difficult to build deterministic models that Link a user's behavior to a user's goals and intentions. The best way to diagnosis a user's needs is to develop appropriate 20 abstractions, based on our understanding of the relationships among various kinds of user's goals and actions, and to represent and reason about the relationships with probability. Probabilistic methods allow us to build models at a level of abstraction that is appropriate in light of our incomplete understanding about users and their actions.
A Bayesian network or a generalization of a Bayesian network, called an 25 influence diagram, can be employed to represent the certain or uncertain relationships among user actions and such hidden, but important, states as user goals, user intentions, and user needs, given observable information such as one or more user actions.
A
Bayesian network is a directed acyclic graph where nodes are random variables and arcs represent probabilistic dependencies among those variables. Variables in a Bayesian 30 network are chance variables or deterministic variables. A Bayesian network represents a joint-probability distribution for the set of variables it represents. The probability distributions over the values of chance variables depend on the values of direct ancestors, or parent variables that are parents of nodes. The value of deterministic variables is a deterministic function of predecessors. Influence diagrams are a generalization of Bayesian networks which represent additional nodes that represent possible actions and the utility of outcomes.
The inference engine 16~ includes inference procedures 166 that operate on the knowledge base 160. The knowledge base 160 includes a Bayesian network 162.
Those skilled in the art will recognize that the knowledge base may be a deterministic knowledge base with logical chaining procedures as the inference procedures of the inference engine. Or the knowledge base may be a Bayesian influence diagrams with inference procedures for operating on Bayesian networks as the inference procedures of the inference engine. Temporal reasoning procedures 167 and value-of information procedures I68 are also included as part of the inference engine 16~. Other specialized procedures may be included.
Bayesian inference procedures operate on Bayesian networks to compute a consistent posterior probability distribution over the values of variables in the network given the setting of states of observable, or evidence, variables to specific states based on the observations. Assume we are interested in the probability distribution over unobserved hypotheses of interest, H,,... Hm, and have access to observations and profile information E,,... En. Bayesian network inference algorithms compute a probability distribution over H given the observations, written p(H~ Et,... EJ. For influence diagrams, decision-theoretic inference is performed to identify the expected utility of alternative actions. The best action is the one associated with the highest expected utility.
In the most general sense, the task of designing inference for an intelligent user assistance facility is best undertaken from the perspective provided by decision theory, an extension of probability theory to reflect concerns of value under uncertainty. Influence diagrams allow us to express fundamental relationships about uncertainty, action, outcomes following action, and the value of those outcomes. This representation is useful in the design, understanding, and in many cases, the actual implementations of intelligent user assistance systems. However, it is often the case, that a simpler systems may be built without explicitly representing decisions, outcomes, and the utility of outcomes, and instead to use Bayesian networks to represent probabilistic relationships, and to use probabilistic inference in combination with procedural controls and thresholds as approximations of more complex influence diagram models.
A general influence diagram for user modeling and action to assist users of software is portrayed in Figure 14 as an exemplary embodiment of a Bayesian influence diagram as a knowledge base. As portrayed in the figure, a user's background 170 influences with uncertainty, a user's overall goals 171 in using software, as well as the user's knowledge 172 in using software. User's knowledge I72 is also influenced by the previous help 173 that user may have seen. The user's background 170 and previous help 173 are variables that can be stored in a persistent file and updated with time. Such persistent information about user background, experience, and competence is referred to as "profile information." As indicated in the influence diagram, the user's goals 171 and knowledge 172 in turn influence with uncertainty, the informational needs 174 of the user. T'he goals 171 and needs 174 in turn influence the sensed activity 175 and the words that might be used in a query I76 to the software, or software's help system. A
utility function of the user 177 is represented as a diamond. The utility 177 is influenced directly by the informational needs 174 of the user, the cost of taking autonomous action 178 (e.g., distraction for the user's current focus of attention) and the action that is taken. Several classes of action may be available to the system, including the providing of advice or help 179, the execution of software actions of various kinds 180 and the acquisition of additional, previously unobserved information from the system, 181 or directly from the user. As indicated in the Figure 14 by the scale on the cost of assistance variable 178, we may wish to allow the user to directly change the cost of the assistance so as to control, in a fluid manner, the degree of autonomy given to the system.
The overall goal of the inference system is to identify actions that will optimize the user's expected utility given the user's needs and the cost of taking autonomous action. Given a set of evidence E about a user's background and actions, the probability distribution over hypotheses of interest about a user's needs, H, is , computed. In the case of the Bayesian influence diagram in Figure 14, the probability distribution over a user's needs for assistance is computed. The expected value of actions in the set of all possible actions A must be considered. To do this, consider the utilities over outcomes, represented as a utility model (diamond node). The utility model contains information about the value or utility of outcomes. Outcomes (A,H) are defined as doublets of actions A taken and the state actual user's needs H.
The utility model tells us the utility associated with each outcome, u(A,H). The best action to take, A*, is the action that ma.~cilnizes the expected utility under uncertainty, computed as follows:
A* = arg maxA ~~ u(t~,H~) p(H~~E) Although influence diagrams represent explicit models of action, Bayesian networks are often easier to build and can be used for decision making and action by employing thresholds or rules about probability over variables in the network to indicate when actions should be taken based on the degree of belief assigned to a variety of states including states that describe preference about action. Influence diagrams such as the one portrayed in Figure 14 can be used to clarify approximations used in the Bayesian network.
Temporal Reasoning and Dynamics The inference system 76 also contains special temporal reasoning knowledge 164 in the knowledge base 160, and procedures 167 for performing inference about the changing relationships of observations to other variables in the model as the observations occur at progressively more distant times in the past. In the general case, applications of Bayesian procedures and knowledge bases to reasoning about patterns of observations seen over time requires the consideration of variables and their interdependencies within a single time and at different times. Building and performing inference with Bayesian networks that include copies of variables and dependencies for different times slides can lead to difficult computational reasoning problems and unacceptable response times for a user interacting with a program. Thus, several approximations are of value. In one kind of approximation, only specific dependencies considered to be of the greatest importance are considered over time.
A portion of a Bayesian network with sets of variables at different times and prototypical dependencies represented is displayed in Figure 15. Figure 15 displays a Bayesian network that explicitly represents random variables at different points in time, aad probabilistic dependencies among variables at a single time and among variables at different times. The lack of dependencies among variables are assertions of assumptions of independence. The figure displays possible influences among the primary relevant assistance to provide a user at present moment with the most relevant assistance to provide users at earlier and later times. Also, the figure highlights potential dependencies among the status of observations (E; and E~) in the present, with observations made in the past and future. The figure also indicates the relationship between the primary relevant assistance in the present and observations in the past. In the general case, such multiple-connected networks are difficult to solve and to assess during the construction of the models.
We will now describe an approach to reasoning about relevant assistance over time that is more tractable to solve and to assess, through making additional assumptions of independence and representing the relationships among evidence seen at progressively more distant times in the past and hypotheses of interest with parameterized functions that dictate the strength of the probabilistic relationship between those observations and current the goals or needs for assistance. The use of this approximation in an exemplary embodiment employs a single explicit Bayesian network, but allows the system to consider implicitly multiple Bayesian networks over time. This method takes advantage of direct assessment during modeling time of functions that adequately describe the dynamics of the probabilistic relationships between observed variables and other variables in a Bayesian network. At modeling time, a horizon and dynamics are assessed for each observational variable E;, and the variable is annotated with this information. At run-time, the inference engine includes procedures 167 for using this information about horizon and dynamic.
The horizon for an event captures the interval of time the probabilistic relationships with 5 hypothesis H; persists as p(E;, to ~ H~, ~) and p(E;, t~ ~ not(H~), t,~
without change. The dynamics is the time-dependent change of the probabilities in the present moment as described earlier.
Heuristics in defining a horizon for events include the use of a default horizon for all observational events considered by a system unless otherwise specified, 10 and the use of an event queue of finite length k, where only the last k modeled events are considered in the analysis. This approach can be combined with dynamics as just described.
Value of Information 15 The inference engine I6~ also includes expected value of information (EVI] procedures 168 to compute the expected value of acquiring information from a user. EVI procedures and information-theoretic approximations that can be employed in probabilistic models are well-known to people skilled in the art of decision theory.
The EVI is a means for computing the expected value of acquiring information about 20 variables that have not yet been observed. Such information includes answers by users to questions posed by the computer system about their goals and needs. The Net Value of information (NEVI) is net value of gathering information, including the costs of gathering the information. The inference system 76 only asks a question, or gathers information autonomously, when the informational benefits of the information 25 outweigh the costs. If EX refers to each previously unobserved variable that can be queried, O,~ is a possible observational value of Ex, and A is the value that will be observed when the variable is evaluated, the NEVI of each variable is:
NEVI(E,~ _ ~k Pox - Ok ~E) * ~m~., ~; u(~~H;) P~;~E~Ex O
- maxA ~; u(A;,H~) p(H~~E) - C(E~
where C(E~ is the cost of evaluating evidence EX. EVI can be applied directly to influence diagrams. Several well-known information-theoretic algorithms, including those based on computation of entropy, function to provide similar functionality for .
decisions about information gathering in probabilistic models such as Bayesian networks. These algorithms are typically more tractable than computing NEVI
with influence diagrams. They do not explicitly manipulate utilities, but nevertheless can be employed in conjunction with heuristics that balance the cost of information with measures of informational value.
IO Figure 16 is a flowchart of value-of information procedures in the value of information component. The value of information component lists all of the previous unobserved information in order of the value of it as alternative information (step 190) and weighs the benefits of obtaining the information against the cost of distracting the user for each item of information (step 192). If the benefit is greater than the cost set by the user, the IIIAF asks the user to supply the item of information that has the highest value (step 194). If the user responds, then the value of information component repeats the first two steps until either the cost of distraction is greater than the benefits of obtaining the information or the user does not respond (step 196).
Scenarios Illustrating the Screen Outuuts of the ILTAF
Figures 17 through 23 illustrate the screen outputs of the intelligent user assistance facility in an exemplary embodiment of a spreadsheet application.
The system considers a user profile, user-specified threshold information, and combines words and user actions to provide intelligent assistance to the user. The system has the ability to respond to requests for assistance as well as to autonomously provide assistance. Figure 17 shows an assistance interface 200 that is generated when the user explicitly has requested help. Guesses about useful assistance to provide the user are displayed in the list box 201 in the upper left-hand corner. The current system settings box 202 in the lower left-hand corner indicates the user is an expert user.
This information was provided by the user profile system. The current system setting box 202 also indicates that both user actions and free-text queries are used by the inference engine during a cycle of inference analysis. Given the profile information and the recent actions involving interaction with the layout of text, in combination with pauses, and with the user taking action to change the appearance of rows and columns shortly before help was actively requested, the system believes the user may best benefit by _ receiving assistance on working with fonts, advanced formatting instructions, and with ' methods for changing the dimensions of the spreadsheet document. To the right of this screen photo is displayed an inference graph 203 showing the probability distribution over a set of task areas that was generated by the Inference System 76. This display was built for engineering purposes but may be displayed to the user. The length of the IO bar graph next to each task area is the probability assigned to that area.
The two highest probability areas (those with the longest associated bar charts) are for the tasks of working with fonts and advanced formatting topics.
In Figure 18, the profile information was changed to a profile for a novice user. The expertise level in the current systems setting box 202 indicates that the user has a level of expertise of a novice. All other events handling remained unchanged. We have used a profile for a novice user. The profile information is used to modify the probabilistic relationships in the Bayesian network. The inference graph 203 displaying the probability distribution over relevant assistance shows the revised probability distribution with the profile information. As can be seen in the best guesses list box 201, the system now believes that the user may most benefit by assistance with basic working with rows and columns of the spreadsheet, changing alignment and basic formatting of charts.
Figure 19 shows the analysis of the same set of user actions, but now, with consideration of the words in a user's query. The user has input to the system the natural-language query, "How do I make this look prettier?" in the query input box 204.
The analysis of the words and the actions are combined to generate a new probability distribution, as indicated in the inference display 203, and a new corresponding list of recommended topics in the best guesses list box 201. Now, the list is resorted and updated with autoformatting assistance being recommended as the most relevant assistance, but also still contains other formatting topics, including changing alignment, working with borders, and working with fonts.
Figure 20 shows the program at a later time when help is again requested after interaction with a chart in the document, coupled with pauses on chart commands. _ The probability distribution computed by the inference system 76 is displayed in the ' inference graph 203. Now, the recommended topics in the best guesses list box focus on a variety of high-Likelihood areas in the area of graphing information. Figure 21 shows the addition of information submitted in the form of a query in the query input box 204. The natural language query from the user is, "I need to access data from a different application". The words are now considered in conjunction with the events.
The inference system 76 provides a revised probability distribution in the inference graph 203 and a revised list of recommended topics in the best guesses list box 201, centering in retrieving data from a database and about changing the data displayed in a chart.
Figure 22 illustrates the screen outputs of the inference system 76 offering autonomous help to a user. While the user has been interacting with a spreadsheet program 205, an assistance monitoring agent computes the probability that a user needs assistance and the inference system 76 also computes the likelihood of alternative kinds of help to give the user, should the user require assistance. The assistance monitoring agent displays the result of inference on the probability that a user needs assistance in the assistance monitoring agent window 206. Like the inference graph 203 display described above, the assistance monitoring agent 206 was built for engineering purposes but may be displayed to the user. The user who has been interacting with a spreadsheet program 205, now selects the whole sheet and pauses. As the assistance monitoring agent window 206 shows, the computed probability that the user needs help moves from 26% to 89%, reaching a threshold, and a small timed window, the autonomous assistance window 207, autonomously comes forward displaying the highest likelihood assistance areas that were computed. In this case, the highest likelihood topics include formatting cells, checking for spelling errors, and performing calculations, topics that have implications for analysis or modification of the entire document. The autonomous assistance window 207 politely offers help. It also provides a chance for the user to reset a threshold that will increase or decrease the probability required on the likelihood the user needs help before displaying the window.
The window 207 will time out and be removed with an apology for distracting the user if the user does not interact with the window 207.
Figure 23 illustrates the display 208 of the topics that have been determined to be of value to the user based on the recording and summarization of ongoing background inference about problems that user has been having during a session using a software program. Profile information about the user's background, competence, and about the assistance that the user has received or reviewed in the past are also included. These topics have been prepared for printing as a custom-tailored manual for later perusal.
Bavesian Network Annotated with Temporal Reasoning Figure 24 shows an example of a Bayesian network that computes the probabilities of alternative forms of assistance to provide a user as a function of profile information about the user, as well as recent actions taken by the user. To indicate that this structure may be repeated for many classes of observations and task-specific needs for assistance, variables x, y, z, and A have been used in several of the nodes.
In this model, we represent profile variables that represent the status of user competency in two different areas of software functionality, x and y, with a set of variables labeled User Competence x 210 and User Competence y 211. The user competencies are each influenced by variables representing (1) specific observed activities by the user, (2) the history of help topics that have been reviewed by the user, and (3) the overall user background 212 that may be uncertain or set by a dialogue with the user. As indicated in the Bayesian network, the user competencies directly influence the prior probability of different assistance being relevant to the user, as represented by the states of the Primary Assistance Needed variable 213. This variable is also influenced by a context variable 2I4, that describes information about program state, such as the existence in the current version of an application of a particular data structure. In this model, the problem is formulated with a single primary assistance variable, with states representing alternate forms of assistance. The model assumes that there is only one state of primary assistance the user needs at any particular time.
Multiple variables about assistance can also be represented to capture the notion that -5 several types of assistance may all be relevant at a single time.
As depicted in the Bayesian network, the Primary Assistance Needed node 213 influences a set of observed modeled events, such as selecting a graphical object Z, then pausing for more than some predefined period of time (Sel Z
Pause 215), or modifying an object and then performing an undo, (Z Cmd Undo 216). At run time, 10 these modeled events are detected and passed to the Inference System 76 by the Event Processor 74. The corresponding observable variables to the modeled events are update and Bayesian inference is performed to update the probabilities over ali of the unobserved variables in the system, including the Primary Assistance Needed variable 213. The time of the event is noted and as the event flows progressively into the past, 15 temporal reasoning procedures 167 are applied to update the probabilistic relationships in the network.
Figure 25 shows an instantiation of the nodes with variables x and z with distinctions with relevance to reasoning about the relationship of actions and profile information on charting or graphing information from a spreadsheet, such as the 20 Microsoft Excel product. Variables that are persistent and stored in a profile are marked with an adjacent P. Variables that are observed as modeled observable events are marked with M. One or more nodes may represent various classes of competency with charting. In this case, a single variable, User Competence Charting 220, is displayed as the variable that contains information about the overall level of charting ability of the 25 user. The probability distribution over the states of this variable are influenced by Hx Help Charting 221 and Hx Activity Charting 222, and User Background 212. Hx Help Charting 221 is set by observations that are stored in persistent profile about one or more charting activities the user has completed successfully and information about the help information on charting in an online manual that the user has reviewed.
30 Competencies about charting and a variety of other areas of software functionality influence the Primary Assistance Needed variable 213. This variable in turn influences the probability of observing different behaviors by the user. For example, the state of a user needing assistance on a specific type of charting task, influences the probability of seeing the user select and pause for s seconds on a chart in the document (Sel Chart -Pause 223). Appendix 1 contains a more complete Bayesian network in an exemplary embodiment of a spreadsheet application.
Figure 26 represents an exemplary embodiment of temporal knowledge and inference. Each observable modeled event is annotated with information about the effects of time flow on the probabilistic relationships of the event with other variables in the Bayesian network. In this case, each node is annotated with temporal dynamics information about a horizon and decay, which may be implemented as a table of probabilities or other data structure. The dimensions of time for the temporal dynamics of each variable are indicated in seconds that have occurred since the event became true.
We now provide an example of a case based on components in the knowledge base displayed in Figure 26.
Let us consider a case where we have the following profile information:
User_Background 212: Consumer Novice HX_Help Charting 221: No_Help Reviewed Context: Normal_View Obj ects: Sheet( 1 ),Chart(2) At time t=50 seconds, we have noted the following modeled events:
Chart_Cmd_Undo 224: Absent Chart_Create_Pause 225: Absent Sel_Chart_Pause 223: Absent Move_Chart_Pause 226: Absent Chart_Dialog UnSucc 227: Absent Menu_Surfing 228: Absent Click Unrelated 229: Absent Sel Chart Pause 223 P~utoIH;~~=.15 p(not(E,)I H~, to) _ .85 p(E,, tv Inot(H~), t~ _ .005 p(not(E,), to I not(H~, t~ _ .995 Horizon: 5 seconds Dynamics:
p(E,, tv I H~ , to): linear convergence at 15 seconds p(E,, to (not(H~), t~: linear convergence at 15 seconds Chart Dialog Unsucc 227 P~z~ ~ I H; ~ ~) _ .08 p(not(E~I H~ , t~ _ .92 p(Ez, to I not(H~), t~ _ .0005 p(not(E~, to I not(~H~), t~ _ .9995 Horizon 20 seconds Dynamics: 0 A cycle of analysis is initiated at t=50 seconds. The cycle includes the composition of modeled events from atomic events local storage 102, the transmitting of the events to the Inference System 76, and inference with the new observations. We will explore the current probability that the primary assistance needed by the user is help with charting information, p(Charting AssistancelE). The prior probability of the user needed help in this area conditioned on the user background, competence, and history of reviewing help is .001. The context information has increased this to a probability of .04. Probabilities are also assigned to other hypotheses about the primary assistance as a function of all of the evidence seen so far (but that is not explicit in this example).
At t=52 seconds the user selects a chart. The atomic event of chart being selected is detected and is stored in the atomic event local storage 102. The user then pauses for 3 seconds without doing anything. A dwell of 3 seconds is detected in the event local storage 102. A cycle of analysis is called at t--60 seconds. A
modeled event Sel Chart Pause 223 is composed by the Event Processor 74, tagged as becoming true at t=55 seconds, and is sent to the Inference System 76.
~ The Inference System accesses the Bayesian network that is annotated with temporal dynamics information. As indicated in the table of probabilities for Sel Chart_Pause 230, there is a horizon of 5 seconds, and a linear convergence of probabilities to the probabilities of absent at 15 seconds.
At t=63 seconds, the user selects a chart dialog and cancels the dialog with out success at t=66 seconds. A cycle of analysis is called at 70 seconds, and the modeled event Chart Dialog Unsucc 227 is composed and passed to the Inference System 76 with a time stamp of 66 seconds. As indicated .in the table of probabilities for Chart Dialog Unsucc 231, there is a horizon of 20 seconds, and no temporal dynamics.
This user has set the threshold for being bothered with automated assistance at p=.12. Thus, the user is offered assistance on charting when the probability that this topic is relevant rises to p= .14.
At t = 50 seconds p(Sel Chart Pause, to ~ H~ , tp) _ .1 ~
p(Sel Chart Pause, to ~not(H~), ~ _ .005 p(not(Sel Chart-Pause) H~ , tp) _ .85 p(not(Sel Chart Pause)~not(H~, t~ _ :995 p(Chart Dialog Unsucc, t~ ~ H~, ~ _ .Og p(Chart Dialog Unsucc, ~ ~not(H~, ~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~), ~ _ .9995 p(not(Chart Dialog Unsucc)~ Fig, t~ _ .92 Inference result: p(Charting Assistance~E, t~ .02 At t = 60 seconds p(Sel Chart Pause, to ~ H~, t~ _ .15 p(Sel Chart Pause, to ~not(H~, t~ _ .005 p(not(Sel Chart Pause)) H~, ~ _ .85 p(not(Sel Chart_Pause)~not(H~, t~ _ .995 p(Chart Dialog Unsucc, to j H~, t~ _ .08 p(Chart Dialog Unsucc, to ~not(H~, t~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~, ~ _ .9995 p(not(Chart Dialog Unsucc)~ H~. ~ _ .92 Inference result : p(Charting Assistance~E, t~= .07 At t = 70 seconds p(Sel Chart Pause, to ~ HJ, t~ _ .15 -> .46 p(Sel Chart Pause, t~ ~not(H~, t~ _ .005 -> .66 IO
p(not(Sel Chart Pause)~not(H~, t~ _ .995 p(not(Sel Chart Pause) Fig, t~ _ .85 p(Chart Dialog Unsucc, to ~ H~, t~ _ .08 p(Chart Dialog Unsucc, to ~not(H~, ~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~, t~ _ .9995 p(not(Chart Dialog Unsucc)~ H~, t~ _ .92 Inference result: p(Charting Assistance~E, tp~ .04 At t = 80 seconds p(Sel Chart Pause, to ~ H~, t~ _ .15 -> .46 -> .85 p(Sel Chart_Pause, to ~not(H~, tp) _ .005 -> .66 -> .995 p(not(Sel Chart_Pause)~not(H~), tp) _ .995 p(not(Sel Chart Pause) H~, tp) _ .85 p(Chart Dialog Unsucc, to ~ H~ , tp) _ .08 p(Chart Dialog Unsucc, to ~not(H~), ~ _ .0005 p(not(Chart Dialo~Unsucc)~not(H~ ), ~ _ .9995 _ p(not(Chart Dialog Unsucc)~ Fig , ~ _ .92 .
Inference result: p(Charting Assistance~E, t~= .13 At t = 90 seconds p(Sel Chart Pause, to ~ H~, t~ _ .85 p(Sel_Chart Pause, t~ ~not(H~, tp) _ .995 -p(not(Sel Chart Pause)~not(H~), ~ _ .995 p(not(Sel Chart Pause) H~ , t~ _ .85 p(Chart Dialog Unsucc, to ~ H~, t~ _ .08 p(Chart Dialog Unsucc, to ~not(H~, t~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~), t~ _ .9995 p(not(Chart Dialog Unsucc)~ H~, t~ _ .92 Inference result: p(Charting Assistance~E, t~= .04 System Level IUAF for Multiple Applications The methods described can not only be applied - to individual applications, they can also be employed at the operating system level to reason about providing assistance in a system with multiple applications. In an exemplary , embodiment to provide system assistance, a Bayesian network is used to assist a user with exiting an application by guessing when the user has completed or will soon be completing a task.
A Bayesian network knowledge base for assisting a user with this task is displayed in Figure 27. Profile information including whether or not the user has reviewed help in past (Reviewed Help in Past 240), has exited before (User Exited Before 241 ), and has demonstrated that they can perform a double-click operation (Successful Double Click 242) are included. Modeled events represented as observations in the Bayesian network include dwell after activity (Dwell after Activity243), sequence of clicks on a sequence of unrelated graphical icons (Click on 1 S Seq of Unrelated 244), mouse meandering (Mouse Meandering 245), whether a document has been recently printed (Print Doc 246) or a message has been sent (Send Msg 247), and the duration of time an application has been up (Duration of Time with App 248). A variable represents a probability distribution over the most desired action, allowing us to use probabilistic inference to reason about the likelihood of alternative best actions.
The sample scenario, demonstrates the results of the effect of monitored events on inference about the user. The sample case assumes the following profile: the user has not exited before on his/her own, the user has not demonstrated a successful double-click action, the user reviewed help about exiting applications but did not dwell on the information. The system now detects that the user has been in email for almost 10 minutes, and that the user has clicked on an unrelated sequence of graphical icons.
Figure 28 displays the states of each random variable, and shows, by the length of bars next to the states, the probability over different states of each random variable. As demonstrated by the bar graph, the Inference System 76 has computed that there is a high probability the user is ignorant about exiting (User Ignorant about Exit 249) and a low probability that the user has or will shortly complete a major task (Major Task Completed 250).
In Figure 29, we see the results of inference after new modeled events are considered. In this case, the user has sent email (Send Msg 247) and has now dwelled for more than three scaled seconds a$er activity (Dwell after Activity 243).
Providing a tip to the user is now identified as the most desired exit-related action (Most Desired Exit Action 251). Figure 30 shows the same scenario with one change:
the user has received help or assistance within the last two minutes (Recent rec Exit Help 252).
Now the most likely desired action is to do nothing, followed by asking the user if the system should exit the application for them (Most Desired Exit Action 251 ).
IUAF Controller There is typically a need for the overall control of various components of the Intelligent User Assistance Facility. Controls are needed for making decisions about the policy for calling the Event Processor 74 and Inference System 76, and for determining when assistance should be provided autonomously. Figure 31 is a diagram illustrating various overall coordinative control modes of the Intelligent User Interface controller that may be used separately or together in combination. The first coordinative control mode illustrated is the timer mode 260 in which a repeated system timer fires to notify the IUAF controller to begin another cycle of monitoring and inference analysis (step 261). The IIJAF controller then invokes the inference system (step 262). Upon completion of the inference analysis, the controller waits until the timer fires again (step 263). The timer mode can be extended to consider other constraints before allowing a cycle to begin. For example, in cases where complex inference occurs during event monitoring and inference, the additional condition of assessing the availability of computational resources (e.g., identifying whether there is idle time) can be added. Thus, the cycle of event processing and inference will occur when a timer fires and when there is also a condition of computational resource availability.
The second mode illustrated is the demand mode 265. This mode only performs inference analysis when the user asks for assistance. When an instrumented application receives a query for help information (step 266), the instrumented application sends notification to the IUAF controller (step 267). As a result of receiving the notification, the ILJAF' controller then invokes the inference system (step 268).
Upon completion of the inference analysis, the controller waits until it receives the next notification (step 269).
The third mode illustrated is the special event trigger mode 270. In this mode, there is a list of special events that trigger a cycle of inference analysis.
Whenever the event processor detects an occurrence of one of the special events (step 271), the event processor sends notification to the IUAF controller (step 272). Again, the IUAF cantroller then invokes the inference system as a result of receiving the notification (step 273). Upon completion of the inference analysis, the controller waits until it receives the next notification (step 274).
Any two or all three modes may be active at the same time. In the case when all three modes are active, if the user has not asked for assistance and has not initiated any user actions that translate into one of the special events during the period of the timer, then a cycle of inference analysis would occur when the period of the timer expired. Otherwise, the cycle of inference analysis would occur during the period of the timer when the user asked for assistance or initiated a user action that translated into one of the special events.
Integration of Analysis of Free-Text Ouerv with Events As mentioned earlier we can treat words in a user's free-text query for assistance as sources of modeled events. The Microsoft Answer Wizard integrated in all applications of the Microsoft Uffice "95 product line makes use of a probabilistic model relating words in free-text query to the probability of relevant help topics. This has been described in U.S. Patent Application No. 08!400,797 (now issued as U.S. Patent No. 5,694,559). Ln one exemplary embodiment, wards and actions can be handled together in a uniform way as modeled events in a system that reasons about the probabilistic relevance of words and actions.
In an alternate embodiment, there are separate subsystems for handling words and for handling actions. The analyses of these separate subsystems are 5 combined after separate probabilistic analyses are completed. Figure 32 is a flowchart of the method for combining separate probabilistic analyses. First a Bayesian analysis of the most relevant help topics is computed for the given free-text query (step 280).
This may be done following the methods described in U.S. Patent Application No.
08400797. The next step is to output a list of help topics in rank order by the highest 10 probability of relevance (step 281). Then a second Bayesian analysis of the most relevant help topics is computed for the given program state, user profile and sequence of user actions (step 282). The next step is to output a second list of help topics in rank order by the highest probability of relevance (step 283). Then both of these lists of help topics are combined, after assigning a weight to the help topics on each list, by 15 computing the probability that each help topic which is the same on both lists is relevant (step 284). The final step is to output the combined list of help topics in order of highest probability of relevance (step 285). In this embodiment, the performance of the system can be tuned by weighting the results of one set of events more heavily than the other by changing the value of a real-numbered parameter that is used to multiply the 20 likelihoods assigned to the probabilities of hypotheses of interest (e.g., relevance of help topics) provided by the word or action analyses.
Building separate systems for analyzing the handling of words in a free-text query and user actions allows the two subsystems to be separately optimized based on specific properties of each class of events. The same can apply to other sets of 25 actions, such as vision and speech events in systems with event monitoring that is extended with these additional sensory modalities.
Separate analyses also allow for the ease of implementing specific policies on combining the results, separating them, or sequencing them. For example, an embodiment might display an initial guess of assistance based on inference based on 30 monitored actions, but when words are added, rely solely on the analysis based on the words, given the performance of a word analysis systems versus an action system. An alternative approach is to have control rules about the combination versus separation of the inference in different subsystems based on a comparative analysis of the results of the words and actions analysis. _ Control of Communicating and Performing Assistance Beyond coordinative control functions, the IUAF controller also makes decisions about when to bother the user with autonomous assistance or with queries fvr information about the user's goals (based on value of information computations). In an exemplary embodiment, a special Bayesian network, or component of a network, resident in the Inference system 76 computes the overall probability that a user would like assistance at the current time. The ILTAF controller uses this changing probability to control when autonomous help is provided. T'he IUAF provides autonomous help whenever the computed probability that a user would like assistance at that time exceeds a threshold which may be changed by the user. A users changes the threshold for offering autonomous assistance using a threshold controller. T'he threshold controller, for example, may be in the form of a sliding threshold control can be displayed to users on an autonomous help window, allowing them to change the threshold depending on the value they find in receiving help versus the distraction it causes. A screen shot of a dynamic assistance window with a threshold slider is displayed in Figure 22.
Figure 33 is a flowchart of the method of providing autonomous assistance in this embodiment. During a cycle of inference analysis, the inference system 76 outputs a list of relevant help topics in order of highest probability of relevance (step 290). This may be done as described previously in the discussion of the inference system. The next step is to determine whether the probability of the user needing help exceeds the threshold set by the user for providing autonomous assistance (step 292). If the probability of the user needing help exceeds the threshold set by the user, then the inference system 76 provides a list of relevant help topics in the order of highest probability (step 294). The list may be truncated so as to only show the most likely or several most likely topics, but within a window that allows the user to scroll through a longer list if desired. Next, the list of relevant topics are checked for redundant help topics that have appeared in the previous list within a specified time period (step 296). If there are no redundant help topics, the list is displayed to the user (step 298). If the user responds to the autonomous offer of help, then steps 290 through 298 are repeated until the user fails to respond before expiration of the response timer.
If the user fails to respond before the expiration of the response timer, then a message of apology is displayed on the title of the window (step 299) and the window is subsequently removed.
Those skilled in the art will recognize that alternate embodiments may be used such as only offering autonomous assistance when directly requested by the user, e.g. by clicking on a graphical icon associated with the intelligent assistance facility.
The system can also use computed measures of expected utility or its approximation for offering autonomous assistance. Furthermore, the system could only offer autonomous assistance when the sum of the individual probabilities of the relevance of each topic in the set of topics that will be offered to provide assistance exceeds some constant value.
In an exemplary embodiment, guesses on help topics are not presented when a user requests assistance, unless the probability computed for the top five topics together do not sum to at least .40. In yet another alternate embodiment, autonomous assistance will not be offered unless the user is pausing, so as to not distract the user while the user is engaged in activity unless it is deemed important to assist the user with that activity while the activity is in progress.
It can also be useful to provide social conventions and metaphors. For example, the results of inference can be used to control the behavioral gestures of a graphical social cartoon character. An example of this kind of interface is displayed in Figure 34. IIJAF inference results about likely problems the user is having or potentially valuable assistance might be alternately be displayed in a "thought cloud"
metaphor over a character's head. Free-text queries can be typed into a graphical text input box that appears to be a means for a character to see or listen to the query.
beech Recognition Enhancement via Integration with IUAF Components Those skilled in the art of speech recognition will appreciate the power that could be gained if the speech language models used to characterize phonemes by probability were dynamically influenced by information computed about the goals, intentions, or needs of computer users. As portrayed in the influence diagram in Figure .
12, user background and goals influence the actions seen in an interface, the words used in free text query, and the speech utterances generated by users in attempting to communicate with a software system or application. Understanding the likelihood of goals and intentions of users in making utterances can enhance our ability to perform automated speech recognition. Such systems will allow us to adjust the probabilities over utterances, and thus, the likelihood of words, and phonemes being uttered by users of computer software.
Speech recognition involves processing the audio signal generated by an utterance into likely phonemes, and the processing of phonemes into likely words.
Probabilistic systems for speech recognition may compute the likelihood of phonemes given the audio signal and the likelihood of alternate words given the signal.
The term speech is used to refer to the evidence provided by the steps of processing an utterance into audio signals or into phonemes.
Bayesian speech understanding systems compute the probability of word or stream of word from an utterance as follows:
p(words~speech) = p(words) * p(speech ~ words) p(speech) By computing the probability over a user's needs or goals given user actions as represented by modeled events (including those based on gaze), we can refine the accuracy of the probability of words given a user speech as follows:
We can update the likelihood that words or word strings will be used in the context of goals by representing the likelihood that words or word strings will be used in the coate~ct of specific goals, and update the base probabilities that words will be used, p(words), contained in a typical language model in real-time. That is, instead of using a static database of p(words), we instead update the probability of words as follows:
p(words) _ ~; p(words ~ user needs * p(user needs This computation takes advantage of a knowledge base of context-specific language models that contain information of the form p(words) _ ~; p(words ( user needs.
At run-time, actions are analyzed by the Event Monitoring System and are passed to the Inference System which provides dynamically the continually recomputed probability distribution over user needs, p(user needs ~ events) as described above. One exemplary application is the reasoning about help topics, and context specific language models contain information of the form p(words ~ relevant help topic).
In a more general statement showing the modification of the above equation for speech recognition, we input dynamic events p(words ~ speech, events) _ [~; p(words ~ user needs;, events) * p(user needs; ~ events), * p(speech ~
words) p(speech) This can be reduced to:
p(words ~ speech, events) = p(words ~ events) * p(speech ~ words) p(speech) Figure 23 displays the key component and flow of analysis employing key components of the intelligent user assistance facility to enhance speech recognition by dynamically changing the likelihoods over words, and thus, phonemes uttered. As displayed in the figure, user actions are tracked by an instrumented program 72, and atomic events are composed into modeled events by the Event Processor 74 and passed to the Inference System 76. The inference system computes a probability distribution of the needs of a user and the weighted list of needs are passed to a Dynamic Language Model Generator 300.
The Dynamic Language Model Generator 300 has a mapping between alternative needs and the likelihood of utterances or words. This component may ;
5 contain information from a Language Model Database 301 that contains language models containing probabilistic information of the form, p(words ~ user need), the probabilities of different words or word strings being uttered by a user given the user needs. These language models can be constructed through training with statistics from a large corpus of strings of words that are recorded in different user-needs contexts. At 10 run-time, the Inference System provides a probability distribution over user needs which is used to control the creation of an updated language model, based on weighting the language models from the different contexts. The updated language model contains information of the form, p(words J events), created by weighting sets of words in terms of the likelihood of alternative needs based on the probabilistic inference 15 results.
Overall, taking advantage of context in listening can enhance the accuracy of speech recognition system 303 by lowering the probability of utterances that are unlikely and raising the probability of utterances that are more likely, per the user modeling.
20 Beyond use with software in receiving assistance, the dynamic speech understanding has application in variety of speech understanding tasks. As one example, the methods have application in a variety of interfaces with command and control, referring to the manipulation of controls and functionality of software. If we model the goals of the user, we can generate grammars that are appropriately weighted 25 to reflect the next steps of the user, and there are less prone to error.
These methods can be used in conjunction with models that generate the probability of the next software command given user goals. In this approach we focus on a particular user need, the desire for action. The user's desire for action may refer to the implementation of one or more software controls by speech, including the access of 30 one or more help topics. We substitute p(desired action J events) for p(user needs J
~6 events) in the above equations, and employ language models that contain information p(words ~ desired action), instead of p(words ~ user needs). Events considered in this model can include context which may include information about the window that is active and the text, buttons, and controls that are displayed to the user while the speech_ utterance is occurring.
For example, dynamic language models for command and control have application in systems that explicitly generate and display probabilistic results as part of their functioning. Consider as an example, a probabilistic reasoning system that employs a Bayesian network or influence diagram for assisting a physician with diagnosis from a set of symptoms. Bayesian reasoning systems in medicine may have sets of windows containing the results of diagnostic inference. A window may list diseases by computed likelihoods, another may list the best next symptoms or tests to evaluate, based on value of information. By building or assessing a model of the probability of the user's actions given the context of displayed information, which may include functions of the computed probabilities and expected utilities assigned to displayed objects, we can compute and use a dynamically re-computed p(words events) in speech recognition.
While the present invention has been illustrated with reference to exemplary embodiments, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the intended scope of the present invention as defined in the appended claims. For example, the general event composition and inference system may also be used in an operating system to provide an intelligent user shell and, more generally, to optimize the functionality of any computer system or software. Because of the variations that can be applied .to the illustrated and described embodiments of the invention, the invention should be defined solely with reference to the appended claims.
A modeled event composer 104 periodically surveys the atomic events in local storage 102 in search of one or more atomic events that may be composed into a modeled event. This typically would be initiated by the inference system 74 at the beginning of an inference cycle; however, the modeled event composer 104 could 15 perform composition of events between inference cycles such as whenever a new atomic event is received, or when special trigger events are received. The modeled event composer 104 then forms modeled events from atomic events according to the event composition rules defined in the event specification. The modeled event composer 104 stores each modeled event that the composer 104 generates from atomic 20 events in the modeled event database 108.
Those skilled in the art will recognize that there could be implemented a modeled event database for each individual program employing the intelligent user assistance facility. For example, there could be a modeled database for each instrumented application program resident on a computer system. Those skilled in the art will also recognize that there may additionally be a separate general modeled database that contains common system or application events such as activating a program or exciting a program. In such a case, the inference system may access both the general modeled event database and the active program's modeled event database during a cycle of inference.
In one example of a modeled event database, there is a single record for every modeled event for that database. When a program with the intelligent user interface is activated, pre-existing persistent events stored in the user profile may be used to initialize the modeled event database. As modeled events are formed by the composer, the record for that event is updated in the modeled event database.
An example of a record format for modeled events in this modeled event database is:
Fields Name Type Description EventName string Name of modeled event Occurred Boolean Flag which indicates that the modeled event occurred within its horizon.
Atomic Boolean . Flag which indicates that the modeled event is derived from a single atomic event.
StartTime real Lowest start time on any atomic events from which this modeled event is composed.
EndTime real Highest end time on any atomic events from which this modeled event is composed.
Value real This field holds the result of the evaluation of an expression LastTime real The time of last occurrence of the modeled event Count real Total number of occurrences over all sessions LastTimeThisSession real Time of last occurrence this session CountThisSession real Number of occurrences this session Persistent Boolean Flag which indicates that this is a persistent modeled event.
Of the fields used to count the number of occurrences and the rate of their occurrence, LastTime and Count are stored in the user profile database for all persistent modeled events. Upon initialization of the program, these fields get loaded from the user profile database into their corresponding modeled events in the modeled event database.
Modeled events are only relevant for a comparatively short time. This period of time during which a modeled event is relevant may be referred to as its horizon. Different events may have different horizons, a si.Yty second horizon is a suitable length of time for the horizon of most events generated by user actions. The duration of the horizon determines the length of time that a modeled event is considered in the inference system 76.
. Each modeled event may be assigned a default system or application horizon or it may be assigned a specific horizon. Modeled events that have resided in the modeled event database 104 beyond their horizon time are discarded whenever the modeled event database 104 is accessed or updated. However, before they are discarded from the modeled event database 104, persistent modeled events are recorded in the user profile. The inference engine 76 and user profiler, in general, act as event sinks and access the modeled event database 104 to extract modeled event records.
However, there is an exception. At program initialization time, pre-existing persistent events stored in the user profile may be used to initialize the modeled event database 104 and, as a consequence, modify the program's behavior.
Event Svstem Specification Tool The event specification tool allows for rapid creation of the modeled event composer and modeled event database for specific application programs during creation of an intelligent user assistance facility for that software program.
A language was developed for constructing modeled events (e.g., menu surfing) from atomic events made available by a program, for instance an application (e.g., multiple menus visited without action within 10 seconds, position of cursor on spreadsheet, etc.).
The language allows atomic events to be used as modeled events directly, as well as for atomic events to be formed into higher-level modeled events. Also, modeled events can be further combined to create other modeled events. More specifically, the language allows for Boolean combinations of atomic events and for combining the Boolean events into modeled events with operators that capture notions of temporality. The event -specification tool can be linked directly with a Bayesian network modeling tool so that the modeled events and their definitions can be generated while building and refining the Bayesian user model.
Figure 7 is a block diagram of an event system specification tool 110 used for generating code for the modeled event composer 104 and modeled event database definitions 118. Each modeled event definition 112 created during the step of defining modeled events is used to specify the events in the event language 114. The event specification 114 is then input to an event language translator 116 which translates the event language statements of the specification into high-level language code, such as C, to create the modeled event composer 104 component. The event language interpreter also creates the modeled event database definition. As part of that process, the event language interpreter 1 I6 may also create the modeled event database 106 directly from the modeled event database definitions 118 in systems employing a database structure to store information on the status of modeled events.
The event language includes Boolean and set operators, with the addition of temporal operations including dwell and sequence. Modeled events are defined in a separate text module written in the event specification language. However, they can also be constructed with the use of special definition forms. The text definitions are created as named statement blocks for each modeled event. These definitions contain the labels of atomic events and indicate operations on one or more atomic events.
The general form of a modeled event declaration in the event definition language is:
declare ModeledEventName <e:cpression list }
An example of the modeled event declaration for the Dwell on Chart modeled event is:
declare Dwell on Chart { _ _ (and (Select Chart, Dwell(S,s)) The interpreter translates the syntax statement of the event definition language to the modeled event name, Dwell on Chart and updates the modeled event database by adding a record with that event name and initializes all record fields. A
modeled event may be a complex synthesis of several other events, including other modeled events, or it may be simply a re,~aming of an atomic event. The event synthesis language supports infix expressions which entail logical combinations of atomic events, modeled events, arithmetic, and logical operators. As each elementary operation is performed, the intermediate result is treated as an event and has similar properties.
The event language has logical, set-theoretic operators, temporal, and arithmetic operators. Operators allow the definition of specific events as well as events defined as generalizations based on sets of events. For example, higher-level events can be built via abstraction or generalization with operators like Element:
Element({xi,...xo}): Any event drawn from the elements of the set of events {x,,...xa }
occurs. Such sets can be defined as specific classes of events.
Temporal operators allow us to define events in terms of sequences of events over time. Temporal intervals may be measured in terms of the number of user actions, the amount of time, or other measures of duration that can be defined as functions of user actions and time. For example, it can be useful to define modeled events in terms of Scaled seconds, a measure of duration where seconds are scaled by the rapidity at which a user is working, so as to adapt the temporal constants to users that work at faster or slower rates than an average user. When specifying temporal operators, we indicate the dimension of the measure of temporal interval by specifying the temporal dimension d with a value (e.g., c for commands, s for seconds, and ss for scaled seconds). Some useful temporal operators are:
Rate(x,t,d): At least x atomic events occur in duration t of dimension d.
Ensemble N({x,,...~,},t,d) At least N events from the set of events{xl,...xo} occurs within interval t of dimension d.
All({x,,...xt,},t,d): All events in a specified set of events {xl,...xn} occur within duration t of dimension d.
10 Sequence({xl,...xn},t,d): All of the events specified in a set of events {xl,...xn} occur within duration t of dimension d.
TightSeq({xl,...xo},t): All of the events specified in a set of events {x,,...xo} occur within occur within duration t of dimension d.
Dwell(t,d): No user action for t seconds or scaled seconds of dimension 15 d.
In addition, parameters can be specified that define the persistence or dynamics of probabilistic relationships between modeled events and other variables in a Bayesian network with increasing amounts of time since the modeled event has occurred. In the general case, we can provide any temporal function that describes how 20 the probabilistic relationships of events to other variables in a Bayesian network change as the time following the occurrence of an event increases.
More specifically, the language allows the specification of how p(E; ,to ~H~, tp), the probability of event E; occurring at time to in the past conditioned on the truth of hypotheses H~ at the present moment, tp changes with the increasing amount of 25 time tp = t since the event last occurred.
Useful distinctions for representing the temporal dynamics include horizon and decay:
Horizon: The interval of time beginning at the time t=to that an event becomes true, that the probabilistic relationship persists as p(E; , to ~ H~, to) without change. The probability remains unchanged as long as difference between the present moment, tp, and the time, tp, the event was most recently seen is less than the horizon.
Dynamic: The time-dependent functions that describe p(E; , to ~ HJ, t~ and p(E; , to ~not(HI), tp) after a horizon is reached. The dynamics can be summarized with a description of the change in the likelihood ratio defined by the ratio of these probabilities. The functions indicate how the initial probabilities p(E; , ~ ~
H~, t~ and p(E; , to ~not(li~), t~ change as the time the event became true becomes increasingly distant from the present time, tp. It can be asserted that these probabilities converge at some time to p(not(Ey H~, ~ and p(not(E~~not(H~), tp), and the initial likelihood ratio, when tp = t~, converges to the ratio of these latter probabilities, and, therefore, functions can be assessed that decay the probabilities or likelihood ratio from the initial likelihood ratio when tp = to to the likelihood ratio associated with absence of the event.
The probabilities can be computed as a function that takes as arguments the initial probabilities (at t 0), the probabilities associated with the finding being absent, and the time of the present moment, tp, and ensures consistency among probabilities at any moment. That is, P~~ ~ ~ ~ H;~ ~)=f CPS; ~ to ~ H;~ ~)~ P(no~~l H;~ ~)~ ~]
and p(E;, to ~ not(H~), tp~-f [p(E; , to ~ not(H~, t~, P(not(E~I(not(H~), tp), tp]
Operators being used to define modeled events may affect the duration information of the resulting modeled event. Conjunction results in storing the latest occurrence time of any subordinate events. Disjunction results in storing the times associated with the first true subordinate event.
The Value field of the generated intermediate result of the modeled event is set according to the result of the expression. The result of all arithmetic comparison operations is either one (Boolean TRUE) or zero (Boolean FALSE). If the argument of an arithmetic operation is an event, the argument is evaluated as one if the event has occurred and zero if the event has not occurred. The time stamps associated with simple numbers are set to the span of the current cycle; in other words, the start time is the time of the last cycle and the end time is the current time.
Since the syntax allows for recursion, care should be taken that infinite recursion not occur. These statements are entered into a text file known as-the event specification module.
The event specification module 114 is then processed by the event -definition language translator 116. The event language translator converts the event specification language 114 into C++ code which may be compiled to machine language and linked directly to the event processor system library, IUAF library, operating system library, or other implementation library. The results of translating a high-level event specification is the modeled event composer 104 which creates modeled events.
When the modeled event composer 104 is invoked during a cycle of inference, it builds the set of currently active modeled events and then returns to the caller. The inference code then performs appropriate actions indicated by the event set.
User Profile Svstem The inference system 76 accesses the user profile system 78 to check for competencies and changes assistance that is given based on user competence.
User's background, successful completion of key, tasks indicative of competence in particular areas, and previous help reviewed by the user are variables that can be stored in a persistent file and updated with time. Such persistent information about user background, experience, and competence is referred to as "profile information."
Figure 9 is a block diagram of the user profile system 78. The user profile system 78 includes a user profile database 120, a user profile access routine I22, an event status update routine 124, a background and competency dialog routine 126, and a customized tutorial routine 128.
The user profile database 120 contains records identical in format to the records in the modeled event database 106 so that there is a common format between the modeled event database records and the user profile database records. This common format makes it more convenient both for updating the user profile database with modeled events and for the inference engine to interpret these records when it access these databases for inference analysis. The user profile database records are stored persistently so that they create an historical record of particular userrs competency with specific software applications. This database is maintained on a by-user-by-application basis. An example of persistently stored modeled events that are indicative of user competency are the completion or non-completion of certain tasks, successful or unsuccessful use of particular features, and assistance received or help information reviewed in the past.
Figure 10 is a flowchart illustrating how the user profile database 120 is updated with event status of a user interacting with a specific application program: The event status update routine 124 in step 130 retrieves the first modeled event record in the modeled event database and checks in step 132 whether the record is marked as a persistent record. If it is marked as a persistent record, then the record is added to the user profile database in step 134. If it is not marked as a persistent record, then it is not added to the user profile database. Each subsequent record in the modeled event record database is retrieved in turn (steps 136 and 138), checked whether the record is marked as a persistent record (step 132), and added to the user profile database if it is marked as a persistent record (step 134).
The availability of a standard stored user profile containing information about "persistent events" could follow a user around over a local network or over the Internet, custom-tailoring the user's software wherever the user may be. While a software application is active, a new user may log on to the system and use the active software application. Figure 11 is a flowchart illustrating how a user profile is accessed whenever a new user signs on to the system or application. When a user logs on to the system or application (step 140), the user profile access routine 122 searches the local user profile database for that user's profile (step 141 ). If the profile is found, the path to that user profile is passed to the inference system 76 (step 142). If that user's profile is not found, because that user may be using the application at a remote location, then the user profile access routine searches the available networks for any user profile databases that contain the user profile (step 143). If the user's profile databases is not found er the user's profile is not found in any user profile databases accessed, then the user profile access routine asks the user for permission to enter a user competency dialog with the user (step 144) and creates a user profile for that user (step 145). The path for that user profile is passed to the inference system 76 (step 142). If the dialog is not-completed, the path to default profile information is passed to the inference system 76 (step 146) and used.
In addition to maintaining a persistent user profile for use during real-time inference, a continuing background analysis of repetitive patterns of a user's needs for assistance during one or more sessions that exceed a threshold, may be used to customize a tutorial of help information that may be offered to the user at the end of a user's session with the that program. Figure 22 is a flowchart of the routine for generating a customized tutorial. During a user's session with a specific application program, a histogram of help topics exceeding a relevant probability threshold is recorded by the inference system 76 (step 150). When the user asks to quit the session with the application program, the user profile saves the histogram in the user's profile (step 152) and prepares a customized tutorial of help information related to the help topics recorded in the histogram (step 154). The user profile system notifies the user upon exiting the application program that a customized tutorial was prepared for online review or for printing as a manual (step 156).
Inference Svstem When a cycle of inference analysis is initiated, the inference system 76 accesses the modeled event database 106 to extract modeled event records for each of the modeled events that has occurred since the last cycle of inference. A
functional block diagram of the inference system is displayed in Figure 13. The inference system 76 includes one or more knowledge bases 160 and an inference engine 165.
Knowledge bases 160 include information relating variables that represent observable states of the world, such as user actions and words, to variables that represent hypotheses of interest about a user's goals and needs for assistance that may not be observed directly or that are costly to observe. For example, the goals of users can not necessarily be directly inspected, but a sequence of computer commands can be recorded. A user can be actively queried about goals, but this may be quite distracting to users.
A deterministic knowledge base consisting of an interrelated set of logical rules may Link observations to hidden hypotheses. However, it is often more appropriate to process the uncertain relationships as probabilities between observations and the likelihood of hypotheses about a user's needs. For example, in attempting to -5 understand and to predict the behavior of a complex system such as human physiology it is typically not possible to completely model with deterministic relationships all components of the system, and to then have access to a deterministic model for performing diagnosis or designing therapy based on a set of symptoms. For diagnosis and decision making about complex systems, we are often forced to reason under 10 uncertainty, and to explicitly address the incompleteness in our understanding.
Probability provides us with a means of diagnosis and forecasting about the behavior of complex systems given knowledge about a set of relationships among observational variables and hidden variables we may identify as being important. We can use probabilistic methods to represent and reason about weak and strong uncertain 15 dependencies among observations, such as symptoms of a patient and variables such as diseases in a patient. A good example of complex system is a user attempting to perform a task while interacting with a computer software application or system. It is very difficult to build deterministic models that Link a user's behavior to a user's goals and intentions. The best way to diagnosis a user's needs is to develop appropriate 20 abstractions, based on our understanding of the relationships among various kinds of user's goals and actions, and to represent and reason about the relationships with probability. Probabilistic methods allow us to build models at a level of abstraction that is appropriate in light of our incomplete understanding about users and their actions.
A Bayesian network or a generalization of a Bayesian network, called an 25 influence diagram, can be employed to represent the certain or uncertain relationships among user actions and such hidden, but important, states as user goals, user intentions, and user needs, given observable information such as one or more user actions.
A
Bayesian network is a directed acyclic graph where nodes are random variables and arcs represent probabilistic dependencies among those variables. Variables in a Bayesian 30 network are chance variables or deterministic variables. A Bayesian network represents a joint-probability distribution for the set of variables it represents. The probability distributions over the values of chance variables depend on the values of direct ancestors, or parent variables that are parents of nodes. The value of deterministic variables is a deterministic function of predecessors. Influence diagrams are a generalization of Bayesian networks which represent additional nodes that represent possible actions and the utility of outcomes.
The inference engine 16~ includes inference procedures 166 that operate on the knowledge base 160. The knowledge base 160 includes a Bayesian network 162.
Those skilled in the art will recognize that the knowledge base may be a deterministic knowledge base with logical chaining procedures as the inference procedures of the inference engine. Or the knowledge base may be a Bayesian influence diagrams with inference procedures for operating on Bayesian networks as the inference procedures of the inference engine. Temporal reasoning procedures 167 and value-of information procedures I68 are also included as part of the inference engine 16~. Other specialized procedures may be included.
Bayesian inference procedures operate on Bayesian networks to compute a consistent posterior probability distribution over the values of variables in the network given the setting of states of observable, or evidence, variables to specific states based on the observations. Assume we are interested in the probability distribution over unobserved hypotheses of interest, H,,... Hm, and have access to observations and profile information E,,... En. Bayesian network inference algorithms compute a probability distribution over H given the observations, written p(H~ Et,... EJ. For influence diagrams, decision-theoretic inference is performed to identify the expected utility of alternative actions. The best action is the one associated with the highest expected utility.
In the most general sense, the task of designing inference for an intelligent user assistance facility is best undertaken from the perspective provided by decision theory, an extension of probability theory to reflect concerns of value under uncertainty. Influence diagrams allow us to express fundamental relationships about uncertainty, action, outcomes following action, and the value of those outcomes. This representation is useful in the design, understanding, and in many cases, the actual implementations of intelligent user assistance systems. However, it is often the case, that a simpler systems may be built without explicitly representing decisions, outcomes, and the utility of outcomes, and instead to use Bayesian networks to represent probabilistic relationships, and to use probabilistic inference in combination with procedural controls and thresholds as approximations of more complex influence diagram models.
A general influence diagram for user modeling and action to assist users of software is portrayed in Figure 14 as an exemplary embodiment of a Bayesian influence diagram as a knowledge base. As portrayed in the figure, a user's background 170 influences with uncertainty, a user's overall goals 171 in using software, as well as the user's knowledge 172 in using software. User's knowledge I72 is also influenced by the previous help 173 that user may have seen. The user's background 170 and previous help 173 are variables that can be stored in a persistent file and updated with time. Such persistent information about user background, experience, and competence is referred to as "profile information." As indicated in the influence diagram, the user's goals 171 and knowledge 172 in turn influence with uncertainty, the informational needs 174 of the user. T'he goals 171 and needs 174 in turn influence the sensed activity 175 and the words that might be used in a query I76 to the software, or software's help system. A
utility function of the user 177 is represented as a diamond. The utility 177 is influenced directly by the informational needs 174 of the user, the cost of taking autonomous action 178 (e.g., distraction for the user's current focus of attention) and the action that is taken. Several classes of action may be available to the system, including the providing of advice or help 179, the execution of software actions of various kinds 180 and the acquisition of additional, previously unobserved information from the system, 181 or directly from the user. As indicated in the Figure 14 by the scale on the cost of assistance variable 178, we may wish to allow the user to directly change the cost of the assistance so as to control, in a fluid manner, the degree of autonomy given to the system.
The overall goal of the inference system is to identify actions that will optimize the user's expected utility given the user's needs and the cost of taking autonomous action. Given a set of evidence E about a user's background and actions, the probability distribution over hypotheses of interest about a user's needs, H, is , computed. In the case of the Bayesian influence diagram in Figure 14, the probability distribution over a user's needs for assistance is computed. The expected value of actions in the set of all possible actions A must be considered. To do this, consider the utilities over outcomes, represented as a utility model (diamond node). The utility model contains information about the value or utility of outcomes. Outcomes (A,H) are defined as doublets of actions A taken and the state actual user's needs H.
The utility model tells us the utility associated with each outcome, u(A,H). The best action to take, A*, is the action that ma.~cilnizes the expected utility under uncertainty, computed as follows:
A* = arg maxA ~~ u(t~,H~) p(H~~E) Although influence diagrams represent explicit models of action, Bayesian networks are often easier to build and can be used for decision making and action by employing thresholds or rules about probability over variables in the network to indicate when actions should be taken based on the degree of belief assigned to a variety of states including states that describe preference about action. Influence diagrams such as the one portrayed in Figure 14 can be used to clarify approximations used in the Bayesian network.
Temporal Reasoning and Dynamics The inference system 76 also contains special temporal reasoning knowledge 164 in the knowledge base 160, and procedures 167 for performing inference about the changing relationships of observations to other variables in the model as the observations occur at progressively more distant times in the past. In the general case, applications of Bayesian procedures and knowledge bases to reasoning about patterns of observations seen over time requires the consideration of variables and their interdependencies within a single time and at different times. Building and performing inference with Bayesian networks that include copies of variables and dependencies for different times slides can lead to difficult computational reasoning problems and unacceptable response times for a user interacting with a program. Thus, several approximations are of value. In one kind of approximation, only specific dependencies considered to be of the greatest importance are considered over time.
A portion of a Bayesian network with sets of variables at different times and prototypical dependencies represented is displayed in Figure 15. Figure 15 displays a Bayesian network that explicitly represents random variables at different points in time, aad probabilistic dependencies among variables at a single time and among variables at different times. The lack of dependencies among variables are assertions of assumptions of independence. The figure displays possible influences among the primary relevant assistance to provide a user at present moment with the most relevant assistance to provide users at earlier and later times. Also, the figure highlights potential dependencies among the status of observations (E; and E~) in the present, with observations made in the past and future. The figure also indicates the relationship between the primary relevant assistance in the present and observations in the past. In the general case, such multiple-connected networks are difficult to solve and to assess during the construction of the models.
We will now describe an approach to reasoning about relevant assistance over time that is more tractable to solve and to assess, through making additional assumptions of independence and representing the relationships among evidence seen at progressively more distant times in the past and hypotheses of interest with parameterized functions that dictate the strength of the probabilistic relationship between those observations and current the goals or needs for assistance. The use of this approximation in an exemplary embodiment employs a single explicit Bayesian network, but allows the system to consider implicitly multiple Bayesian networks over time. This method takes advantage of direct assessment during modeling time of functions that adequately describe the dynamics of the probabilistic relationships between observed variables and other variables in a Bayesian network. At modeling time, a horizon and dynamics are assessed for each observational variable E;, and the variable is annotated with this information. At run-time, the inference engine includes procedures 167 for using this information about horizon and dynamic.
The horizon for an event captures the interval of time the probabilistic relationships with 5 hypothesis H; persists as p(E;, to ~ H~, ~) and p(E;, t~ ~ not(H~), t,~
without change. The dynamics is the time-dependent change of the probabilities in the present moment as described earlier.
Heuristics in defining a horizon for events include the use of a default horizon for all observational events considered by a system unless otherwise specified, 10 and the use of an event queue of finite length k, where only the last k modeled events are considered in the analysis. This approach can be combined with dynamics as just described.
Value of Information 15 The inference engine I6~ also includes expected value of information (EVI] procedures 168 to compute the expected value of acquiring information from a user. EVI procedures and information-theoretic approximations that can be employed in probabilistic models are well-known to people skilled in the art of decision theory.
The EVI is a means for computing the expected value of acquiring information about 20 variables that have not yet been observed. Such information includes answers by users to questions posed by the computer system about their goals and needs. The Net Value of information (NEVI) is net value of gathering information, including the costs of gathering the information. The inference system 76 only asks a question, or gathers information autonomously, when the informational benefits of the information 25 outweigh the costs. If EX refers to each previously unobserved variable that can be queried, O,~ is a possible observational value of Ex, and A is the value that will be observed when the variable is evaluated, the NEVI of each variable is:
NEVI(E,~ _ ~k Pox - Ok ~E) * ~m~., ~; u(~~H;) P~;~E~Ex O
- maxA ~; u(A;,H~) p(H~~E) - C(E~
where C(E~ is the cost of evaluating evidence EX. EVI can be applied directly to influence diagrams. Several well-known information-theoretic algorithms, including those based on computation of entropy, function to provide similar functionality for .
decisions about information gathering in probabilistic models such as Bayesian networks. These algorithms are typically more tractable than computing NEVI
with influence diagrams. They do not explicitly manipulate utilities, but nevertheless can be employed in conjunction with heuristics that balance the cost of information with measures of informational value.
IO Figure 16 is a flowchart of value-of information procedures in the value of information component. The value of information component lists all of the previous unobserved information in order of the value of it as alternative information (step 190) and weighs the benefits of obtaining the information against the cost of distracting the user for each item of information (step 192). If the benefit is greater than the cost set by the user, the IIIAF asks the user to supply the item of information that has the highest value (step 194). If the user responds, then the value of information component repeats the first two steps until either the cost of distraction is greater than the benefits of obtaining the information or the user does not respond (step 196).
Scenarios Illustrating the Screen Outuuts of the ILTAF
Figures 17 through 23 illustrate the screen outputs of the intelligent user assistance facility in an exemplary embodiment of a spreadsheet application.
The system considers a user profile, user-specified threshold information, and combines words and user actions to provide intelligent assistance to the user. The system has the ability to respond to requests for assistance as well as to autonomously provide assistance. Figure 17 shows an assistance interface 200 that is generated when the user explicitly has requested help. Guesses about useful assistance to provide the user are displayed in the list box 201 in the upper left-hand corner. The current system settings box 202 in the lower left-hand corner indicates the user is an expert user.
This information was provided by the user profile system. The current system setting box 202 also indicates that both user actions and free-text queries are used by the inference engine during a cycle of inference analysis. Given the profile information and the recent actions involving interaction with the layout of text, in combination with pauses, and with the user taking action to change the appearance of rows and columns shortly before help was actively requested, the system believes the user may best benefit by _ receiving assistance on working with fonts, advanced formatting instructions, and with ' methods for changing the dimensions of the spreadsheet document. To the right of this screen photo is displayed an inference graph 203 showing the probability distribution over a set of task areas that was generated by the Inference System 76. This display was built for engineering purposes but may be displayed to the user. The length of the IO bar graph next to each task area is the probability assigned to that area.
The two highest probability areas (those with the longest associated bar charts) are for the tasks of working with fonts and advanced formatting topics.
In Figure 18, the profile information was changed to a profile for a novice user. The expertise level in the current systems setting box 202 indicates that the user has a level of expertise of a novice. All other events handling remained unchanged. We have used a profile for a novice user. The profile information is used to modify the probabilistic relationships in the Bayesian network. The inference graph 203 displaying the probability distribution over relevant assistance shows the revised probability distribution with the profile information. As can be seen in the best guesses list box 201, the system now believes that the user may most benefit by assistance with basic working with rows and columns of the spreadsheet, changing alignment and basic formatting of charts.
Figure 19 shows the analysis of the same set of user actions, but now, with consideration of the words in a user's query. The user has input to the system the natural-language query, "How do I make this look prettier?" in the query input box 204.
The analysis of the words and the actions are combined to generate a new probability distribution, as indicated in the inference display 203, and a new corresponding list of recommended topics in the best guesses list box 201. Now, the list is resorted and updated with autoformatting assistance being recommended as the most relevant assistance, but also still contains other formatting topics, including changing alignment, working with borders, and working with fonts.
Figure 20 shows the program at a later time when help is again requested after interaction with a chart in the document, coupled with pauses on chart commands. _ The probability distribution computed by the inference system 76 is displayed in the ' inference graph 203. Now, the recommended topics in the best guesses list box focus on a variety of high-Likelihood areas in the area of graphing information. Figure 21 shows the addition of information submitted in the form of a query in the query input box 204. The natural language query from the user is, "I need to access data from a different application". The words are now considered in conjunction with the events.
The inference system 76 provides a revised probability distribution in the inference graph 203 and a revised list of recommended topics in the best guesses list box 201, centering in retrieving data from a database and about changing the data displayed in a chart.
Figure 22 illustrates the screen outputs of the inference system 76 offering autonomous help to a user. While the user has been interacting with a spreadsheet program 205, an assistance monitoring agent computes the probability that a user needs assistance and the inference system 76 also computes the likelihood of alternative kinds of help to give the user, should the user require assistance. The assistance monitoring agent displays the result of inference on the probability that a user needs assistance in the assistance monitoring agent window 206. Like the inference graph 203 display described above, the assistance monitoring agent 206 was built for engineering purposes but may be displayed to the user. The user who has been interacting with a spreadsheet program 205, now selects the whole sheet and pauses. As the assistance monitoring agent window 206 shows, the computed probability that the user needs help moves from 26% to 89%, reaching a threshold, and a small timed window, the autonomous assistance window 207, autonomously comes forward displaying the highest likelihood assistance areas that were computed. In this case, the highest likelihood topics include formatting cells, checking for spelling errors, and performing calculations, topics that have implications for analysis or modification of the entire document. The autonomous assistance window 207 politely offers help. It also provides a chance for the user to reset a threshold that will increase or decrease the probability required on the likelihood the user needs help before displaying the window.
The window 207 will time out and be removed with an apology for distracting the user if the user does not interact with the window 207.
Figure 23 illustrates the display 208 of the topics that have been determined to be of value to the user based on the recording and summarization of ongoing background inference about problems that user has been having during a session using a software program. Profile information about the user's background, competence, and about the assistance that the user has received or reviewed in the past are also included. These topics have been prepared for printing as a custom-tailored manual for later perusal.
Bavesian Network Annotated with Temporal Reasoning Figure 24 shows an example of a Bayesian network that computes the probabilities of alternative forms of assistance to provide a user as a function of profile information about the user, as well as recent actions taken by the user. To indicate that this structure may be repeated for many classes of observations and task-specific needs for assistance, variables x, y, z, and A have been used in several of the nodes.
In this model, we represent profile variables that represent the status of user competency in two different areas of software functionality, x and y, with a set of variables labeled User Competence x 210 and User Competence y 211. The user competencies are each influenced by variables representing (1) specific observed activities by the user, (2) the history of help topics that have been reviewed by the user, and (3) the overall user background 212 that may be uncertain or set by a dialogue with the user. As indicated in the Bayesian network, the user competencies directly influence the prior probability of different assistance being relevant to the user, as represented by the states of the Primary Assistance Needed variable 213. This variable is also influenced by a context variable 2I4, that describes information about program state, such as the existence in the current version of an application of a particular data structure. In this model, the problem is formulated with a single primary assistance variable, with states representing alternate forms of assistance. The model assumes that there is only one state of primary assistance the user needs at any particular time.
Multiple variables about assistance can also be represented to capture the notion that -5 several types of assistance may all be relevant at a single time.
As depicted in the Bayesian network, the Primary Assistance Needed node 213 influences a set of observed modeled events, such as selecting a graphical object Z, then pausing for more than some predefined period of time (Sel Z
Pause 215), or modifying an object and then performing an undo, (Z Cmd Undo 216). At run time, 10 these modeled events are detected and passed to the Inference System 76 by the Event Processor 74. The corresponding observable variables to the modeled events are update and Bayesian inference is performed to update the probabilities over ali of the unobserved variables in the system, including the Primary Assistance Needed variable 213. The time of the event is noted and as the event flows progressively into the past, 15 temporal reasoning procedures 167 are applied to update the probabilistic relationships in the network.
Figure 25 shows an instantiation of the nodes with variables x and z with distinctions with relevance to reasoning about the relationship of actions and profile information on charting or graphing information from a spreadsheet, such as the 20 Microsoft Excel product. Variables that are persistent and stored in a profile are marked with an adjacent P. Variables that are observed as modeled observable events are marked with M. One or more nodes may represent various classes of competency with charting. In this case, a single variable, User Competence Charting 220, is displayed as the variable that contains information about the overall level of charting ability of the 25 user. The probability distribution over the states of this variable are influenced by Hx Help Charting 221 and Hx Activity Charting 222, and User Background 212. Hx Help Charting 221 is set by observations that are stored in persistent profile about one or more charting activities the user has completed successfully and information about the help information on charting in an online manual that the user has reviewed.
30 Competencies about charting and a variety of other areas of software functionality influence the Primary Assistance Needed variable 213. This variable in turn influences the probability of observing different behaviors by the user. For example, the state of a user needing assistance on a specific type of charting task, influences the probability of seeing the user select and pause for s seconds on a chart in the document (Sel Chart -Pause 223). Appendix 1 contains a more complete Bayesian network in an exemplary embodiment of a spreadsheet application.
Figure 26 represents an exemplary embodiment of temporal knowledge and inference. Each observable modeled event is annotated with information about the effects of time flow on the probabilistic relationships of the event with other variables in the Bayesian network. In this case, each node is annotated with temporal dynamics information about a horizon and decay, which may be implemented as a table of probabilities or other data structure. The dimensions of time for the temporal dynamics of each variable are indicated in seconds that have occurred since the event became true.
We now provide an example of a case based on components in the knowledge base displayed in Figure 26.
Let us consider a case where we have the following profile information:
User_Background 212: Consumer Novice HX_Help Charting 221: No_Help Reviewed Context: Normal_View Obj ects: Sheet( 1 ),Chart(2) At time t=50 seconds, we have noted the following modeled events:
Chart_Cmd_Undo 224: Absent Chart_Create_Pause 225: Absent Sel_Chart_Pause 223: Absent Move_Chart_Pause 226: Absent Chart_Dialog UnSucc 227: Absent Menu_Surfing 228: Absent Click Unrelated 229: Absent Sel Chart Pause 223 P~utoIH;~~=.15 p(not(E,)I H~, to) _ .85 p(E,, tv Inot(H~), t~ _ .005 p(not(E,), to I not(H~, t~ _ .995 Horizon: 5 seconds Dynamics:
p(E,, tv I H~ , to): linear convergence at 15 seconds p(E,, to (not(H~), t~: linear convergence at 15 seconds Chart Dialog Unsucc 227 P~z~ ~ I H; ~ ~) _ .08 p(not(E~I H~ , t~ _ .92 p(Ez, to I not(H~), t~ _ .0005 p(not(E~, to I not(~H~), t~ _ .9995 Horizon 20 seconds Dynamics: 0 A cycle of analysis is initiated at t=50 seconds. The cycle includes the composition of modeled events from atomic events local storage 102, the transmitting of the events to the Inference System 76, and inference with the new observations. We will explore the current probability that the primary assistance needed by the user is help with charting information, p(Charting AssistancelE). The prior probability of the user needed help in this area conditioned on the user background, competence, and history of reviewing help is .001. The context information has increased this to a probability of .04. Probabilities are also assigned to other hypotheses about the primary assistance as a function of all of the evidence seen so far (but that is not explicit in this example).
At t=52 seconds the user selects a chart. The atomic event of chart being selected is detected and is stored in the atomic event local storage 102. The user then pauses for 3 seconds without doing anything. A dwell of 3 seconds is detected in the event local storage 102. A cycle of analysis is called at t--60 seconds. A
modeled event Sel Chart Pause 223 is composed by the Event Processor 74, tagged as becoming true at t=55 seconds, and is sent to the Inference System 76.
~ The Inference System accesses the Bayesian network that is annotated with temporal dynamics information. As indicated in the table of probabilities for Sel Chart_Pause 230, there is a horizon of 5 seconds, and a linear convergence of probabilities to the probabilities of absent at 15 seconds.
At t=63 seconds, the user selects a chart dialog and cancels the dialog with out success at t=66 seconds. A cycle of analysis is called at 70 seconds, and the modeled event Chart Dialog Unsucc 227 is composed and passed to the Inference System 76 with a time stamp of 66 seconds. As indicated .in the table of probabilities for Chart Dialog Unsucc 231, there is a horizon of 20 seconds, and no temporal dynamics.
This user has set the threshold for being bothered with automated assistance at p=.12. Thus, the user is offered assistance on charting when the probability that this topic is relevant rises to p= .14.
At t = 50 seconds p(Sel Chart Pause, to ~ H~ , tp) _ .1 ~
p(Sel Chart Pause, to ~not(H~), ~ _ .005 p(not(Sel Chart-Pause) H~ , tp) _ .85 p(not(Sel Chart Pause)~not(H~, t~ _ :995 p(Chart Dialog Unsucc, t~ ~ H~, ~ _ .Og p(Chart Dialog Unsucc, ~ ~not(H~, ~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~), ~ _ .9995 p(not(Chart Dialog Unsucc)~ Fig, t~ _ .92 Inference result: p(Charting Assistance~E, t~ .02 At t = 60 seconds p(Sel Chart Pause, to ~ H~, t~ _ .15 p(Sel Chart Pause, to ~not(H~, t~ _ .005 p(not(Sel Chart Pause)) H~, ~ _ .85 p(not(Sel Chart_Pause)~not(H~, t~ _ .995 p(Chart Dialog Unsucc, to j H~, t~ _ .08 p(Chart Dialog Unsucc, to ~not(H~, t~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~, ~ _ .9995 p(not(Chart Dialog Unsucc)~ H~. ~ _ .92 Inference result : p(Charting Assistance~E, t~= .07 At t = 70 seconds p(Sel Chart Pause, to ~ HJ, t~ _ .15 -> .46 p(Sel Chart Pause, t~ ~not(H~, t~ _ .005 -> .66 IO
p(not(Sel Chart Pause)~not(H~, t~ _ .995 p(not(Sel Chart Pause) Fig, t~ _ .85 p(Chart Dialog Unsucc, to ~ H~, t~ _ .08 p(Chart Dialog Unsucc, to ~not(H~, ~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~, t~ _ .9995 p(not(Chart Dialog Unsucc)~ H~, t~ _ .92 Inference result: p(Charting Assistance~E, tp~ .04 At t = 80 seconds p(Sel Chart Pause, to ~ H~, t~ _ .15 -> .46 -> .85 p(Sel Chart_Pause, to ~not(H~, tp) _ .005 -> .66 -> .995 p(not(Sel Chart_Pause)~not(H~), tp) _ .995 p(not(Sel Chart Pause) H~, tp) _ .85 p(Chart Dialog Unsucc, to ~ H~ , tp) _ .08 p(Chart Dialog Unsucc, to ~not(H~), ~ _ .0005 p(not(Chart Dialo~Unsucc)~not(H~ ), ~ _ .9995 _ p(not(Chart Dialog Unsucc)~ Fig , ~ _ .92 .
Inference result: p(Charting Assistance~E, t~= .13 At t = 90 seconds p(Sel Chart Pause, to ~ H~, t~ _ .85 p(Sel_Chart Pause, t~ ~not(H~, tp) _ .995 -p(not(Sel Chart Pause)~not(H~), ~ _ .995 p(not(Sel Chart Pause) H~ , t~ _ .85 p(Chart Dialog Unsucc, to ~ H~, t~ _ .08 p(Chart Dialog Unsucc, to ~not(H~, t~ _ .0005 p(not(Chart Dialog Unsucc)~not(H~), t~ _ .9995 p(not(Chart Dialog Unsucc)~ H~, t~ _ .92 Inference result: p(Charting Assistance~E, t~= .04 System Level IUAF for Multiple Applications The methods described can not only be applied - to individual applications, they can also be employed at the operating system level to reason about providing assistance in a system with multiple applications. In an exemplary , embodiment to provide system assistance, a Bayesian network is used to assist a user with exiting an application by guessing when the user has completed or will soon be completing a task.
A Bayesian network knowledge base for assisting a user with this task is displayed in Figure 27. Profile information including whether or not the user has reviewed help in past (Reviewed Help in Past 240), has exited before (User Exited Before 241 ), and has demonstrated that they can perform a double-click operation (Successful Double Click 242) are included. Modeled events represented as observations in the Bayesian network include dwell after activity (Dwell after Activity243), sequence of clicks on a sequence of unrelated graphical icons (Click on 1 S Seq of Unrelated 244), mouse meandering (Mouse Meandering 245), whether a document has been recently printed (Print Doc 246) or a message has been sent (Send Msg 247), and the duration of time an application has been up (Duration of Time with App 248). A variable represents a probability distribution over the most desired action, allowing us to use probabilistic inference to reason about the likelihood of alternative best actions.
The sample scenario, demonstrates the results of the effect of monitored events on inference about the user. The sample case assumes the following profile: the user has not exited before on his/her own, the user has not demonstrated a successful double-click action, the user reviewed help about exiting applications but did not dwell on the information. The system now detects that the user has been in email for almost 10 minutes, and that the user has clicked on an unrelated sequence of graphical icons.
Figure 28 displays the states of each random variable, and shows, by the length of bars next to the states, the probability over different states of each random variable. As demonstrated by the bar graph, the Inference System 76 has computed that there is a high probability the user is ignorant about exiting (User Ignorant about Exit 249) and a low probability that the user has or will shortly complete a major task (Major Task Completed 250).
In Figure 29, we see the results of inference after new modeled events are considered. In this case, the user has sent email (Send Msg 247) and has now dwelled for more than three scaled seconds a$er activity (Dwell after Activity 243).
Providing a tip to the user is now identified as the most desired exit-related action (Most Desired Exit Action 251). Figure 30 shows the same scenario with one change:
the user has received help or assistance within the last two minutes (Recent rec Exit Help 252).
Now the most likely desired action is to do nothing, followed by asking the user if the system should exit the application for them (Most Desired Exit Action 251 ).
IUAF Controller There is typically a need for the overall control of various components of the Intelligent User Assistance Facility. Controls are needed for making decisions about the policy for calling the Event Processor 74 and Inference System 76, and for determining when assistance should be provided autonomously. Figure 31 is a diagram illustrating various overall coordinative control modes of the Intelligent User Interface controller that may be used separately or together in combination. The first coordinative control mode illustrated is the timer mode 260 in which a repeated system timer fires to notify the IUAF controller to begin another cycle of monitoring and inference analysis (step 261). The IIJAF controller then invokes the inference system (step 262). Upon completion of the inference analysis, the controller waits until the timer fires again (step 263). The timer mode can be extended to consider other constraints before allowing a cycle to begin. For example, in cases where complex inference occurs during event monitoring and inference, the additional condition of assessing the availability of computational resources (e.g., identifying whether there is idle time) can be added. Thus, the cycle of event processing and inference will occur when a timer fires and when there is also a condition of computational resource availability.
The second mode illustrated is the demand mode 265. This mode only performs inference analysis when the user asks for assistance. When an instrumented application receives a query for help information (step 266), the instrumented application sends notification to the IUAF controller (step 267). As a result of receiving the notification, the ILJAF' controller then invokes the inference system (step 268).
Upon completion of the inference analysis, the controller waits until it receives the next notification (step 269).
The third mode illustrated is the special event trigger mode 270. In this mode, there is a list of special events that trigger a cycle of inference analysis.
Whenever the event processor detects an occurrence of one of the special events (step 271), the event processor sends notification to the IUAF controller (step 272). Again, the IUAF cantroller then invokes the inference system as a result of receiving the notification (step 273). Upon completion of the inference analysis, the controller waits until it receives the next notification (step 274).
Any two or all three modes may be active at the same time. In the case when all three modes are active, if the user has not asked for assistance and has not initiated any user actions that translate into one of the special events during the period of the timer, then a cycle of inference analysis would occur when the period of the timer expired. Otherwise, the cycle of inference analysis would occur during the period of the timer when the user asked for assistance or initiated a user action that translated into one of the special events.
Integration of Analysis of Free-Text Ouerv with Events As mentioned earlier we can treat words in a user's free-text query for assistance as sources of modeled events. The Microsoft Answer Wizard integrated in all applications of the Microsoft Uffice "95 product line makes use of a probabilistic model relating words in free-text query to the probability of relevant help topics. This has been described in U.S. Patent Application No. 08!400,797 (now issued as U.S. Patent No. 5,694,559). Ln one exemplary embodiment, wards and actions can be handled together in a uniform way as modeled events in a system that reasons about the probabilistic relevance of words and actions.
In an alternate embodiment, there are separate subsystems for handling words and for handling actions. The analyses of these separate subsystems are 5 combined after separate probabilistic analyses are completed. Figure 32 is a flowchart of the method for combining separate probabilistic analyses. First a Bayesian analysis of the most relevant help topics is computed for the given free-text query (step 280).
This may be done following the methods described in U.S. Patent Application No.
08400797. The next step is to output a list of help topics in rank order by the highest 10 probability of relevance (step 281). Then a second Bayesian analysis of the most relevant help topics is computed for the given program state, user profile and sequence of user actions (step 282). The next step is to output a second list of help topics in rank order by the highest probability of relevance (step 283). Then both of these lists of help topics are combined, after assigning a weight to the help topics on each list, by 15 computing the probability that each help topic which is the same on both lists is relevant (step 284). The final step is to output the combined list of help topics in order of highest probability of relevance (step 285). In this embodiment, the performance of the system can be tuned by weighting the results of one set of events more heavily than the other by changing the value of a real-numbered parameter that is used to multiply the 20 likelihoods assigned to the probabilities of hypotheses of interest (e.g., relevance of help topics) provided by the word or action analyses.
Building separate systems for analyzing the handling of words in a free-text query and user actions allows the two subsystems to be separately optimized based on specific properties of each class of events. The same can apply to other sets of 25 actions, such as vision and speech events in systems with event monitoring that is extended with these additional sensory modalities.
Separate analyses also allow for the ease of implementing specific policies on combining the results, separating them, or sequencing them. For example, an embodiment might display an initial guess of assistance based on inference based on 30 monitored actions, but when words are added, rely solely on the analysis based on the words, given the performance of a word analysis systems versus an action system. An alternative approach is to have control rules about the combination versus separation of the inference in different subsystems based on a comparative analysis of the results of the words and actions analysis. _ Control of Communicating and Performing Assistance Beyond coordinative control functions, the IUAF controller also makes decisions about when to bother the user with autonomous assistance or with queries fvr information about the user's goals (based on value of information computations). In an exemplary embodiment, a special Bayesian network, or component of a network, resident in the Inference system 76 computes the overall probability that a user would like assistance at the current time. The ILTAF controller uses this changing probability to control when autonomous help is provided. T'he IUAF provides autonomous help whenever the computed probability that a user would like assistance at that time exceeds a threshold which may be changed by the user. A users changes the threshold for offering autonomous assistance using a threshold controller. T'he threshold controller, for example, may be in the form of a sliding threshold control can be displayed to users on an autonomous help window, allowing them to change the threshold depending on the value they find in receiving help versus the distraction it causes. A screen shot of a dynamic assistance window with a threshold slider is displayed in Figure 22.
Figure 33 is a flowchart of the method of providing autonomous assistance in this embodiment. During a cycle of inference analysis, the inference system 76 outputs a list of relevant help topics in order of highest probability of relevance (step 290). This may be done as described previously in the discussion of the inference system. The next step is to determine whether the probability of the user needing help exceeds the threshold set by the user for providing autonomous assistance (step 292). If the probability of the user needing help exceeds the threshold set by the user, then the inference system 76 provides a list of relevant help topics in the order of highest probability (step 294). The list may be truncated so as to only show the most likely or several most likely topics, but within a window that allows the user to scroll through a longer list if desired. Next, the list of relevant topics are checked for redundant help topics that have appeared in the previous list within a specified time period (step 296). If there are no redundant help topics, the list is displayed to the user (step 298). If the user responds to the autonomous offer of help, then steps 290 through 298 are repeated until the user fails to respond before expiration of the response timer.
If the user fails to respond before the expiration of the response timer, then a message of apology is displayed on the title of the window (step 299) and the window is subsequently removed.
Those skilled in the art will recognize that alternate embodiments may be used such as only offering autonomous assistance when directly requested by the user, e.g. by clicking on a graphical icon associated with the intelligent assistance facility.
The system can also use computed measures of expected utility or its approximation for offering autonomous assistance. Furthermore, the system could only offer autonomous assistance when the sum of the individual probabilities of the relevance of each topic in the set of topics that will be offered to provide assistance exceeds some constant value.
In an exemplary embodiment, guesses on help topics are not presented when a user requests assistance, unless the probability computed for the top five topics together do not sum to at least .40. In yet another alternate embodiment, autonomous assistance will not be offered unless the user is pausing, so as to not distract the user while the user is engaged in activity unless it is deemed important to assist the user with that activity while the activity is in progress.
It can also be useful to provide social conventions and metaphors. For example, the results of inference can be used to control the behavioral gestures of a graphical social cartoon character. An example of this kind of interface is displayed in Figure 34. IIJAF inference results about likely problems the user is having or potentially valuable assistance might be alternately be displayed in a "thought cloud"
metaphor over a character's head. Free-text queries can be typed into a graphical text input box that appears to be a means for a character to see or listen to the query.
beech Recognition Enhancement via Integration with IUAF Components Those skilled in the art of speech recognition will appreciate the power that could be gained if the speech language models used to characterize phonemes by probability were dynamically influenced by information computed about the goals, intentions, or needs of computer users. As portrayed in the influence diagram in Figure .
12, user background and goals influence the actions seen in an interface, the words used in free text query, and the speech utterances generated by users in attempting to communicate with a software system or application. Understanding the likelihood of goals and intentions of users in making utterances can enhance our ability to perform automated speech recognition. Such systems will allow us to adjust the probabilities over utterances, and thus, the likelihood of words, and phonemes being uttered by users of computer software.
Speech recognition involves processing the audio signal generated by an utterance into likely phonemes, and the processing of phonemes into likely words.
Probabilistic systems for speech recognition may compute the likelihood of phonemes given the audio signal and the likelihood of alternate words given the signal.
The term speech is used to refer to the evidence provided by the steps of processing an utterance into audio signals or into phonemes.
Bayesian speech understanding systems compute the probability of word or stream of word from an utterance as follows:
p(words~speech) = p(words) * p(speech ~ words) p(speech) By computing the probability over a user's needs or goals given user actions as represented by modeled events (including those based on gaze), we can refine the accuracy of the probability of words given a user speech as follows:
We can update the likelihood that words or word strings will be used in the context of goals by representing the likelihood that words or word strings will be used in the coate~ct of specific goals, and update the base probabilities that words will be used, p(words), contained in a typical language model in real-time. That is, instead of using a static database of p(words), we instead update the probability of words as follows:
p(words) _ ~; p(words ~ user needs * p(user needs This computation takes advantage of a knowledge base of context-specific language models that contain information of the form p(words) _ ~; p(words ( user needs.
At run-time, actions are analyzed by the Event Monitoring System and are passed to the Inference System which provides dynamically the continually recomputed probability distribution over user needs, p(user needs ~ events) as described above. One exemplary application is the reasoning about help topics, and context specific language models contain information of the form p(words ~ relevant help topic).
In a more general statement showing the modification of the above equation for speech recognition, we input dynamic events p(words ~ speech, events) _ [~; p(words ~ user needs;, events) * p(user needs; ~ events), * p(speech ~
words) p(speech) This can be reduced to:
p(words ~ speech, events) = p(words ~ events) * p(speech ~ words) p(speech) Figure 23 displays the key component and flow of analysis employing key components of the intelligent user assistance facility to enhance speech recognition by dynamically changing the likelihoods over words, and thus, phonemes uttered. As displayed in the figure, user actions are tracked by an instrumented program 72, and atomic events are composed into modeled events by the Event Processor 74 and passed to the Inference System 76. The inference system computes a probability distribution of the needs of a user and the weighted list of needs are passed to a Dynamic Language Model Generator 300.
The Dynamic Language Model Generator 300 has a mapping between alternative needs and the likelihood of utterances or words. This component may ;
5 contain information from a Language Model Database 301 that contains language models containing probabilistic information of the form, p(words ~ user need), the probabilities of different words or word strings being uttered by a user given the user needs. These language models can be constructed through training with statistics from a large corpus of strings of words that are recorded in different user-needs contexts. At 10 run-time, the Inference System provides a probability distribution over user needs which is used to control the creation of an updated language model, based on weighting the language models from the different contexts. The updated language model contains information of the form, p(words J events), created by weighting sets of words in terms of the likelihood of alternative needs based on the probabilistic inference 15 results.
Overall, taking advantage of context in listening can enhance the accuracy of speech recognition system 303 by lowering the probability of utterances that are unlikely and raising the probability of utterances that are more likely, per the user modeling.
20 Beyond use with software in receiving assistance, the dynamic speech understanding has application in variety of speech understanding tasks. As one example, the methods have application in a variety of interfaces with command and control, referring to the manipulation of controls and functionality of software. If we model the goals of the user, we can generate grammars that are appropriately weighted 25 to reflect the next steps of the user, and there are less prone to error.
These methods can be used in conjunction with models that generate the probability of the next software command given user goals. In this approach we focus on a particular user need, the desire for action. The user's desire for action may refer to the implementation of one or more software controls by speech, including the access of 30 one or more help topics. We substitute p(desired action J events) for p(user needs J
~6 events) in the above equations, and employ language models that contain information p(words ~ desired action), instead of p(words ~ user needs). Events considered in this model can include context which may include information about the window that is active and the text, buttons, and controls that are displayed to the user while the speech_ utterance is occurring.
For example, dynamic language models for command and control have application in systems that explicitly generate and display probabilistic results as part of their functioning. Consider as an example, a probabilistic reasoning system that employs a Bayesian network or influence diagram for assisting a physician with diagnosis from a set of symptoms. Bayesian reasoning systems in medicine may have sets of windows containing the results of diagnostic inference. A window may list diseases by computed likelihoods, another may list the best next symptoms or tests to evaluate, based on value of information. By building or assessing a model of the probability of the user's actions given the context of displayed information, which may include functions of the computed probabilities and expected utilities assigned to displayed objects, we can compute and use a dynamically re-computed p(words events) in speech recognition.
While the present invention has been illustrated with reference to exemplary embodiments, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the intended scope of the present invention as defined in the appended claims. For example, the general event composition and inference system may also be used in an operating system to provide an intelligent user shell and, more generally, to optimize the functionality of any computer system or software. Because of the variations that can be applied .to the illustrated and described embodiments of the invention, the invention should be defined solely with reference to the appended claims.
Claims (31)
1. An event processor in a computing device having an event source comprising:
a local store for temporarily storing low-level events;
a composer for generating high-level modeled events from at least one or more of the low-level events according to event composition rules defined in an event specification; and a database for storing the high-level modeled events generated.
a local store for temporarily storing low-level events;
a composer for generating high-level modeled events from at least one or more of the low-level events according to event composition rules defined in an event specification; and a database for storing the high-level modeled events generated.
2. The event processor of claim 1, wherein the events stored in the local store are assigned a horizon.
3. The event processor of claim 1, wherein the modeled events generated have an assigned horizon.
4. The event processor of claim 1, wherein the events are user-interface actions.
5. The event processor of claim 1, wherein the source of events is chosen from a group stored on a computer readable medium, the group consisting of a software program, a software library or an instantiated object.
6. The event processor of claim 1, wherein the events are atomic events and the composer generates modeled events from one or more atomic events.
7. The event processor of claim 1 wherein the events are atomic events and the composer generates modeled events from one or more atomic events and from one or more modeled events.
8. The event processor of claim 1 wherein the composer generates modeled events from one or more modeled events.
9. The event processor of claim 1 wherein the local store is a circular queue.
10. The event processor of claim 1 wherein the local store is a database.
11. The event processor of claim 1 wherein the database for storing the modeled events is in system cache.
12. The event processor of claim 1, wherein there is a single record for every modeled event for that database.
13. The event processor of claim 1 wherein the database contains records that include a set of counters of the number of times a modeled event has occurred.
14. The event processor of claim 1 wherein a record of the database contains horizon and probability dynamics information.
15. The event processor of claim 1 wherein a record of the database is marked for persistent storage.
16a The event of processor claim 1, wherein the modeled events generated are logical events.
17. The event processor of claim 1, wherein there is a separate database for each source of events that stores the modeled events generated.
18. The event processor of claim 1, wherein there is a separate database for each source of events that stores the modeled events generated and a general database that stores common system or application events.
19. A computer readable medium employing the event processor of claim 1.
20. A computer readable medium storing an event system specification tool for a computing device comprising:
an event language for specifying construction of high-level modeled events from at least one or more low-level events by statements indicating operations on at least the one or more low-level events; and an event language interpreter for translating the statements of the event language into a target code language.
an event language for specifying construction of high-level modeled events from at least one or more low-level events by statements indicating operations on at least the one or more low-level events; and an event language interpreter for translating the statements of the event language into a target code language.
21. The computer readable medium of claim 20, wherein the event language interpreter is used for translating statements of the event language into modeled database definitions.
22. The computer readable medium of claim 20, wherein the event language includes temporal operators.
23. The computer readable medium of claim 20, wherein the event language includes Boolean operators.
24. The computer readable medium of claim 20, wherein the event language includes set-theoretic operators.
25. The computer readable medium of claim 20, wherein the modeled events are some of the variables of a reasoning model and the event language includes temporal functions that define the persistence of probabilistic relationships between one or more modeled events.
26. The computer readable medium of claim 20, wherein the modeled events are some of the variables of a reasoning model and the event language includes temporal functions that define the persistence of probabilistic relationships between one or more modeled events and one or more other variables in the reasoning model.
27. The computer readable medium of claim 26, wherein the reasoning model is a Bayesian network.
28. The computer readable medium of claim 26, wherein the reasoning model is rule-based.
29. The computer readable medium of claim 22, wherein a temporal operator is chosen from the group consisting of rate, sequence and dwell.
30. A method of providing an event processor in a computing device having an event source, the method comprising the steps of:
temporarily storing low-level events;
generating high-level modeled events from at least one or more of the low-level events according to event composition rules defined in an event specification;
and storing the high-level modeled events generated.
temporarily storing low-level events;
generating high-level modeled events from at least one or more of the low-level events according to event composition rules defined in an event specification;
and storing the high-level modeled events generated.
31. The method as claimed in claim 30, the method being utilised in the system according to any one of the claims 1 to 18.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/684,003 US6021403A (en) | 1996-07-19 | 1996-07-19 | Intelligent user assistance facility |
US08/684,003 | 1996-07-19 |
Publications (2)
Publication Number | Publication Date |
---|---|
CA2210601A1 CA2210601A1 (en) | 1998-01-19 |
CA2210601C true CA2210601C (en) | 2005-01-11 |
Family
ID=24746328
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CA002210601A Expired - Lifetime CA2210601C (en) | 1996-07-19 | 1997-07-16 | Intelligent user assistance facility |
Country Status (7)
Country | Link |
---|---|
US (4) | US6021403A (en) |
EP (2) | EP1742174A2 (en) |
JP (3) | JP4676037B2 (en) |
CN (2) | CN1280715C (en) |
CA (1) | CA2210601C (en) |
DE (1) | DE69736552T2 (en) |
WO (1) | WO1998003907A2 (en) |
Families Citing this family (786)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6418424B1 (en) | 1991-12-23 | 2002-07-09 | Steven M. Hoffberg | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
USRE47908E1 (en) | 1991-12-23 | 2020-03-17 | Blanding Hovenweep, Llc | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US5903454A (en) | 1991-12-23 | 1999-05-11 | Hoffberg; Linda Irene | Human-factored interface corporating adaptive pattern recognition based controller apparatus |
US10361802B1 (en) | 1999-02-01 | 2019-07-23 | Blanding Hovenweep, Llc | Adaptive pattern recognition based control system and method |
USRE48056E1 (en) | 1991-12-23 | 2020-06-16 | Blanding Hovenweep, Llc | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US8352400B2 (en) | 1991-12-23 | 2013-01-08 | Hoffberg Steven M | Adaptive pattern recognition based controller apparatus and method and human-factored interface therefore |
US6850252B1 (en) * | 1999-10-05 | 2005-02-01 | Steven M. Hoffberg | Intelligent electronic appliance system and method |
USRE46310E1 (en) | 1991-12-23 | 2017-02-14 | Blanding Hovenweep, Llc | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US6400996B1 (en) | 1999-02-01 | 2002-06-04 | Steven M. Hoffberg | Adaptive pattern recognition based control system and method |
US5822720A (en) | 1994-02-16 | 1998-10-13 | Sentius Corporation | System amd method for linking streams of multimedia data for reference material for display |
US6434530B1 (en) * | 1996-05-30 | 2002-08-13 | Retail Multimedia Corporation | Interactive shopping system with mobile apparatus |
US20080028292A1 (en) * | 1997-12-22 | 2008-01-31 | Ricoh Company, Ltd. | Techniques to facilitate reading of a document |
US7263659B2 (en) | 1998-09-09 | 2007-08-28 | Ricoh Company, Ltd. | Paper-based interface for multimedia information |
JP4183311B2 (en) | 1997-12-22 | 2008-11-19 | 株式会社リコー | Document annotation method, annotation device, and recording medium |
US7257589B1 (en) * | 1997-12-22 | 2007-08-14 | Ricoh Company, Ltd. | Techniques for targeting information to users |
US7954056B2 (en) | 1997-12-22 | 2011-05-31 | Ricoh Company, Ltd. | Television-based visualization and navigation interface |
US7596755B2 (en) | 1997-12-22 | 2009-09-29 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US7124093B1 (en) | 1997-12-22 | 2006-10-17 | Ricoh Company, Ltd. | Method, system and computer code for content based web advertising |
JP3079079B2 (en) * | 1998-01-07 | 2000-08-21 | インターナショナル・ビジネス・マシーンズ・コーポレ−ション | Screen display device and screen display control method |
GB9800590D0 (en) * | 1998-01-13 | 1998-03-11 | Bae Sema Ltd | Intelligent human computer interface system |
JP3585837B2 (en) * | 1998-02-05 | 2004-11-04 | 富士通株式会社 | Action proposal device |
US6151585A (en) * | 1998-04-24 | 2000-11-21 | Microsoft Corporation | Methods and apparatus for determining or inferring influential rumormongers from resource usage data |
US6199061B1 (en) * | 1998-06-17 | 2001-03-06 | Microsoft Corporation | Method and apparatus for providing dynamic help topic titles to a user |
US6256633B1 (en) * | 1998-06-25 | 2001-07-03 | U.S. Philips Corporation | Context-based and user-profile driven information retrieval |
US6307544B1 (en) * | 1998-07-23 | 2001-10-23 | International Business Machines Corporation | Method and apparatus for delivering a dynamic context sensitive integrated user assistance solution |
US6405188B1 (en) * | 1998-07-31 | 2002-06-11 | Genuity Inc. | Information retrieval system |
US6779060B1 (en) * | 1998-08-05 | 2004-08-17 | British Telecommunications Public Limited Company | Multimodal user interface |
IL125684A0 (en) * | 1998-08-06 | 1999-04-11 | Univ Ben Gurion | Method for computer operation by an intelligent user adaptive interface |
US7272604B1 (en) * | 1999-09-03 | 2007-09-18 | Atle Hedloy | Method, system and computer readable medium for addressing handling from an operating system |
US7496854B2 (en) | 1998-11-10 | 2009-02-24 | Arendi Holding Limited | Method, system and computer readable medium for addressing handling from a computer program |
NO984066L (en) * | 1998-09-03 | 2000-03-06 | Arendi As | Computer function button |
US7266782B2 (en) * | 1998-09-09 | 2007-09-04 | Ricoh Company, Ltd. | Techniques for generating a coversheet for a paper-based interface for multimedia information |
US7263671B2 (en) * | 1998-09-09 | 2007-08-28 | Ricoh Company, Ltd. | Techniques for annotating multimedia information |
US6582475B2 (en) | 1998-09-09 | 2003-06-24 | Ricoh Company Limited | Automatic adaptive document printing help system |
US7215436B2 (en) * | 1998-09-09 | 2007-05-08 | Ricoh Company, Ltd. | Device for generating a multimedia paper document |
US6369811B1 (en) | 1998-09-09 | 2002-04-09 | Ricoh Company Limited | Automatic adaptive document help for paper documents |
JP2000146629A (en) * | 1998-09-23 | 2000-05-26 | Agilent Technol Inc | Help screen information display for measuring apparatus |
EP1044416A1 (en) * | 1998-10-09 | 2000-10-18 | Scansoft, Inc. | Automatic inquiry method and system |
US6446076B1 (en) | 1998-11-12 | 2002-09-03 | Accenture Llp. | Voice interactive web-based agent system responsive to a user location for prioritizing and formatting information |
US6845370B2 (en) | 1998-11-12 | 2005-01-18 | Accenture Llp | Advanced information gathering for targeted activities |
US8121891B2 (en) | 1998-11-12 | 2012-02-21 | Accenture Global Services Gmbh | Personalized product report |
US6320623B1 (en) * | 1998-11-13 | 2001-11-20 | Philips Electronics North America Corporation | Method and device for detecting an event in a program of a video and/ or audio signal and for providing the program to a display upon detection of the event |
WO2000031671A1 (en) * | 1998-11-19 | 2000-06-02 | Accenture Llp | Collection and analysis of user profile information |
CA2350314C (en) * | 1998-11-19 | 2010-01-19 | Andersen Consulting, Llp | A system, method and article of manufacture for effectively interacting with a network user |
US6195651B1 (en) | 1998-11-19 | 2001-02-27 | Andersen Consulting Properties Bv | System, method and article of manufacture for a tuned user application experience |
US20030187925A1 (en) * | 1998-12-08 | 2003-10-02 | Inala Suman Kumar | Software engine for enabling proxy chat-room interaction |
US6236989B1 (en) * | 1998-12-11 | 2001-05-22 | International Business Machines Corporation | Network-based help architecture |
US6747675B1 (en) * | 1998-12-18 | 2004-06-08 | Tangis Corporation | Mediating conflicts in computer user's context data |
US6018730A (en) * | 1998-12-22 | 2000-01-25 | Ac Properties B.V. | System, method and article of manufacture for a simulation engine with a help website and processing engine |
FR2788615B1 (en) * | 1999-01-18 | 2001-02-16 | Thomson Multimedia Sa | APPARATUS COMPRISING A VOICE OR MANUAL USER INTERFACE AND METHOD FOR ASSISTING IN LEARNING VOICE COMMANDS FROM SUCH AN APPARATUS |
US7966078B2 (en) | 1999-02-01 | 2011-06-21 | Steven Hoffberg | Network media appliance system and method |
US6310633B1 (en) | 1999-03-23 | 2001-10-30 | Ricoh Company Limited | Method and system for organizing document information |
US6741995B1 (en) * | 1999-03-23 | 2004-05-25 | Metaedge Corporation | Method for dynamically creating a profile |
US7107253B1 (en) * | 1999-04-05 | 2006-09-12 | American Board Of Family Practice, Inc. | Computer architecture and process of patient generation, evolution and simulation for computer based testing system using bayesian networks as a scripting language |
US6999955B1 (en) * | 1999-04-20 | 2006-02-14 | Microsoft Corporation | Systems and methods for estimating and integrating measures of human cognitive load into the behavior of computational applications and services |
US6581050B1 (en) * | 1999-04-20 | 2003-06-17 | Microsoft Corporation | Learning by observing a user's activity for enhancing the provision of automated services |
US6340977B1 (en) * | 1999-05-07 | 2002-01-22 | Philip Lui | System and method for dynamic assistance in software applications using behavior and host application models |
US6434527B1 (en) * | 1999-05-17 | 2002-08-13 | Microsoft Corporation | Signalling and controlling the status of an automatic speech recognition system for use in handsfree conversational dialogue |
US6671661B1 (en) * | 1999-05-19 | 2003-12-30 | Microsoft Corporation | Bayesian principal component analysis |
US6633852B1 (en) * | 1999-05-21 | 2003-10-14 | Microsoft Corporation | Preference-based catalog browser that utilizes a belief network |
US6606613B1 (en) * | 1999-06-03 | 2003-08-12 | Microsoft Corporation | Methods and apparatus for using task models to help computer users complete tasks |
US6421655B1 (en) * | 1999-06-04 | 2002-07-16 | Microsoft Corporation | Computer-based representations and reasoning methods for engaging users in goal-oriented conversations |
US6415274B1 (en) * | 1999-06-24 | 2002-07-02 | Sandia Corporation | Alpha-beta coordination method for collective search |
US6647534B1 (en) | 1999-06-30 | 2003-11-11 | Ricoh Company Limited | Method and system for organizing document information in a non-directed arrangement of documents |
US7228492B1 (en) * | 1999-07-06 | 2007-06-05 | Ricoh Company, Ltd. | 2D graph displaying document locations of user-specified concept of interest |
AU763131B2 (en) * | 1999-07-09 | 2003-07-17 | Acularo Pty Ltd | Methods of organising information |
US6618716B1 (en) | 1999-07-30 | 2003-09-09 | Microsoft Corporation | Computational architecture for managing the transmittal and rendering of information, alerts, and notifications |
US6622160B1 (en) * | 1999-07-30 | 2003-09-16 | Microsoft Corporation | Methods for routing items for communications based on a measure of criticality |
US6714967B1 (en) * | 1999-07-30 | 2004-03-30 | Microsoft Corporation | Integration of a computer-based message priority system with mobile electronic devices |
US7120865B1 (en) | 1999-07-30 | 2006-10-10 | Microsoft Corporation | Methods for display, notification, and interaction with prioritized messages |
US7194681B1 (en) * | 1999-07-30 | 2007-03-20 | Microsoft Corporation | Method for automatically assigning priorities to documents and messages |
US7007240B1 (en) * | 1999-08-03 | 2006-02-28 | Gateway Inc. | Method and system for displaying non-overlapping program and auxiliary windows |
JP2001056889A (en) * | 1999-08-19 | 2001-02-27 | Konica Corp | Alarm information control method for equipment and equipment having relevant function |
US6499024B1 (en) * | 1999-08-24 | 2002-12-24 | Stream International, Inc. | Method and system for development of a knowledge base system |
US6556960B1 (en) * | 1999-09-01 | 2003-04-29 | Microsoft Corporation | Variational inference engine for probabilistic graphical models |
AU1432601A (en) * | 1999-10-13 | 2001-04-23 | Troba, Inc. | Electronic shopping management: user interface |
US20080221927A1 (en) * | 1999-10-29 | 2008-09-11 | Victor Levy | Web-Enabled, Evidence Based Medical Diagnostic System |
ATE504880T1 (en) * | 1999-11-03 | 2011-04-15 | Sublinks Aps | METHOD, SYSTEM AND COMPUTER READABLE MEDIUM FOR MANAGING CONNECTIONS BETWEEN RESOURCES |
US6526395B1 (en) * | 1999-12-31 | 2003-02-25 | Intel Corporation | Application of personality models and interaction with synthetic characters in a computing system |
US7099855B1 (en) | 2000-01-13 | 2006-08-29 | International Business Machines Corporation | System and method for electronic communication management |
US6513026B1 (en) | 2000-06-17 | 2003-01-28 | Microsoft Corporation | Decision theoretic principles and policies for notification |
US20020087649A1 (en) * | 2000-03-16 | 2002-07-04 | Horvitz Eric J. | Bounded-deferral policies for reducing the disruptiveness of notifications |
US7243130B2 (en) * | 2000-03-16 | 2007-07-10 | Microsoft Corporation | Notification platform architecture |
US6847924B1 (en) * | 2000-06-19 | 2005-01-25 | Ncr Corporation | Method and system for aggregating data distribution models |
US7634528B2 (en) * | 2000-03-16 | 2009-12-15 | Microsoft Corporation | Harnessing information about the timing of a user's client-server interactions to enhance messaging and collaboration services |
US8024415B2 (en) * | 2001-03-16 | 2011-09-20 | Microsoft Corporation | Priorities generation and management |
US7249159B1 (en) | 2000-03-16 | 2007-07-24 | Microsoft Corporation | Notification platform architecture |
US8701027B2 (en) * | 2000-03-16 | 2014-04-15 | Microsoft Corporation | Scope user interface for displaying the priorities and properties of multiple informational items |
US6601012B1 (en) | 2000-03-16 | 2003-07-29 | Microsoft Corporation | Contextual models and methods for inferring attention and location |
US7743340B2 (en) * | 2000-03-16 | 2010-06-22 | Microsoft Corporation | Positioning and rendering notification heralds based on user's focus of attention and activity |
US7444383B2 (en) * | 2000-06-17 | 2008-10-28 | Microsoft Corporation | Bounded-deferral policies for guiding the timing of alerting, interaction and communications using local sensory information |
US7181693B1 (en) * | 2000-03-17 | 2007-02-20 | Gateway Inc. | Affective control of information systems |
WO2001071695A1 (en) * | 2000-03-22 | 2001-09-27 | Virtual Gold, Inc. | Data-driven self-training system and technique |
US6427146B1 (en) * | 2000-03-31 | 2002-07-30 | Wesley W. Chu | Database event detection and notification system using type abstraction hierarchy (TAH) |
US6671802B1 (en) * | 2000-04-13 | 2003-12-30 | Hewlett-Packard Development Company, L.P. | Performance optimization of computer system by dynamically and immediately updating a configuration setting based on detected change in preferred use |
US7730072B2 (en) * | 2000-04-14 | 2010-06-01 | Rightnow Technologies, Inc. | Automated adaptive classification system for knowledge networks |
US6704728B1 (en) | 2000-05-02 | 2004-03-09 | Iphase.Com, Inc. | Accessing information from a collection of data |
US8478732B1 (en) | 2000-05-02 | 2013-07-02 | International Business Machines Corporation | Database aliasing in information access system |
US6938024B1 (en) | 2000-05-04 | 2005-08-30 | Microsoft Corporation | Transmitting information given constrained resources |
US6883019B1 (en) * | 2000-05-08 | 2005-04-19 | Intel Corporation | Providing information to a communications device |
US6957213B1 (en) | 2000-05-17 | 2005-10-18 | Inquira, Inc. | Method of utilizing implicit references to answer a query |
AU2001261631A1 (en) * | 2000-05-17 | 2001-11-26 | Answerfriend.Com | Answering natural language queries |
US7062475B1 (en) * | 2000-05-30 | 2006-06-13 | Alberti Anemometer Llc | Personalized multi-service computer environment |
US6684202B1 (en) * | 2000-05-31 | 2004-01-27 | Lexis Nexis | Computer-based system and method for finding rules of law in text |
US8086672B2 (en) * | 2000-06-17 | 2011-12-27 | Microsoft Corporation | When-free messaging |
US8290768B1 (en) | 2000-06-21 | 2012-10-16 | International Business Machines Corporation | System and method for determining a set of attributes based on content of communications |
US9699129B1 (en) | 2000-06-21 | 2017-07-04 | International Business Machines Corporation | System and method for increasing email productivity |
US7000230B1 (en) * | 2000-06-21 | 2006-02-14 | Microsoft Corporation | Network-based software extensions |
US6948135B1 (en) * | 2000-06-21 | 2005-09-20 | Microsoft Corporation | Method and systems of providing information to computer users |
US7346848B1 (en) * | 2000-06-21 | 2008-03-18 | Microsoft Corporation | Single window navigation methods and systems |
US6883168B1 (en) * | 2000-06-21 | 2005-04-19 | Microsoft Corporation | Methods, systems, architectures and data structures for delivering software via a network |
US7191394B1 (en) * | 2000-06-21 | 2007-03-13 | Microsoft Corporation | Authoring arbitrary XML documents using DHTML and XSLT |
US7155667B1 (en) * | 2000-06-21 | 2006-12-26 | Microsoft Corporation | User interface for integrated spreadsheets and word processing tables |
US6408277B1 (en) | 2000-06-21 | 2002-06-18 | Banter Limited | System and method for automatic task prioritization |
AU6867401A (en) | 2000-06-22 | 2002-01-02 | Microsoft Corp | Distributed computing services platform |
JP2004531780A (en) * | 2000-06-22 | 2004-10-14 | マイクロソフト コーポレーション | Distributed computing service platform |
US7080321B2 (en) * | 2000-06-23 | 2006-07-18 | Aspect Software, Inc. | Dynamic help option for internet customers |
JP2004502234A (en) * | 2000-06-29 | 2004-01-22 | アイナステクノロジー インコーポレイテッド | Modeling Method of Discrete Event System Using Event Flowchart |
DE10032225A1 (en) * | 2000-07-03 | 2002-01-17 | Pixfind Com Gmbh | Search method for digital images in which a computer is used to help speed searching by use of a computer training process and associated decision function that enables the computer to select possible images from a database |
US6788313B1 (en) | 2000-09-28 | 2004-09-07 | International Business Machines Corporation | Method and apparatus for providing on line help for custom application interfaces |
AU7730301A (en) * | 2000-09-29 | 2002-04-11 | International Business Machines Corporation | Improvements relating to contextual help information |
US6993528B1 (en) * | 2000-10-04 | 2006-01-31 | Microsoft Corporation | Methods and systems for allowing third party client applications to influence implementation of high-level document commands |
US9819561B2 (en) | 2000-10-26 | 2017-11-14 | Liveperson, Inc. | System and methods for facilitating object assignments |
US8868448B2 (en) | 2000-10-26 | 2014-10-21 | Liveperson, Inc. | Systems and methods to facilitate selling of products and services |
AU2002234014A1 (en) * | 2000-10-30 | 2002-05-15 | Harvard Business School Publishing | System and method for network-based personalized educational environment |
US7100115B1 (en) * | 2000-10-31 | 2006-08-29 | Sun Microsystems, Inc. | Method and apparatus for providing computer-based help |
US6691127B1 (en) * | 2000-11-09 | 2004-02-10 | General Electric Company | Storage and retrieval of process capability and guidance information, and computer media therefor |
US6941290B2 (en) * | 2000-11-10 | 2005-09-06 | Nec Laboratories America, Inc. | Method for computing all occurrences of a compound event from occurrences of primitive events |
FR2816728B1 (en) * | 2000-11-14 | 2003-09-12 | Evidian | ASSISTANCE TOOL FOR SETTING SOFTWARE |
US6834120B1 (en) * | 2000-11-15 | 2004-12-21 | Sri International | Method and system for estimating the accuracy of inference algorithms using the self-consistency methodology |
US8751957B1 (en) * | 2000-11-22 | 2014-06-10 | Pace Micro Technology Plc | Method and apparatus for obtaining auditory and gestural feedback in a recommendation system |
US7844666B2 (en) * | 2000-12-12 | 2010-11-30 | Microsoft Corporation | Controls and displays for acquiring preferences, inspecting behavior, and guiding the learning and decision policies of an adaptive communications prioritization and routing system |
US7206997B2 (en) * | 2000-12-18 | 2007-04-17 | General Motors Corporation | Functional visualization of spreadsheets |
US6931644B2 (en) | 2000-12-21 | 2005-08-16 | International Business Machines Corporation | Hierarchical connected graph model for implementation of event management design |
JP2002197181A (en) * | 2000-12-27 | 2002-07-12 | Sharp Corp | Service management method and computer-readable information record medium stored with program executing service management method |
US7644057B2 (en) * | 2001-01-03 | 2010-01-05 | International Business Machines Corporation | System and method for electronic communication management |
AUPR242801A0 (en) * | 2001-01-08 | 2001-02-01 | Telstra New Wave Pty Ltd | A contact centre management system |
US7366990B2 (en) * | 2001-01-19 | 2008-04-29 | C-Sam, Inc. | Method and system for managing user activities and information using a customized computer interface |
SE518484C2 (en) * | 2001-02-27 | 2002-10-15 | Peder Holmbom | Apparatus and method for disinfecting water for medical or dental purposes |
US7013297B2 (en) * | 2001-02-27 | 2006-03-14 | Microsoft Corporation | Expert system for generating user interfaces |
US7024662B2 (en) * | 2001-03-14 | 2006-04-04 | Microsoft Corporation | Executing dynamically assigned functions while providing services |
US7302634B2 (en) * | 2001-03-14 | 2007-11-27 | Microsoft Corporation | Schema-based services for identity-based data access |
US7376576B2 (en) * | 2001-03-16 | 2008-05-20 | Portblue Corporation | Decision making and implementation system |
US20020178440A1 (en) * | 2001-03-28 | 2002-11-28 | Philips Electronics North America Corp. | Method and apparatus for automatically selecting an alternate item based on user behavior |
US7065524B1 (en) * | 2001-03-30 | 2006-06-20 | Pharsight Corporation | Identification and correction of confounders in a statistical analysis |
US7478051B2 (en) * | 2001-04-02 | 2009-01-13 | Illah Nourbakhsh | Method and apparatus for long-range planning |
US7155399B2 (en) | 2001-04-03 | 2006-12-26 | Witness Systems, Inc. | System and method for complex schedule generation |
US6952732B2 (en) * | 2001-04-30 | 2005-10-04 | Blue Pumpkin Software, Inc. | Method and apparatus for multi-contact scheduling |
US6959405B2 (en) | 2001-04-18 | 2005-10-25 | Blue Pumpkin Software, Inc. | Method and system for concurrent error identification in resource scheduling |
DE60136052D1 (en) * | 2001-05-04 | 2008-11-20 | Microsoft Corp | Interface control |
EP1412823A4 (en) * | 2001-05-04 | 2007-09-12 | Logic Junction | System and method for logical agent engine |
US7107254B1 (en) | 2001-05-07 | 2006-09-12 | Microsoft Corporation | Probablistic models and methods for combining multiple content classifiers |
EP1256875A1 (en) * | 2001-05-10 | 2002-11-13 | Nokia Corporation | Method and device for context dependent user input prediction |
US6795778B2 (en) * | 2001-05-24 | 2004-09-21 | Lincoln Global, Inc. | System and method for facilitating welding system diagnostics |
US7363348B2 (en) * | 2001-05-29 | 2008-04-22 | International Business Machines Corporation | Method and system in an office application for providing content dependent help information |
US7209912B2 (en) * | 2001-06-06 | 2007-04-24 | Microsoft Corporation | Wizard and help file search and management |
JP2002366166A (en) * | 2001-06-11 | 2002-12-20 | Pioneer Electronic Corp | System and method for providing contents and computer program for the same |
US6785664B2 (en) * | 2001-06-21 | 2004-08-31 | Kevin Wade Jameson | Collection knowledge system |
US6782093B2 (en) | 2001-06-27 | 2004-08-24 | Blue Pumpkin Software, Inc. | Graphical method and system for visualizing performance levels in time-varying environment |
US7689521B2 (en) * | 2001-06-28 | 2010-03-30 | Microsoft Corporation | Continuous time bayesian network models for predicting users' presence, activities, and component usage |
US7233933B2 (en) | 2001-06-28 | 2007-06-19 | Microsoft Corporation | Methods and architecture for cross-device activity monitoring, reasoning, and visualization for providing status and forecasts of a users' presence and availability |
US7089226B1 (en) * | 2001-06-28 | 2006-08-08 | Microsoft Corporation | System, representation, and method providing multilevel information retrieval with clarification dialog |
US7409423B2 (en) * | 2001-06-28 | 2008-08-05 | Horvitz Eric J | Methods for and applications of learning and inferring the periods of time until people are available or unavailable for different forms of communication, collaboration, and information access |
US7493369B2 (en) * | 2001-06-28 | 2009-02-17 | Microsoft Corporation | Composable presence and availability services |
US7409335B1 (en) | 2001-06-29 | 2008-08-05 | Microsoft Corporation | Inferring informational goals and preferred level of detail of answers based on application being employed by the user |
US7519529B1 (en) | 2001-06-29 | 2009-04-14 | Microsoft Corporation | System and methods for inferring informational goals and preferred level of detail of results in response to questions posed to an automated information-retrieval or question-answering service |
DE10135287A1 (en) * | 2001-07-19 | 2003-03-06 | Siemens Ag | Apparatus and method for analyzing the operation of complex systems |
US6694308B2 (en) * | 2001-07-23 | 2004-02-17 | Hewlett-Packard Development Company, L.P. | System and method for user adaptive software interface |
US7526439B2 (en) | 2001-08-06 | 2009-04-28 | Proficient Systems, Incorporated | Systems and methods to facilitate selling of products and services |
US7130861B2 (en) | 2001-08-16 | 2006-10-31 | Sentius International Corporation | Automated creation and delivery of database content |
US20030041159A1 (en) * | 2001-08-17 | 2003-02-27 | David Tinsley | Systems and method for presenting customizable multimedia presentations |
US20030043191A1 (en) * | 2001-08-17 | 2003-03-06 | David Tinsley | Systems and methods for displaying a graphical user interface |
US6744729B2 (en) * | 2001-08-17 | 2004-06-01 | Interactive Sapience Corp. | Intelligent fabric |
US20040205648A1 (en) * | 2001-08-17 | 2004-10-14 | David Tinsley | Systems and methods for authoring content |
WO2003017082A1 (en) * | 2001-08-17 | 2003-02-27 | Interactive Sapience Corp. | System and method for processing media-file in graphical user interface |
US6976067B2 (en) * | 2001-09-06 | 2005-12-13 | International Business Machines Corporation | Method and apparatus for providing entitlement information for interactive support |
US6871322B2 (en) * | 2001-09-06 | 2005-03-22 | International Business Machines Corporation | Method and apparatus for providing user support through an intelligent help agent |
US20030043178A1 (en) * | 2001-09-06 | 2003-03-06 | International Business Machines Corporation | Initiation of interactive support from a computer desktop |
US6973620B2 (en) * | 2001-09-06 | 2005-12-06 | International Business Machines Corporation | Method and apparatus for providing user support based on contextual information |
CA2357160A1 (en) * | 2001-09-10 | 2003-03-10 | Ibm Canada Limited-Ibm Canada Limitee | Wizard user interface providing interim system product generation and reversion during wizard traversal |
KR100438857B1 (en) * | 2001-09-26 | 2004-07-05 | 엘지전자 주식회사 | Multimedia Searching And Browsing System Based On User Profile |
US7149957B2 (en) | 2001-11-19 | 2006-12-12 | Ricoh Company, Ltd. | Techniques for retrieving multimedia information using a paper-based interface |
US7703044B2 (en) * | 2001-11-19 | 2010-04-20 | Ricoh Company, Ltd. | Techniques for generating a static representation for time-based media information |
US8539344B2 (en) | 2001-11-19 | 2013-09-17 | Ricoh Company, Ltd. | Paper-based interface for multimedia information stored by multiple multimedia documents |
US7495795B2 (en) * | 2002-02-21 | 2009-02-24 | Ricoh Company, Ltd. | Interface for printing multimedia information |
US7788080B2 (en) * | 2001-11-19 | 2010-08-31 | Ricoh Company, Ltd. | Paper interface for simulation environments |
US8635531B2 (en) * | 2002-02-21 | 2014-01-21 | Ricoh Company, Ltd. | Techniques for displaying information stored in multiple multimedia documents |
US7747655B2 (en) * | 2001-11-19 | 2010-06-29 | Ricoh Co. Ltd. | Printable representations for time-based media |
US7743347B2 (en) * | 2001-11-19 | 2010-06-22 | Ricoh Company, Ltd. | Paper-based interface for specifying ranges |
US7861169B2 (en) | 2001-11-19 | 2010-12-28 | Ricoh Co. Ltd. | Multimedia print driver dialog interfaces |
TWI233024B (en) * | 2001-11-30 | 2005-05-21 | Penbex Data Systems Incoporati | Data synchronizing system and method |
US20030140063A1 (en) * | 2001-12-17 | 2003-07-24 | Pizzorno Joseph E. | System and method for providing health care advice by diagnosing system function |
US7685278B2 (en) * | 2001-12-18 | 2010-03-23 | Hewlett-Packard Development Company, L.P. | Adapting Bayesian network parameters on-line in a dynamic environment |
US7013275B2 (en) * | 2001-12-28 | 2006-03-14 | Sri International | Method and apparatus for providing a dynamic speech-driven control and remote service access system |
US7167917B2 (en) * | 2002-01-03 | 2007-01-23 | International Business Machines Corporation | Visual tool for developing service components for use in advanced intelligent networks |
US7716170B2 (en) * | 2002-01-08 | 2010-05-11 | Wafik Farag | Holistic dynamic information management platform for end-users to interact with and share all information categories, including data, functions, and results, in collaborative secure venue |
US7200635B2 (en) * | 2002-01-09 | 2007-04-03 | International Business Machines Corporation | Smart messenger |
US7310636B2 (en) * | 2002-01-15 | 2007-12-18 | International Business Machines Corporation | Shortcut enabled, context aware information management |
US7146348B2 (en) * | 2002-01-22 | 2006-12-05 | Honeywell International Inc. | Probabilistic goal recognition system and method incorporating inferred unobserved actions |
US20030171657A1 (en) * | 2002-01-22 | 2003-09-11 | Ralph Leonard | Selection of optimal medication methodology (SOOMM) |
US6990639B2 (en) * | 2002-02-07 | 2006-01-24 | Microsoft Corporation | System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration |
US6996558B2 (en) | 2002-02-26 | 2006-02-07 | International Business Machines Corporation | Application portability and extensibility through database schema and query abstraction |
AU2003225620A1 (en) * | 2002-02-26 | 2003-09-09 | Sap Aktiengesellschaft | Intelligent personal assistants |
US7003730B2 (en) * | 2002-03-08 | 2006-02-21 | International Business Machines Corporation | Graphical user interface to build event-based dynamic searches or queries using event profiles |
US7016849B2 (en) * | 2002-03-25 | 2006-03-21 | Sri International | Method and apparatus for providing speech-driven routing between spoken language applications |
US20040006593A1 (en) * | 2002-06-14 | 2004-01-08 | Vogler Hartmut K. | Multidimensional approach to context-awareness |
US20040002988A1 (en) * | 2002-06-26 | 2004-01-01 | Praveen Seshadri | System and method for modeling subscriptions and subscribers as data |
US7177859B2 (en) * | 2002-06-26 | 2007-02-13 | Microsoft Corporation | Programming model for subscription services |
US7698276B2 (en) * | 2002-06-26 | 2010-04-13 | Microsoft Corporation | Framework for providing a subscription based notification system |
US20040002958A1 (en) | 2002-06-26 | 2004-01-01 | Praveen Seshadri | System and method for providing notification(s) |
US9886309B2 (en) | 2002-06-28 | 2018-02-06 | Microsoft Technology Licensing, Llc | Identity-based distributed computing for device resources |
US7363398B2 (en) * | 2002-08-16 | 2008-04-22 | The Board Of Trustees Of The Leland Stanford Junior University | Intelligent total access system |
US20040036715A1 (en) * | 2002-08-26 | 2004-02-26 | Peter Warren | Multi-level user help |
US20040064725A1 (en) * | 2002-09-18 | 2004-04-01 | Microsoft Corporation | Method and system for detecting a communication problem in a computer network |
EP1408674B1 (en) * | 2002-10-09 | 2005-09-07 | Matsushita Electric Industrial Co., Ltd. | Method and device for anticipating operation |
US7158965B1 (en) * | 2002-11-26 | 2007-01-02 | Microsoft Corporation | Method and apparatus for providing help content corresponding to the occurrence of an event within a computer |
US20040109030A1 (en) * | 2002-12-09 | 2004-06-10 | International Business Machines Corporation | Adaptive timing and adaptive content for graphical user interfaces |
FR2850185A1 (en) * | 2003-01-20 | 2004-07-23 | France Telecom | PERSONALIZED NAVIGATION ASSISTANCE FROCED WITHIN A TELECOMMNICATION NETWORK SUCH AS THE INTERNET NETWORK |
US20040145601A1 (en) * | 2003-01-29 | 2004-07-29 | International Business Machines Corporation | Method and a device for providing additional functionality to a separate application |
US7415672B1 (en) | 2003-03-24 | 2008-08-19 | Microsoft Corporation | System and method for designing electronic forms |
US7370066B1 (en) * | 2003-03-24 | 2008-05-06 | Microsoft Corporation | System and method for offline editing of data files |
US7296017B2 (en) * | 2003-03-28 | 2007-11-13 | Microsoft Corporation | Validation of XML data files |
US7913159B2 (en) | 2003-03-28 | 2011-03-22 | Microsoft Corporation | System and method for real-time validation of structured data files |
US20070050696A1 (en) * | 2003-03-31 | 2007-03-01 | Piersol Kurt W | Physical key for accessing a securely stored digital document |
US7509569B2 (en) | 2003-03-31 | 2009-03-24 | Ricoh Co., Ltd. | Action stickers for nested collections |
US7757162B2 (en) * | 2003-03-31 | 2010-07-13 | Ricoh Co. Ltd. | Document collection manipulation |
US7703002B2 (en) | 2003-03-31 | 2010-04-20 | Ricoh Company, Ltd. | Method and apparatus for composing multimedia documents |
US7552381B2 (en) * | 2003-03-31 | 2009-06-23 | Ricoh Co., Ltd. | Check boxes for identifying and processing stored documents |
US7536638B2 (en) * | 2003-03-31 | 2009-05-19 | Ricoh Co., Ltd. | Action stickers for identifying and processing stored documents |
US7739583B2 (en) * | 2003-03-31 | 2010-06-15 | Ricoh Company, Ltd. | Multimedia document sharing method and apparatus |
US7457879B2 (en) | 2003-04-01 | 2008-11-25 | Microsoft Corporation | Notification platform architecture |
US7389230B1 (en) | 2003-04-22 | 2008-06-17 | International Business Machines Corporation | System and method for classification of voice signals |
US7319993B2 (en) * | 2003-04-25 | 2008-01-15 | Intel Corporation | Automatically generating a path to an item depending on its access history |
US20040243531A1 (en) * | 2003-04-28 | 2004-12-02 | Dean Michael Anthony | Methods and systems for representing, using and displaying time-varying information on the Semantic Web |
US20050187913A1 (en) | 2003-05-06 | 2005-08-25 | Yoram Nelken | Web-based customer service interface |
US8495002B2 (en) * | 2003-05-06 | 2013-07-23 | International Business Machines Corporation | Software tool for training and testing a knowledge base |
US7185231B2 (en) * | 2003-05-14 | 2007-02-27 | Microsoft Corporation | Methods and systems for collecting, analyzing, and reporting software reliability and availability |
US7197447B2 (en) * | 2003-05-14 | 2007-03-27 | Microsoft Corporation | Methods and systems for analyzing software reliability and availability |
US7739661B2 (en) * | 2003-05-14 | 2010-06-15 | Microsoft Corporation | Methods and systems for planning and tracking software reliability and availability |
JP4027269B2 (en) * | 2003-06-02 | 2007-12-26 | キヤノン株式会社 | Information processing method and apparatus |
US20040268267A1 (en) * | 2003-06-25 | 2004-12-30 | Xerox Corporation. | Methods and apparatus for prompted activation of an inactive control element in graphical user interface, and context-based help therefor |
US7882434B2 (en) * | 2003-06-27 | 2011-02-01 | Benjamin Slotznick | User prompting when potentially mistaken actions occur during user interaction with content on a display screen |
US8739146B2 (en) * | 2003-07-30 | 2014-05-27 | International Business Machines Corporation | Systems and methods for generating and distributing executable procedures for technical desk-side support |
US7406660B1 (en) | 2003-08-01 | 2008-07-29 | Microsoft Corporation | Mapping between structured data and a visual surface |
US7334187B1 (en) * | 2003-08-06 | 2008-02-19 | Microsoft Corporation | Electronic form aggregation |
US7275159B2 (en) * | 2003-08-11 | 2007-09-25 | Ricoh Company, Ltd. | Multimedia output device having embedded encryption functionality |
US20050081153A1 (en) * | 2003-08-12 | 2005-04-14 | Gbs Global Business Software And Services Limited | Method for providing process-dependent data |
US20050054381A1 (en) * | 2003-09-05 | 2005-03-10 | Samsung Electronics Co., Ltd. | Proactive user interface |
EP1671268A4 (en) * | 2003-09-19 | 2006-12-06 | Lattix Inc | Apparatus and method for managing design of a software system using dependency structure |
US8335755B1 (en) * | 2003-09-23 | 2012-12-18 | Evolving Logic | Derived contexts for computational experiments |
US20050071479A1 (en) * | 2003-09-30 | 2005-03-31 | Dimitris Achlioptas | Smart button |
US7383170B2 (en) * | 2003-10-10 | 2008-06-03 | At&T Knowledge Ventures, L.P. | System and method for analyzing automatic speech recognition performance data |
EP1524589A1 (en) * | 2003-10-17 | 2005-04-20 | Alcatel | A method for providing a functionality set to a user, a device, a computer software product, a user interface network service, and a user interface therefor |
US7669177B2 (en) | 2003-10-24 | 2010-02-23 | Microsoft Corporation | System and method for preference application installation and execution |
US7620667B2 (en) * | 2003-11-17 | 2009-11-17 | Microsoft Corporation | Transfer of user profiles using portable storage devices |
DE10356399B4 (en) * | 2003-12-03 | 2006-06-14 | Ontoprise Gmbh | Data processing system |
US7620888B2 (en) * | 2003-12-04 | 2009-11-17 | Microsoft Corporation | Quality enhancement systems and methods for technical documentation |
US7559057B2 (en) * | 2003-12-05 | 2009-07-07 | International Business Machines Corporation | Alignment and generalization of demonstrated procedure traces |
US7900133B2 (en) | 2003-12-09 | 2011-03-01 | International Business Machines Corporation | Annotation structure type determination |
US7865829B1 (en) | 2003-12-31 | 2011-01-04 | Intuit Inc. | Providing software application help based on heuristics |
US7707039B2 (en) | 2004-02-15 | 2010-04-27 | Exbiblio B.V. | Automatic modification of web pages |
US8442331B2 (en) * | 2004-02-15 | 2013-05-14 | Google Inc. | Capturing text from rendered documents using supplemental information |
US8819072B1 (en) | 2004-02-02 | 2014-08-26 | Microsoft Corporation | Promoting data from structured data files |
US20060041605A1 (en) * | 2004-04-01 | 2006-02-23 | King Martin T | Determining actions involving captured information and electronic content associated with rendered documents |
US10635723B2 (en) | 2004-02-15 | 2020-04-28 | Google Llc | Search engines and systems with handheld document data capture devices |
US20060122983A1 (en) * | 2004-12-03 | 2006-06-08 | King Martin T | Locating electronic instances of documents based on rendered instances, document fragment digest generation, and digest based document fragment determination |
US7812860B2 (en) * | 2004-04-01 | 2010-10-12 | Exbiblio B.V. | Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device |
US20060041484A1 (en) * | 2004-04-01 | 2006-02-23 | King Martin T | Methods and systems for initiating application processes by data capture from rendered documents |
US20060053097A1 (en) * | 2004-04-01 | 2006-03-09 | King Martin T | Searching and accessing documents on private networks for use with captures from rendered documents |
US7277884B2 (en) * | 2004-02-17 | 2007-10-02 | Microsoft Corporation | Method and system for generating help files based on user queries |
US7430711B2 (en) * | 2004-02-17 | 2008-09-30 | Microsoft Corporation | Systems and methods for editing XML documents |
US20080096174A1 (en) * | 2004-03-01 | 2008-04-24 | Koninklijke Philips Electronics, N.V. | Tutorial generation unit, multimedia management system, portable apparatus, method of explanation of multimedia management behavior, computer program product |
US7802265B2 (en) * | 2004-03-15 | 2010-09-21 | Imi Innovations, Inc. | Computer interface system using multiple independent graphical data input devices |
US7983835B2 (en) | 2004-11-03 | 2011-07-19 | Lagassey Paul J | Modular intelligent transportation system |
US7836461B2 (en) * | 2004-03-15 | 2010-11-16 | Imi Innovations, Inc. | Computer interface system using multiple independent hardware and virtual human-computer input devices and related enabling subroutines |
US7779355B1 (en) | 2004-03-30 | 2010-08-17 | Ricoh Company, Ltd. | Techniques for using paper documents as media templates |
US9143638B2 (en) | 2004-04-01 | 2015-09-22 | Google Inc. | Data capture from rendered documents using handheld device |
US8146156B2 (en) | 2004-04-01 | 2012-03-27 | Google Inc. | Archive of text captures from rendered documents |
US8081849B2 (en) * | 2004-12-03 | 2011-12-20 | Google Inc. | Portable scanning and memory device |
US9116890B2 (en) | 2004-04-01 | 2015-08-25 | Google Inc. | Triggering actions in response to optically or acoustically capturing keywords from a rendered document |
US9008447B2 (en) * | 2004-04-01 | 2015-04-14 | Google Inc. | Method and system for character recognition |
US20070300142A1 (en) * | 2005-04-01 | 2007-12-27 | King Martin T | Contextual dynamic advertising based upon captured rendered text |
WO2008028674A2 (en) | 2006-09-08 | 2008-03-13 | Exbiblio B.V. | Optical scanners, such as hand-held optical scanners |
US7894670B2 (en) | 2004-04-01 | 2011-02-22 | Exbiblio B.V. | Triggering actions in response to optically or acoustically capturing keywords from a rendered document |
US20080313172A1 (en) * | 2004-12-03 | 2008-12-18 | King Martin T | Determining actions involving captured information and electronic content associated with rendered documents |
US7990556B2 (en) | 2004-12-03 | 2011-08-02 | Google Inc. | Association of a portable scanner with input/output and storage devices |
US20060098900A1 (en) * | 2004-09-27 | 2006-05-11 | King Martin T | Secure data gathering from rendered documents |
US20060081714A1 (en) * | 2004-08-23 | 2006-04-20 | King Martin T | Portable scanning device |
US20050246434A1 (en) * | 2004-04-05 | 2005-11-03 | International Business Machines Corporation | Services for capturing and modeling computer usage |
US8082264B2 (en) | 2004-04-07 | 2011-12-20 | Inquira, Inc. | Automated scheme for identifying user intent in real-time |
US8612208B2 (en) | 2004-04-07 | 2013-12-17 | Oracle Otc Subsidiary Llc | Ontology for use with a system, method, and computer readable medium for retrieving information and response to a query |
US7890744B2 (en) * | 2004-04-07 | 2011-02-15 | Microsoft Corporation | Activating content based on state |
US7747601B2 (en) * | 2006-08-14 | 2010-06-29 | Inquira, Inc. | Method and apparatus for identifying and classifying query intent |
US7822992B2 (en) * | 2004-04-07 | 2010-10-26 | Microsoft Corporation | In-place content substitution via code-invoking link |
US8713418B2 (en) * | 2004-04-12 | 2014-04-29 | Google Inc. | Adding value to a rendered document |
KR100584380B1 (en) * | 2004-04-13 | 2006-05-26 | 삼성전자주식회사 | System and method for implementing growing agent of mobile station based on network |
US8489624B2 (en) | 2004-05-17 | 2013-07-16 | Google, Inc. | Processing techniques for text capture from a rendered document |
US8874504B2 (en) * | 2004-12-03 | 2014-10-28 | Google Inc. | Processing techniques for visual capture data from a rendered document |
US8620083B2 (en) | 2004-12-03 | 2013-12-31 | Google Inc. | Method and system for character recognition |
US20050246241A1 (en) * | 2004-04-30 | 2005-11-03 | Rightnow Technologies, Inc. | Method and system for monitoring successful use of application software |
US20110099053A1 (en) * | 2004-04-30 | 2011-04-28 | Irizarry Jr Robert T | Method and system for monitoring successful use of application software |
WO2005111840A2 (en) * | 2004-05-11 | 2005-11-24 | Gerjets Sven W | Computerized comprehensive health assessment and physician directed systems |
US20060107219A1 (en) * | 2004-05-26 | 2006-05-18 | Motorola, Inc. | Method to enhance user interface and target applications based on context awareness |
US20060031465A1 (en) * | 2004-05-26 | 2006-02-09 | Motorola, Inc. | Method and system of arranging configurable options in a user interface |
US20050266866A1 (en) * | 2004-05-26 | 2005-12-01 | Motorola, Inc. | Feature finding assistant on a user interface |
US7774620B1 (en) | 2004-05-27 | 2010-08-10 | Microsoft Corporation | Executing applications at appropriate trust levels |
US20050278340A1 (en) * | 2004-05-28 | 2005-12-15 | Rehberg Charles P | Problem solving assistant |
US7440933B2 (en) * | 2004-06-18 | 2008-10-21 | International Business Machines Corporation | Method for facilitating problem resolution |
US7350190B2 (en) * | 2004-06-29 | 2008-03-25 | International Business Machines Corporation | Computer implemented modeling and analysis of an application user interface |
US20060010381A1 (en) * | 2004-07-07 | 2006-01-12 | International Business Machines Corporation | Method for visually indicating the quality of on-screen help messages |
US8346620B2 (en) | 2004-07-19 | 2013-01-01 | Google Inc. | Automatic modification of web pages |
US8341172B2 (en) * | 2004-07-22 | 2012-12-25 | International Business Machines Corporation | Method and system for providing aggregate data access |
US20060026531A1 (en) * | 2004-07-29 | 2006-02-02 | Sony Coporation | State-based computer help utility |
GB2416877A (en) * | 2004-08-03 | 2006-02-08 | Tis Software Ltd | Context sensitive information provision |
US20060031780A1 (en) * | 2004-08-05 | 2006-02-09 | Schlotzhauer Ed O | Dynamically configurable, task oriented communication between user and application |
US20060053100A1 (en) * | 2004-09-03 | 2006-03-09 | Ich Servers, S.L. | Rapid search system |
US7730010B2 (en) * | 2004-09-20 | 2010-06-01 | Microsoft Corporation | Method, system, and apparatus for maintaining user privacy in a knowledge interchange system |
US7593924B2 (en) * | 2004-09-20 | 2009-09-22 | Microsoft Corporation | Method, system, and apparatus for receiving and responding to knowledge interchange queries |
US7707167B2 (en) * | 2004-09-20 | 2010-04-27 | Microsoft Corporation | Method, system, and apparatus for creating a knowledge interchange profile |
US7950044B2 (en) * | 2004-09-28 | 2011-05-24 | Rockwell Automation Technologies, Inc. | Centrally managed proxy-based security for legacy automation systems |
US7692636B2 (en) * | 2004-09-30 | 2010-04-06 | Microsoft Corporation | Systems and methods for handwriting to a screen |
US10445799B2 (en) | 2004-09-30 | 2019-10-15 | Uber Technologies, Inc. | Supply-chain side assistance |
US9307577B2 (en) | 2005-01-21 | 2016-04-05 | The Invention Science Fund I, Llc | User assistance |
US10514816B2 (en) | 2004-12-01 | 2019-12-24 | Uber Technologies, Inc. | Enhanced user assistance |
US9747579B2 (en) * | 2004-09-30 | 2017-08-29 | The Invention Science Fund I, Llc | Enhanced user assistance |
US10687166B2 (en) | 2004-09-30 | 2020-06-16 | Uber Technologies, Inc. | Obtaining user assistance |
US20060074933A1 (en) * | 2004-09-30 | 2006-04-06 | Microsoft Corporation | Workflow interaction |
US7275017B2 (en) * | 2004-10-13 | 2007-09-25 | Cisco Technology, Inc. | Method and apparatus for generating diagnoses of network problems |
US8487879B2 (en) * | 2004-10-29 | 2013-07-16 | Microsoft Corporation | Systems and methods for interacting with a computer through handwriting to a screen |
US20060106854A1 (en) * | 2004-11-01 | 2006-05-18 | Haigh Karen Z | System and method for automatically including supplemental information in reminder messages |
US8805894B2 (en) * | 2004-11-05 | 2014-08-12 | Michael Valdiserri | Interactive 3-dimensional object-oriented database information storage/retrieval system |
US7584417B2 (en) * | 2004-11-15 | 2009-09-01 | Microsoft Corporation | Role-dependent action for an electronic form |
US20060107224A1 (en) * | 2004-11-15 | 2006-05-18 | Microsoft Corporation | Building a dynamic action for an electronic form |
US7712022B2 (en) * | 2004-11-15 | 2010-05-04 | Microsoft Corporation | Mutually exclusive options in electronic forms |
US8942985B2 (en) | 2004-11-16 | 2015-01-27 | Microsoft Corporation | Centralized method and system for clarifying voice commands |
US7721190B2 (en) * | 2004-11-16 | 2010-05-18 | Microsoft Corporation | Methods and systems for server side form processing |
US8176422B2 (en) * | 2004-11-19 | 2012-05-08 | International Business Machines Corporation | Method for aligning demonstrated user actions with existing documentation |
US7480648B2 (en) * | 2004-12-06 | 2009-01-20 | International Business Machines Corporation | Research rapidity and efficiency improvement by analysis of research artifact similarity |
US20060150077A1 (en) * | 2004-12-30 | 2006-07-06 | Microsoft Corporation | User interface method |
US20060155686A1 (en) * | 2005-01-13 | 2006-07-13 | Peterson John W | Facilitating direct access to live controls for features of a system or application via a keyword search |
US7937651B2 (en) * | 2005-01-14 | 2011-05-03 | Microsoft Corporation | Structural editing operations for network forms |
US7624097B2 (en) * | 2005-01-14 | 2009-11-24 | International Business Machines Corporation | Abstract records |
US8122012B2 (en) * | 2005-01-14 | 2012-02-21 | International Business Machines Corporation | Abstract record timeline rendering/display |
US7272587B1 (en) * | 2005-01-28 | 2007-09-18 | Hrl Laboratories, Llc | Generation of decision trees by means of a probabilistic model |
US9124729B2 (en) * | 2005-01-31 | 2015-09-01 | The Invention Science Fund I, Llc | Shared image device synchronization or designation |
US20060170956A1 (en) * | 2005-01-31 | 2006-08-03 | Jung Edward K | Shared image devices |
US9910341B2 (en) | 2005-01-31 | 2018-03-06 | The Invention Science Fund I, Llc | Shared image device designation |
US20060187230A1 (en) * | 2005-01-31 | 2006-08-24 | Searete Llc | Peripheral shared image device sharing |
US9325781B2 (en) | 2005-01-31 | 2016-04-26 | Invention Science Fund I, Llc | Audio sharing |
US20060172267A1 (en) * | 2005-01-31 | 2006-08-03 | Microsoft Corporation | Input device training and automatic assignment |
US9082456B2 (en) | 2005-01-31 | 2015-07-14 | The Invention Science Fund I Llc | Shared image device designation |
US9489717B2 (en) * | 2005-01-31 | 2016-11-08 | Invention Science Fund I, Llc | Shared image device |
US7725834B2 (en) | 2005-03-04 | 2010-05-25 | Microsoft Corporation | Designer-created aspect for an electronic form template |
US20060206444A1 (en) * | 2005-03-11 | 2006-09-14 | International Business Machines Corporation | Services using beliefs and rules for improved flexibility |
US7673228B2 (en) * | 2005-03-30 | 2010-03-02 | Microsoft Corporation | Data-driven actions for network forms |
US8010515B2 (en) * | 2005-04-15 | 2011-08-30 | Microsoft Corporation | Query to an electronic form |
US7703037B2 (en) * | 2005-04-20 | 2010-04-20 | Microsoft Corporation | Searchable task-based interface to control panel functionality |
US20070222865A1 (en) * | 2006-03-15 | 2007-09-27 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Enhanced video/still image correlation |
US20090144391A1 (en) * | 2007-11-30 | 2009-06-04 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Audio sharing |
US9191611B2 (en) * | 2005-06-02 | 2015-11-17 | Invention Science Fund I, Llc | Conditional alteration of a saved image |
US8964054B2 (en) * | 2006-08-18 | 2015-02-24 | The Invention Science Fund I, Llc | Capturing selected image objects |
US10003762B2 (en) | 2005-04-26 | 2018-06-19 | Invention Science Fund I, Llc | Shared image devices |
US9942511B2 (en) | 2005-10-31 | 2018-04-10 | Invention Science Fund I, Llc | Preservation/degradation of video/audio aspects of a data stream |
US20070109411A1 (en) * | 2005-06-02 | 2007-05-17 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Composite image selectivity |
US9967424B2 (en) * | 2005-06-02 | 2018-05-08 | Invention Science Fund I, Llc | Data storage usage protocol |
US9093121B2 (en) | 2006-02-28 | 2015-07-28 | The Invention Science Fund I, Llc | Data management of an audio data stream |
US9621749B2 (en) * | 2005-06-02 | 2017-04-11 | Invention Science Fund I, Llc | Capturing selected image objects |
US9451200B2 (en) * | 2005-06-02 | 2016-09-20 | Invention Science Fund I, Llc | Storage access technique for captured data |
US9167195B2 (en) * | 2005-10-31 | 2015-10-20 | Invention Science Fund I, Llc | Preservation/degradation of video/audio aspects of a data stream |
US9819490B2 (en) | 2005-05-04 | 2017-11-14 | Invention Science Fund I, Llc | Regional proximity for shared image device(s) |
US9001215B2 (en) * | 2005-06-02 | 2015-04-07 | The Invention Science Fund I, Llc | Estimating shared image device operational capabilities or resources |
US20070139529A1 (en) * | 2005-06-02 | 2007-06-21 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Dual mode image capture technique |
US9076208B2 (en) * | 2006-02-28 | 2015-07-07 | The Invention Science Fund I, Llc | Imagery processing |
US20070008326A1 (en) * | 2005-06-02 | 2007-01-11 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Dual mode image capture technique |
US20070098348A1 (en) * | 2005-10-31 | 2007-05-03 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Degradation/preservation management of captured data |
US8280719B2 (en) * | 2005-05-05 | 2012-10-02 | Ramp, Inc. | Methods and systems relating to information extraction |
US20060279531A1 (en) * | 2005-05-25 | 2006-12-14 | Jung Edward K | Physical interaction-responsive user interface |
US20070011665A1 (en) * | 2005-06-21 | 2007-01-11 | Microsoft Corporation | Content syndication platform |
JP4774825B2 (en) * | 2005-06-22 | 2011-09-14 | ソニー株式会社 | Performance evaluation apparatus and method |
US8200975B2 (en) * | 2005-06-29 | 2012-06-12 | Microsoft Corporation | Digital signatures for network forms |
US7428521B2 (en) * | 2005-06-29 | 2008-09-23 | Microsoft Corporation | Precomputation of context-sensitive policies for automated inquiry and action under uncertainty |
US7484179B2 (en) * | 2005-06-30 | 2009-01-27 | Microsoft Corporation | Integrated work management and tracking |
WO2007008556A2 (en) * | 2005-07-07 | 2007-01-18 | Sermo, Inc. | Method and apparatus for conducting an information brokering service |
US9183752B2 (en) * | 2005-07-14 | 2015-11-10 | Red Hat, Inc. | Tutorial generator with automatic capture of screenshots |
JP2007025497A (en) * | 2005-07-20 | 2007-02-01 | Fujitsu Ltd | Educational assistance program and educational assistant device |
FR2888968B1 (en) * | 2005-07-25 | 2008-05-09 | Airbus Sas | METHOD OF PROCESSING DATA FOR DETERMINING VISUAL PATTERNS IN A VISUAL SCENE |
US7601904B2 (en) * | 2005-08-03 | 2009-10-13 | Richard Dreyfuss | Interactive tool and appertaining method for creating a graphical music display |
JP2007048035A (en) * | 2005-08-10 | 2007-02-22 | Konica Minolta Business Technologies Inc | Controller, program and operation interface |
US7613996B2 (en) * | 2005-08-15 | 2009-11-03 | Microsoft Corporation | Enabling selection of an inferred schema part |
US20070043717A1 (en) * | 2005-08-18 | 2007-02-22 | Arellanes Paul T | Relevancy association architecture |
US8738732B2 (en) | 2005-09-14 | 2014-05-27 | Liveperson, Inc. | System and method for performing follow up based on user interactions |
US20070061706A1 (en) * | 2005-09-14 | 2007-03-15 | Microsoft Corporation | Mapping property hierarchies to schemas |
US9432468B2 (en) | 2005-09-14 | 2016-08-30 | Liveperson, Inc. | System and method for design and dynamic generation of a web page |
US20070061467A1 (en) * | 2005-09-15 | 2007-03-15 | Microsoft Corporation | Sessions and session states |
US9047269B2 (en) * | 2005-10-28 | 2015-06-02 | Openconnect Systems Incorporated | Modeling interactions with a computer system |
US20070120980A1 (en) | 2005-10-31 | 2007-05-31 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Preservation/degradation of video/audio aspects of a data stream |
US20070101287A1 (en) * | 2005-11-03 | 2007-05-03 | International Business Machines Corporation | Pop-up windows in a computer system |
US8805675B2 (en) * | 2005-11-07 | 2014-08-12 | Sap Ag | Representing a computer system state to a user |
US7840451B2 (en) * | 2005-11-07 | 2010-11-23 | Sap Ag | Identifying the most relevant computer system state information |
US7979295B2 (en) * | 2005-12-02 | 2011-07-12 | Sap Ag | Supporting user interaction with a computer system |
US8001459B2 (en) * | 2005-12-05 | 2011-08-16 | Microsoft Corporation | Enabling electronic documents for limited-capability computing devices |
US7676489B2 (en) * | 2005-12-06 | 2010-03-09 | Sap Ag | Providing natural-language interface to repository |
US20070136222A1 (en) * | 2005-12-09 | 2007-06-14 | Microsoft Corporation | Question and answer architecture for reasoning and clarifying intentions, goals, and needs from contextual clues and content |
US20070143275A1 (en) * | 2005-12-21 | 2007-06-21 | International Business Machines Corporation | Work-context search strings for work-embedded e-learning |
US20070157093A1 (en) * | 2005-12-30 | 2007-07-05 | Patrick Karcher | Systems and methods for adaptive help mechanisms for a user |
US7917904B2 (en) | 2006-01-06 | 2011-03-29 | Microsoft Corporation | Automated analysis tasks of complex computer system |
US7904887B2 (en) * | 2006-02-16 | 2011-03-08 | International Business Machines Corporation | Learning and cache management in software defined contexts |
US7599861B2 (en) | 2006-03-02 | 2009-10-06 | Convergys Customer Management Group, Inc. | System and method for closed loop decisionmaking in an automated care system |
US7925975B2 (en) * | 2006-03-10 | 2011-04-12 | Microsoft Corporation | Searching for commands to execute in applications |
JP4783181B2 (en) * | 2006-03-13 | 2011-09-28 | 株式会社東芝 | Behavior prediction device |
US7752545B2 (en) * | 2006-03-15 | 2010-07-06 | Microsoft Corporation | Generator system |
US7752152B2 (en) | 2006-03-17 | 2010-07-06 | Microsoft Corporation | Using predictive user models for language modeling on a personal device with user behavior models based on statistical modeling |
US8032375B2 (en) | 2006-03-17 | 2011-10-04 | Microsoft Corporation | Using generic predictive models for slot values in language modeling |
WO2007109444A2 (en) * | 2006-03-17 | 2007-09-27 | Schmitt William C | Common format learning device |
US8358976B2 (en) | 2006-03-24 | 2013-01-22 | The Invention Science Fund I, Llc | Wireless device with an aggregate user interface for controlling other devices |
US8095490B2 (en) * | 2006-03-27 | 2012-01-10 | Panasonic Corporation | User support device, method, and program |
US8244545B2 (en) * | 2006-03-30 | 2012-08-14 | Microsoft Corporation | Dialog repair based on discrepancies between user model predictions and speech recognition results |
US7921099B2 (en) | 2006-05-10 | 2011-04-05 | Inquira, Inc. | Guided navigation system |
US9507778B2 (en) | 2006-05-19 | 2016-11-29 | Yahoo! Inc. | Summarization of media object collections |
US8379830B1 (en) | 2006-05-22 | 2013-02-19 | Convergys Customer Management Delaware Llc | System and method for automated customer service with contingent live interaction |
US7809663B1 (en) | 2006-05-22 | 2010-10-05 | Convergys Cmg Utah, Inc. | System and method for supporting the utilization of machine language |
US8010094B2 (en) * | 2006-06-06 | 2011-08-30 | Turner Broadcasting System, Inc. | Mobile device with themed multimedia effects |
US7730068B2 (en) * | 2006-06-13 | 2010-06-01 | Microsoft Corporation | Extensible data collectors |
US7970746B2 (en) * | 2006-06-13 | 2011-06-28 | Microsoft Corporation | Declarative management framework |
WO2007145680A1 (en) * | 2006-06-13 | 2007-12-21 | Microsoft Corporation | Declarative management framework |
US20080155409A1 (en) * | 2006-06-19 | 2008-06-26 | Andy Santana | Internet search engine |
CN101473295B (en) * | 2006-06-21 | 2011-11-16 | 松下电器产业株式会社 | Device for estimating user operation intention and electronic device using the same |
US8364514B2 (en) * | 2006-06-27 | 2013-01-29 | Microsoft Corporation | Monitoring group activities |
US7970637B2 (en) | 2006-06-27 | 2011-06-28 | Microsoft Corporation | Activity-centric granular application functionality |
US20070300225A1 (en) * | 2006-06-27 | 2007-12-27 | Microsoft Coporation | Providing user information to introspection |
US7548895B2 (en) * | 2006-06-30 | 2009-06-16 | Microsoft Corporation | Communication-prompted user assistance |
US8781813B2 (en) * | 2006-08-14 | 2014-07-15 | Oracle Otc Subsidiary Llc | Intent management tool for identifying concepts associated with a plurality of users' queries |
US9202184B2 (en) | 2006-09-07 | 2015-12-01 | International Business Machines Corporation | Optimizing the selection, verification, and deployment of expert resources in a time of chaos |
US7630948B2 (en) * | 2006-09-07 | 2009-12-08 | International Business Machines Corporation | System and method for managing a chaotic event |
US7653609B2 (en) * | 2006-09-07 | 2010-01-26 | International Business Machines Corporation | System and method for managing a chaotic event by optimizing decision subdivisions subject to multidimensional constraints |
US7647288B2 (en) * | 2006-09-07 | 2010-01-12 | International Business Machines Corporation | System and method for optimally customizable and adaptive personalized information display for information associated with managing a chaotic event |
US7698246B2 (en) * | 2006-09-07 | 2010-04-13 | International Business Machines Corporation | System and method for optimal and adaptive process unification of decision support functions associated with managing a chaotic event |
US7647286B2 (en) * | 2006-09-07 | 2010-01-12 | International Business Machines Corporation | System and method for managing a chaotic event by providing optimal and adaptive sequencing of decision sets with supporting data |
US7584160B2 (en) * | 2006-10-27 | 2009-09-01 | International Business Machines Corporation | System and method for optimizing project subdivision using data and requirements focuses subject to multidimensional constraints |
US9318108B2 (en) * | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
KR101443404B1 (en) * | 2006-09-15 | 2014-10-02 | 구글 인코포레이티드 | Capture and display of annotations in paper and electronic documents |
US7752154B2 (en) * | 2007-02-26 | 2010-07-06 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for analysis of criminal and security information |
US20090287503A1 (en) * | 2008-05-16 | 2009-11-19 | International Business Machines Corporation | Analysis of individual and group healthcare data in order to provide real time healthcare recommendations |
US7809660B2 (en) * | 2006-10-03 | 2010-10-05 | International Business Machines Corporation | System and method to optimize control cohorts using clustering algorithms |
US8055603B2 (en) * | 2006-10-03 | 2011-11-08 | International Business Machines Corporation | Automatic generation of new rules for processing synthetic events using computer-based learning processes |
US8145582B2 (en) | 2006-10-03 | 2012-03-27 | International Business Machines Corporation | Synthetic events for real time patient analysis |
US8589190B1 (en) | 2006-10-06 | 2013-11-19 | Liberty Mutual Insurance Company | System and method for underwriting a prepackaged business owners insurance policy |
US20080102428A1 (en) * | 2006-10-25 | 2008-05-01 | Monitor Company Group Limited Partnership | System and Method for Providing a User-Centric Interactive Learning Environment |
US20080104101A1 (en) * | 2006-10-27 | 2008-05-01 | Kirshenbaum Evan R | Producing a feature in response to a received expression |
US8594702B2 (en) * | 2006-11-06 | 2013-11-26 | Yahoo! Inc. | Context server for associating information based on context |
US8032834B2 (en) * | 2006-11-07 | 2011-10-04 | International Business Machines Corporation | Context-based user assistance |
US7720868B2 (en) * | 2006-11-13 | 2010-05-18 | Microsoft Corporation | Providing assistance with the creation of an XPath expression |
US8640034B2 (en) * | 2006-11-16 | 2014-01-28 | International Business Machines Corporation | Remote GUI control by replication of local interactions |
US8595636B2 (en) * | 2006-11-16 | 2013-11-26 | International Business Machines Corporation | Method and system for mapping GUI widgets |
US9110903B2 (en) * | 2006-11-22 | 2015-08-18 | Yahoo! Inc. | Method, system and apparatus for using user profile electronic device data in media delivery |
US8402356B2 (en) * | 2006-11-22 | 2013-03-19 | Yahoo! Inc. | Methods, systems and apparatus for delivery of media |
US8095476B2 (en) | 2006-11-27 | 2012-01-10 | Inquira, Inc. | Automated support scheme for electronic forms |
US7881990B2 (en) * | 2006-11-30 | 2011-02-01 | Intuit Inc. | Automatic time tracking based on user interface events |
US8060385B1 (en) | 2006-12-08 | 2011-11-15 | Safeco Insurance Company Of America | System, program product and method for segmenting and underwriting using voting status |
US7711462B2 (en) * | 2006-12-15 | 2010-05-04 | International Business Machines Corporation | Vehicle help system and method |
US8285732B2 (en) * | 2006-12-18 | 2012-10-09 | Sap Ag | Method and system for providing themes for software applications |
US8769099B2 (en) * | 2006-12-28 | 2014-07-01 | Yahoo! Inc. | Methods and systems for pre-caching information on a mobile computing device |
US8131536B2 (en) * | 2007-01-12 | 2012-03-06 | Raytheon Bbn Technologies Corp. | Extraction-empowered machine translation |
US8024660B1 (en) | 2007-01-31 | 2011-09-20 | Intuit Inc. | Method and apparatus for variable help content and abandonment intervention based on user behavior |
US20080189163A1 (en) * | 2007-02-05 | 2008-08-07 | Inquira, Inc. | Information management system |
US8635307B2 (en) * | 2007-02-08 | 2014-01-21 | Microsoft Corporation | Sensor discovery and configuration |
US7783586B2 (en) * | 2007-02-26 | 2010-08-24 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for analysis of biological systems |
US7792774B2 (en) * | 2007-02-26 | 2010-09-07 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for analysis of chaotic events |
US7917478B2 (en) * | 2007-02-26 | 2011-03-29 | International Business Machines Corporation | System and method for quality control in healthcare settings to continuously monitor outcomes and undesirable outcomes such as infections, re-operations, excess mortality, and readmissions |
US7788202B2 (en) * | 2007-02-26 | 2010-08-31 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for clinical applications |
US7853611B2 (en) * | 2007-02-26 | 2010-12-14 | International Business Machines Corporation | System and method for deriving a hierarchical event based database having action triggers based on inferred probabilities |
US7702605B2 (en) * | 2007-02-26 | 2010-04-20 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for privacy and security filtering |
US7805390B2 (en) * | 2007-02-26 | 2010-09-28 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for analysis of complex accidents |
US7970759B2 (en) | 2007-02-26 | 2011-06-28 | International Business Machines Corporation | System and method for deriving a hierarchical event based database optimized for pharmaceutical analysis |
US7788203B2 (en) * | 2007-02-26 | 2010-08-31 | International Business Machines Corporation | System and method of accident investigation for complex situations involving numerous known and unknown factors along with their probabilistic weightings |
US8281234B2 (en) * | 2007-03-20 | 2012-10-02 | Microsoft Corporation | Definable application assistant |
US7903970B2 (en) * | 2007-03-30 | 2011-03-08 | Georgia Tech Research Corporation | Optical network evaluation systems and methods |
JP2008257345A (en) * | 2007-04-02 | 2008-10-23 | Fuji Xerox Co Ltd | Information processing apparatus, information processing program and image processing system |
US7870491B1 (en) | 2007-04-27 | 2011-01-11 | Intuit Inc. | System and method for user support based on user interaction histories |
US7788204B1 (en) * | 2007-04-27 | 2010-08-31 | Intuit Inc. | System and method for adapting program elements based on application usage |
US7930676B1 (en) | 2007-04-27 | 2011-04-19 | Intuit Inc. | System and method for adapting software elements based on mood state profiling |
US20080288865A1 (en) * | 2007-05-16 | 2008-11-20 | Yahoo! Inc. | Application with in-context video assistance |
US7747988B2 (en) * | 2007-06-15 | 2010-06-29 | Microsoft Corporation | Software feature usage analysis and reporting |
US7680645B2 (en) * | 2007-06-15 | 2010-03-16 | Microsoft Corporation | Software feature modeling and recognition |
US7870114B2 (en) | 2007-06-15 | 2011-01-11 | Microsoft Corporation | Efficient data infrastructure for high dimensional data analysis |
US7739666B2 (en) * | 2007-06-15 | 2010-06-15 | Microsoft Corporation | Analyzing software users with instrumentation data and user group modeling and analysis |
US7996400B2 (en) * | 2007-06-23 | 2011-08-09 | Microsoft Corporation | Identification and use of web searcher expertise |
US7962344B2 (en) * | 2007-06-29 | 2011-06-14 | Microsoft Corporation | Depicting a speech user interface via graphical elements |
US9224147B2 (en) * | 2007-07-16 | 2015-12-29 | Hector Franco | Customer service management system |
US20090030902A1 (en) * | 2007-07-24 | 2009-01-29 | Microsoft Corporation | Schematized data intelligent assistance for development environments |
US8244503B1 (en) | 2007-08-30 | 2012-08-14 | The United States Of America As Represented By The Secretary Of The Air Force | Quantitative measurements of system complexity |
US20110145068A1 (en) * | 2007-09-17 | 2011-06-16 | King Martin T | Associating rendered advertisements with digital content |
US8307334B2 (en) * | 2007-09-17 | 2012-11-06 | International Business Machines Corporation | Method for assisting a user in the process of creating software code |
US20110035662A1 (en) * | 2009-02-18 | 2011-02-10 | King Martin T | Interacting with rendered documents using a multi-function mobile device, such as a mobile phone |
US8307333B2 (en) * | 2007-09-17 | 2012-11-06 | International Business Machines Corporation | System and computer program product for assisting a user in the process of creating software code |
US7836003B2 (en) * | 2007-09-21 | 2010-11-16 | Intel Corporation | Assisting a user experiencing cognitive decline by retracing footsteps |
US8595642B1 (en) | 2007-10-04 | 2013-11-26 | Great Northern Research, LLC | Multiple shell multi faceted graphical user interface |
US7890539B2 (en) * | 2007-10-10 | 2011-02-15 | Raytheon Bbn Technologies Corp. | Semantic matching using predicate-argument structure |
US8073682B2 (en) * | 2007-10-12 | 2011-12-06 | Palo Alto Research Center Incorporated | System and method for prospecting digital information |
US8671104B2 (en) | 2007-10-12 | 2014-03-11 | Palo Alto Research Center Incorporated | System and method for providing orientation into digital information |
US8165985B2 (en) * | 2007-10-12 | 2012-04-24 | Palo Alto Research Center Incorporated | System and method for performing discovery of digital information in a subject area |
US7930262B2 (en) | 2007-10-18 | 2011-04-19 | International Business Machines Corporation | System and method for the longitudinal analysis of education outcomes using cohort life cycles, cluster analytics-based cohort analysis, and probabilistic data schemas |
US8150791B2 (en) * | 2007-10-19 | 2012-04-03 | Hartwell Brown | Benefits services privacy architecture |
US20090138849A1 (en) * | 2007-10-30 | 2009-05-28 | Microsoft Corporation | Enhanced Prompting in a Visual Editor |
US9063979B2 (en) * | 2007-11-01 | 2015-06-23 | Ebay, Inc. | Analyzing event streams of user sessions |
US7940168B2 (en) * | 2007-11-19 | 2011-05-10 | Intel-Ge Care Innovations Llc | System, apparatus and method for automated emergency assistance with manual cancellation |
US10083420B2 (en) | 2007-11-21 | 2018-09-25 | Sermo, Inc | Community moderated information |
US8069142B2 (en) * | 2007-12-06 | 2011-11-29 | Yahoo! Inc. | System and method for synchronizing data on a network |
US8671154B2 (en) * | 2007-12-10 | 2014-03-11 | Yahoo! Inc. | System and method for contextual addressing of communications on a network |
US8307029B2 (en) * | 2007-12-10 | 2012-11-06 | Yahoo! Inc. | System and method for conditional delivery of messages |
US8166168B2 (en) | 2007-12-17 | 2012-04-24 | Yahoo! Inc. | System and method for disambiguating non-unique identifiers using information obtained from disparate communication channels |
US7779051B2 (en) * | 2008-01-02 | 2010-08-17 | International Business Machines Corporation | System and method for optimizing federated and ETL'd databases with considerations of specialized data structures within an environment having multidimensional constraints |
US9706345B2 (en) * | 2008-01-04 | 2017-07-11 | Excalibur Ip, Llc | Interest mapping system |
US9626685B2 (en) | 2008-01-04 | 2017-04-18 | Excalibur Ip, Llc | Systems and methods of mapping attention |
US8762285B2 (en) * | 2008-01-06 | 2014-06-24 | Yahoo! Inc. | System and method for message clustering |
US20090182618A1 (en) | 2008-01-16 | 2009-07-16 | Yahoo! Inc. | System and Method for Word-of-Mouth Advertising |
US8151192B2 (en) * | 2008-02-01 | 2012-04-03 | Microsoft Corporation | Context sensitive help |
US8554623B2 (en) | 2008-03-03 | 2013-10-08 | Yahoo! Inc. | Method and apparatus for social network marketing with consumer referral |
US8538811B2 (en) * | 2008-03-03 | 2013-09-17 | Yahoo! Inc. | Method and apparatus for social network marketing with advocate referral |
US8560390B2 (en) | 2008-03-03 | 2013-10-15 | Yahoo! Inc. | Method and apparatus for social network marketing with brand referral |
US20090240539A1 (en) * | 2008-03-21 | 2009-09-24 | Microsoft Corporation | Machine learning system for a task brokerage system |
US20090240549A1 (en) * | 2008-03-21 | 2009-09-24 | Microsoft Corporation | Recommendation system for a task brokerage system |
US8589486B2 (en) | 2008-03-28 | 2013-11-19 | Yahoo! Inc. | System and method for addressing communications |
US8745133B2 (en) | 2008-03-28 | 2014-06-03 | Yahoo! Inc. | System and method for optimizing the storage of data |
US8271506B2 (en) | 2008-03-31 | 2012-09-18 | Yahoo! Inc. | System and method for modeling relationships between entities |
US8808178B2 (en) * | 2008-04-30 | 2014-08-19 | Welch Allyn, Inc. | On demand help/in-service for a medical device |
US20090299766A1 (en) * | 2008-05-30 | 2009-12-03 | International Business Machines Corporation | System and method for optimizing medical treatment planning and support in difficult situations subject to multiple constraints and uncertainties |
US20090300495A1 (en) * | 2008-06-02 | 2009-12-03 | Novell, Inc. | System and method for an adaptive wizard utility for graphical user interfaces |
US8812809B2 (en) | 2008-06-10 | 2014-08-19 | Oracle America, Inc. | Method and apparatus for allocating memory for immutable data on a computing device |
US8010512B2 (en) * | 2008-06-16 | 2011-08-30 | International Business Machines Corporation | System and method for model-driven object store |
US8103598B2 (en) * | 2008-06-20 | 2012-01-24 | Microsoft Corporation | Compiler for probabilistic programs |
US9396099B2 (en) * | 2008-06-24 | 2016-07-19 | International Business Machines Corporation | Application state detector and inducer |
US8411085B2 (en) | 2008-06-27 | 2013-04-02 | Microsoft Corporation | Constructing view compositions for domain-specific environments |
US8706406B2 (en) * | 2008-06-27 | 2014-04-22 | Yahoo! Inc. | System and method for determination and display of personalized distance |
US20090322739A1 (en) * | 2008-06-27 | 2009-12-31 | Microsoft Corporation | Visual Interactions with Analytics |
US8620635B2 (en) * | 2008-06-27 | 2013-12-31 | Microsoft Corporation | Composition of analytics models |
US8813107B2 (en) | 2008-06-27 | 2014-08-19 | Yahoo! Inc. | System and method for location based media delivery |
US8452855B2 (en) | 2008-06-27 | 2013-05-28 | Yahoo! Inc. | System and method for presentation of media related to a context |
US8117145B2 (en) * | 2008-06-27 | 2012-02-14 | Microsoft Corporation | Analytical model solver framework |
US8255192B2 (en) * | 2008-06-27 | 2012-08-28 | Microsoft Corporation | Analytical map models |
WO2010006087A1 (en) * | 2008-07-08 | 2010-01-14 | David Seaberg | Process for providing and editing instructions, data, data structures, and algorithms in a computer system |
US20110125702A1 (en) * | 2008-07-10 | 2011-05-26 | Prasanna Gorur Narayana Srinivasa | Decision support methods under uncertainty |
US8799200B2 (en) | 2008-07-25 | 2014-08-05 | Liveperson, Inc. | Method and system for creating a predictive model for targeting webpage to a surfer |
US8762313B2 (en) | 2008-07-25 | 2014-06-24 | Liveperson, Inc. | Method and system for creating a predictive model for targeting web-page to a surfer |
US8086700B2 (en) | 2008-07-29 | 2011-12-27 | Yahoo! Inc. | Region and duration uniform resource identifiers (URI) for media objects |
US8583668B2 (en) | 2008-07-30 | 2013-11-12 | Yahoo! Inc. | System and method for context enhanced mapping |
US10230803B2 (en) * | 2008-07-30 | 2019-03-12 | Excalibur Ip, Llc | System and method for improved mapping and routing |
US8805844B2 (en) | 2008-08-04 | 2014-08-12 | Liveperson, Inc. | Expert search |
US20100049800A1 (en) * | 2008-08-20 | 2010-02-25 | International Business Machines Corporation | Facilitated Help With Program Function From Experienced Users |
US8386506B2 (en) * | 2008-08-21 | 2013-02-26 | Yahoo! Inc. | System and method for context enhanced messaging |
US9940582B2 (en) * | 2008-08-27 | 2018-04-10 | International Business Machines Corporation | Intelligent problem tracking electronic system for optimizing technical support |
US8010545B2 (en) * | 2008-08-28 | 2011-08-30 | Palo Alto Research Center Incorporated | System and method for providing a topic-directed search |
US20100057577A1 (en) * | 2008-08-28 | 2010-03-04 | Palo Alto Research Center Incorporated | System And Method For Providing Topic-Guided Broadening Of Advertising Targets In Social Indexing |
US8209616B2 (en) * | 2008-08-28 | 2012-06-26 | Palo Alto Research Center Incorporated | System and method for interfacing a web browser widget with social indexing |
US20100057536A1 (en) * | 2008-08-28 | 2010-03-04 | Palo Alto Research Center Incorporated | System And Method For Providing Community-Based Advertising Term Disambiguation |
US20100063993A1 (en) * | 2008-09-08 | 2010-03-11 | Yahoo! Inc. | System and method for socially aware identity manager |
US8949161B2 (en) * | 2008-09-12 | 2015-02-03 | Alcatel Lucent | Cache management system and method and content distribution system incorporating the same |
US8281027B2 (en) * | 2008-09-19 | 2012-10-02 | Yahoo! Inc. | System and method for distributing media related to a location |
US8108778B2 (en) * | 2008-09-30 | 2012-01-31 | Yahoo! Inc. | System and method for context enhanced mapping within a user interface |
US9600484B2 (en) * | 2008-09-30 | 2017-03-21 | Excalibur Ip, Llc | System and method for reporting and analysis of media consumption data |
KR101025743B1 (en) * | 2008-10-13 | 2011-04-04 | 한국전자통신연구원 | The artificial retina driving apparatus using middle-distance wireless power transfer technology |
US9892417B2 (en) | 2008-10-29 | 2018-02-13 | Liveperson, Inc. | System and method for applying tracing tools for network locations |
US8549016B2 (en) * | 2008-11-14 | 2013-10-01 | Palo Alto Research Center Incorporated | System and method for providing robust topic identification in social indexes |
US9805123B2 (en) * | 2008-11-18 | 2017-10-31 | Excalibur Ip, Llc | System and method for data privacy in URL based context queries |
US8024317B2 (en) | 2008-11-18 | 2011-09-20 | Yahoo! Inc. | System and method for deriving income from URL based context queries |
US8060492B2 (en) * | 2008-11-18 | 2011-11-15 | Yahoo! Inc. | System and method for generation of URL based context queries |
US8032508B2 (en) * | 2008-11-18 | 2011-10-04 | Yahoo! Inc. | System and method for URL based query for retrieving data related to a context |
US8190406B2 (en) * | 2008-11-26 | 2012-05-29 | Microsoft Corporation | Hybrid solver for data-driven analytics |
US8155931B2 (en) * | 2008-11-26 | 2012-04-10 | Microsoft Corporation | Use of taxonomized analytics reference model |
US8103608B2 (en) * | 2008-11-26 | 2012-01-24 | Microsoft Corporation | Reference model for data-driven analytics |
US8145615B2 (en) * | 2008-11-26 | 2012-03-27 | Microsoft Corporation | Search and exploration using analytics reference model |
US9224172B2 (en) | 2008-12-02 | 2015-12-29 | Yahoo! Inc. | Customizable content for distribution in social networks |
US8055675B2 (en) | 2008-12-05 | 2011-11-08 | Yahoo! Inc. | System and method for context based query augmentation |
US8166016B2 (en) * | 2008-12-19 | 2012-04-24 | Yahoo! Inc. | System and method for automated service recommendations |
US8314793B2 (en) | 2008-12-24 | 2012-11-20 | Microsoft Corporation | Implied analytical reasoning and computation |
US8090750B2 (en) | 2008-12-30 | 2012-01-03 | International Business Machines Corporation | Prompting of an end user with commands |
US20100185517A1 (en) * | 2009-01-21 | 2010-07-22 | Yahoo! Inc. | User interface for interest-based targeted marketing |
US20100185518A1 (en) * | 2009-01-21 | 2010-07-22 | Yahoo! Inc. | Interest-based activity marketing |
US8239397B2 (en) * | 2009-01-27 | 2012-08-07 | Palo Alto Research Center Incorporated | System and method for managing user attention by detecting hot and cold topics in social indexes |
US8452781B2 (en) * | 2009-01-27 | 2013-05-28 | Palo Alto Research Center Incorporated | System and method for using banded topic relevance and time for article prioritization |
US8356044B2 (en) * | 2009-01-27 | 2013-01-15 | Palo Alto Research Center Incorporated | System and method for providing default hierarchical training for social indexing |
IL197196A0 (en) * | 2009-02-23 | 2009-12-24 | Univ Ben Gurion | Intention prediction using hidden markov models and user profile |
US20100228582A1 (en) * | 2009-03-06 | 2010-09-09 | Yahoo! Inc. | System and method for contextual advertising based on status messages |
EP2406767A4 (en) * | 2009-03-12 | 2016-03-16 | Google Inc | Automatically providing content associated with captured information, such as information captured in real-time |
US8447066B2 (en) | 2009-03-12 | 2013-05-21 | Google Inc. | Performing actions based on capturing information from rendered documents, such as documents under copyright |
US20100241689A1 (en) * | 2009-03-19 | 2010-09-23 | Yahoo! Inc. | Method and apparatus for associating advertising with computer enabled maps |
US8150967B2 (en) * | 2009-03-24 | 2012-04-03 | Yahoo! Inc. | System and method for verified presence tracking |
US20100251118A1 (en) * | 2009-03-26 | 2010-09-30 | Peterson Michael L | Help utility with expanded content display |
US20100280913A1 (en) * | 2009-05-01 | 2010-11-04 | Yahoo! Inc. | Gift credit matching engine |
US20100280879A1 (en) * | 2009-05-01 | 2010-11-04 | Yahoo! Inc. | Gift incentive engine |
KR101562792B1 (en) * | 2009-06-10 | 2015-10-23 | 삼성전자주식회사 | Apparatus and method for providing goal predictive interface |
US8259134B2 (en) * | 2009-06-19 | 2012-09-04 | Microsoft Corporation | Data-driven model implemented with spreadsheets |
US8493406B2 (en) | 2009-06-19 | 2013-07-23 | Microsoft Corporation | Creating new charts and data visualizations |
US8692826B2 (en) | 2009-06-19 | 2014-04-08 | Brian C. Beckman | Solver-based visualization framework |
US9330503B2 (en) | 2009-06-19 | 2016-05-03 | Microsoft Technology Licensing, Llc | Presaging and surfacing interactivity within data visualizations |
US8866818B2 (en) | 2009-06-19 | 2014-10-21 | Microsoft Corporation | Composing shapes and data series in geometries |
US8788574B2 (en) | 2009-06-19 | 2014-07-22 | Microsoft Corporation | Data-driven visualization of pseudo-infinite scenes |
US8531451B2 (en) | 2009-06-19 | 2013-09-10 | Microsoft Corporation | Data-driven visualization transformation |
US10223701B2 (en) * | 2009-08-06 | 2019-03-05 | Excalibur Ip, Llc | System and method for verified monetization of commercial campaigns |
US8914342B2 (en) | 2009-08-12 | 2014-12-16 | Yahoo! Inc. | Personal data platform |
US8364611B2 (en) | 2009-08-13 | 2013-01-29 | Yahoo! Inc. | System and method for precaching information on a mobile device |
US8352397B2 (en) | 2009-09-10 | 2013-01-08 | Microsoft Corporation | Dependency graph in data-driven model |
US20110087515A1 (en) * | 2009-10-08 | 2011-04-14 | Miller Bradford W | Cognitive interactive mission planning system and method |
US8635584B2 (en) * | 2009-10-13 | 2014-01-21 | Yahoo! Inc. | Tutorial systems for code creation and provenance tracking |
JP5316363B2 (en) | 2009-10-20 | 2013-10-16 | ソニー株式会社 | Information processing apparatus, function management method, computer program, and information processing system |
US9081799B2 (en) * | 2009-12-04 | 2015-07-14 | Google Inc. | Using gestalt information to identify locations in printed information |
US9323784B2 (en) * | 2009-12-09 | 2016-04-26 | Google Inc. | Image search using text-based elements within the contents of images |
EP2339576B1 (en) * | 2009-12-23 | 2019-08-07 | Google LLC | Multi-modal input on an electronic device |
US11416214B2 (en) | 2009-12-23 | 2022-08-16 | Google Llc | Multi-modal input on an electronic device |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US8370156B1 (en) * | 2010-01-20 | 2013-02-05 | The Mathworks, Inc. | Editing suggestions in different formalisms |
US9805101B2 (en) * | 2010-02-26 | 2017-10-31 | Ebay Inc. | Parallel data stream processing system |
US20110238603A1 (en) * | 2010-03-29 | 2011-09-29 | Raytheon Company | System and Method for Predicting Events Via Dynamic Ontologies |
EP2556449A1 (en) | 2010-04-07 | 2013-02-13 | Liveperson Inc. | System and method for dynamically enabling customized web content and applications |
US8396815B2 (en) * | 2010-04-29 | 2013-03-12 | International Business Machines Corporation | Adaptive business process automation |
US9031944B2 (en) | 2010-04-30 | 2015-05-12 | Palo Alto Research Center Incorporated | System and method for providing multi-core and multi-level topical organization in social indexes |
CN102238268B (en) * | 2010-04-30 | 2013-10-30 | 腾讯科技(深圳)有限公司 | Information prompting method and device |
WO2011149558A2 (en) | 2010-05-28 | 2011-12-01 | Abelow Daniel H | Reality alternate |
JP2011253375A (en) * | 2010-06-02 | 2011-12-15 | Sony Corp | Information processing device, information processing method and program |
JP2011253374A (en) * | 2010-06-02 | 2011-12-15 | Sony Corp | Information processing device, information processing method and program |
US8560365B2 (en) | 2010-06-08 | 2013-10-15 | International Business Machines Corporation | Probabilistic optimization of resource discovery, reservation and assignment |
US9043296B2 (en) | 2010-07-30 | 2015-05-26 | Microsoft Technology Licensing, Llc | System of providing suggestions based on accessible and contextual information |
US9646271B2 (en) | 2010-08-06 | 2017-05-09 | International Business Machines Corporation | Generating candidate inclusion/exclusion cohorts for a multiply constrained group |
US8968197B2 (en) | 2010-09-03 | 2015-03-03 | International Business Machines Corporation | Directing a user to a medical resource |
US9292577B2 (en) | 2010-09-17 | 2016-03-22 | International Business Machines Corporation | User accessibility to data analytics |
US9443211B2 (en) | 2010-10-13 | 2016-09-13 | International Business Machines Corporation | Describing a paradigmatic member of a task directed community in a complex heterogeneous environment based on non-linear attributes |
US8429182B2 (en) | 2010-10-13 | 2013-04-23 | International Business Machines Corporation | Populating a task directed community in a complex heterogeneous environment based on non-linear attributes of a paradigmatic cohort member |
US10318877B2 (en) | 2010-10-19 | 2019-06-11 | International Business Machines Corporation | Cohort-based prediction of a future event |
US20120136883A1 (en) * | 2010-11-27 | 2012-05-31 | Kwabi Christopher K | Automatic Dynamic Multi-Variable Matching Engine |
US9589254B2 (en) | 2010-12-08 | 2017-03-07 | Microsoft Technology Licensing, Llc | Using e-mail message characteristics for prioritization |
US8918465B2 (en) | 2010-12-14 | 2014-12-23 | Liveperson, Inc. | Authentication of service requests initiated from a social networking site |
US9350598B2 (en) | 2010-12-14 | 2016-05-24 | Liveperson, Inc. | Authentication of service requests using a communications initiation feature |
US9141405B2 (en) * | 2010-12-15 | 2015-09-22 | International Business Machines Corporation | User interface construction |
US20120159341A1 (en) | 2010-12-21 | 2012-06-21 | Microsoft Corporation | Interactions with contextual and task-based computing environments |
US8352245B1 (en) | 2010-12-30 | 2013-01-08 | Google Inc. | Adjusting language models |
US8296142B2 (en) | 2011-01-21 | 2012-10-23 | Google Inc. | Speech recognition using dock context |
US9182879B2 (en) * | 2011-03-29 | 2015-11-10 | Schlumberger Technology Corporation | Immersive interaction model interpretation |
US9842168B2 (en) | 2011-03-31 | 2017-12-12 | Microsoft Technology Licensing, Llc | Task driven user intents |
US9298287B2 (en) | 2011-03-31 | 2016-03-29 | Microsoft Technology Licensing, Llc | Combined activation for natural user interface systems |
US9760566B2 (en) | 2011-03-31 | 2017-09-12 | Microsoft Technology Licensing, Llc | Augmented conversational understanding agent to identify conversation context between two humans and taking an agent action thereof |
US10642934B2 (en) | 2011-03-31 | 2020-05-05 | Microsoft Technology Licensing, Llc | Augmented conversational understanding architecture |
US8806444B1 (en) * | 2011-04-07 | 2014-08-12 | Intuit Inc. | Modifying software based on tracked activities |
US20120281704A1 (en) * | 2011-05-02 | 2012-11-08 | Butterworth Ashley I | Methods and apparatus for isochronous data delivery within a network |
US9064006B2 (en) | 2012-08-23 | 2015-06-23 | Microsoft Technology Licensing, Llc | Translating natural language utterances to keyword search queries |
JP5496947B2 (en) * | 2011-05-27 | 2014-05-21 | 株式会社Nttドコモ | Function presentation system, terminal device, and program |
US9489375B2 (en) | 2011-06-19 | 2016-11-08 | Mmodal Ip Llc | Speech recognition using an operating system hooking component for context-aware recognition models |
US8843851B1 (en) * | 2011-07-28 | 2014-09-23 | Intuit Inc. | Proactive chat support |
CN102955763B (en) * | 2011-08-22 | 2016-07-06 | 联想(北京)有限公司 | Display packing and display device |
US8954364B2 (en) | 2011-09-19 | 2015-02-10 | International Business Machines Corporation | Hierarchical contexts to drive live sensor applications |
US9495962B2 (en) * | 2011-09-19 | 2016-11-15 | Personetics Technologies Ltd. | System and method for evaluating intent of a human partner to a dialogue between human user and computerized system |
US8214904B1 (en) | 2011-12-21 | 2012-07-03 | Kaspersky Lab Zao | System and method for detecting computer security threats based on verdicts of computer users |
CN103827758B (en) * | 2011-09-30 | 2016-10-05 | 西门子瑞士有限公司 | A kind of for the equipment in building automation system |
US9256396B2 (en) * | 2011-10-10 | 2016-02-09 | Microsoft Technology Licensing, Llc | Speech recognition for context switching |
US9454528B2 (en) * | 2011-10-17 | 2016-09-27 | Xerox Corporation | Method and system for creating ordered reading lists from unstructured document sets |
US8214905B1 (en) | 2011-12-21 | 2012-07-03 | Kaspersky Lab Zao | System and method for dynamically allocating computing resources for processing security information |
US8209758B1 (en) | 2011-12-21 | 2012-06-26 | Kaspersky Lab Zao | System and method for classifying users of antivirus software based on their level of expertise in the field of computer security |
US8943002B2 (en) | 2012-02-10 | 2015-01-27 | Liveperson, Inc. | Analytics driven engagement |
USD752086S1 (en) * | 2012-02-24 | 2016-03-22 | Samsung Electronics Co., Ltd. | Portable electronic device with an animated graphical user interface |
US8805941B2 (en) | 2012-03-06 | 2014-08-12 | Liveperson, Inc. | Occasionally-connected computing interface |
US9411619B2 (en) | 2012-03-08 | 2016-08-09 | Vmware, Inc. | Performance management of system objects based on consequence probabilities |
CN104503895A (en) * | 2012-03-29 | 2015-04-08 | 北京奇虎科技有限公司 | Function prompting method and function prompting device |
US20130262365A1 (en) * | 2012-03-31 | 2013-10-03 | Sharp Kabushiki Kaisha | Educational system, method and program to adapt learning content based on predicted user reaction |
US9275342B2 (en) * | 2012-04-09 | 2016-03-01 | 24/7 Customer, Inc. | Method and apparatus for intent modeling and prediction |
US9563336B2 (en) | 2012-04-26 | 2017-02-07 | Liveperson, Inc. | Dynamic user interface customization |
US9672196B2 (en) | 2012-05-15 | 2017-06-06 | Liveperson, Inc. | Methods and systems for presenting specialized content using campaign metrics |
US9098111B2 (en) | 2012-06-22 | 2015-08-04 | Microsoft Technology Licensing, Llc | Focus guidance within a three-dimensional interface |
US9465833B2 (en) | 2012-07-31 | 2016-10-11 | Veveo, Inc. | Disambiguating user intent in conversational interaction system for large corpus information retrieval |
US9799328B2 (en) * | 2012-08-03 | 2017-10-24 | Veveo, Inc. | Method for using pauses detected in speech input to assist in interpreting the input during conversational interaction for information retrieval |
US9720527B2 (en) | 2012-08-06 | 2017-08-01 | Tracfone Wireless, Inc. | Evolutionary touch-based graphical user interface for electronic devices |
US20140047334A1 (en) * | 2012-08-09 | 2014-02-13 | Sap Ag | Computer application learning solution |
US10163058B2 (en) * | 2012-08-14 | 2018-12-25 | Sri International | Method, system and device for inferring a mobile user's current context and proactively providing assistance |
US9015099B2 (en) * | 2012-08-14 | 2015-04-21 | Sri International | Method, system and device for inferring a mobile user's current context and proactively providing assistance |
US20140149177A1 (en) * | 2012-11-23 | 2014-05-29 | Ari M. Frank | Responding to uncertainty of a user regarding an experience by presenting a prior experience |
JP5823996B2 (en) | 2013-01-31 | 2015-11-25 | グリー株式会社 | COMMUNICATION SYSTEM, COMMUNICATION SYSTEM CONTROL METHOD, AND PROGRAM |
US20140245141A1 (en) * | 2013-02-26 | 2014-08-28 | Microsoft Corporation | Contextual user assistance for cloud services |
US9864498B2 (en) | 2013-03-13 | 2018-01-09 | Tobii Ab | Automatic scrolling based on gaze detection |
US20140247232A1 (en) | 2013-03-01 | 2014-09-04 | Tobii Technology Ab | Two step gaze interaction |
US9251467B2 (en) | 2013-03-03 | 2016-02-02 | Microsoft Technology Licensing, Llc | Probabilistic parsing |
US9256341B2 (en) | 2013-03-20 | 2016-02-09 | Microsoft Technology Licensing, Llc | Tracking changes in collaborative authoring environment |
US20140310681A1 (en) * | 2013-04-12 | 2014-10-16 | Microsoft Corporation | Assisted creation of control event |
DK2994908T3 (en) * | 2013-05-07 | 2019-09-23 | Veveo Inc | INTERFACE FOR INCREMENTAL SPEECH INPUT WITH REALTIME FEEDBACK |
US9804730B2 (en) | 2013-06-03 | 2017-10-31 | Microsoft Technology Licensing, Llc | Automatically changing a display of graphical user interface |
CN104298683B (en) * | 2013-07-18 | 2017-11-28 | 佳能株式会社 | Topics Crawling method and apparatus and enquiry expanding method and equipment |
CN104469709B (en) * | 2013-09-13 | 2018-08-10 | 联想(北京)有限公司 | Identify the method and electronic equipment of short message |
US10558262B2 (en) | 2013-11-18 | 2020-02-11 | Tobii Ab | Component determination and gaze provoked interaction |
US10317995B2 (en) | 2013-11-18 | 2019-06-11 | Tobii Ab | Component determination and gaze provoked interaction |
US10824787B2 (en) * | 2013-12-21 | 2020-11-03 | Microsoft Technology Licensing, Llc | Authoring through crowdsourcing based suggestions |
US11514399B2 (en) * | 2013-12-21 | 2022-11-29 | Microsoft Technology Licensing, Llc | Authoring through suggestion |
WO2015097689A1 (en) * | 2013-12-29 | 2015-07-02 | Inuitive Ltd. | A device and a method for establishing a personal digital profile of a user |
US10133589B2 (en) * | 2013-12-31 | 2018-11-20 | Microsoft Technology Licensing, Llc | Identifying help information based on application context |
JP2015133047A (en) | 2014-01-15 | 2015-07-23 | ソニー株式会社 | Information processing apparatus, information processing method, and program |
US9716745B2 (en) * | 2014-01-18 | 2017-07-25 | International Business Machines Corporation | Expertise-matched help systems |
US9842592B2 (en) | 2014-02-12 | 2017-12-12 | Google Inc. | Language models using non-linguistic context |
US9432472B2 (en) | 2014-02-24 | 2016-08-30 | Microsoft Technology Licensing, Llc | Accelerated training of personal daemons |
US9473944B2 (en) | 2014-02-24 | 2016-10-18 | Microsoft Technology Licensing, Llc | Local personal daemon |
US9218497B2 (en) | 2014-02-24 | 2015-12-22 | Microsoft Technology Licensing, Llc | Incentive-based app execution |
US9412365B2 (en) | 2014-03-24 | 2016-08-09 | Google Inc. | Enhanced maximum entropy models |
US11386442B2 (en) | 2014-03-31 | 2022-07-12 | Liveperson, Inc. | Online behavioral predictor |
US9560055B2 (en) | 2014-04-30 | 2017-01-31 | Microsoft Technology Licensing, Llc | Client-side integration framework of services |
US11100434B2 (en) | 2014-05-06 | 2021-08-24 | Uber Technologies, Inc. | Real-time carpooling coordinating system and methods |
US9552559B2 (en) | 2014-05-06 | 2017-01-24 | Elwha Llc | System and methods for verifying that one or more directives that direct transport of a second end user does not conflict with one or more obligations to transport a first end user |
US9483744B2 (en) | 2014-05-06 | 2016-11-01 | Elwha Llc | Real-time carpooling coordinating systems and methods |
US10458801B2 (en) | 2014-05-06 | 2019-10-29 | Uber Technologies, Inc. | Systems and methods for travel planning that calls for at least one transportation vehicle unit |
US11017412B2 (en) * | 2014-05-23 | 2021-05-25 | Microsoft Technology Licensing, Llc | Contextual information monitoring |
US20150356061A1 (en) * | 2014-06-06 | 2015-12-10 | Microsoft Corporation | Summary view suggestion based on user interaction pattern |
DE102014211094B4 (en) * | 2014-06-11 | 2016-04-07 | BSH Hausgeräte GmbH | Household appliance with a statistically controllable operation and method for controllably operating a household appliance |
CN104166498B (en) * | 2014-07-08 | 2018-02-23 | 惠州Tcl移动通信有限公司 | Show the method and mobile terminal of instruction manual book |
KR102327403B1 (en) * | 2014-08-01 | 2021-11-17 | 대우조선해양 주식회사 | Personalized support systems based on the event log and methods thereof |
US9952883B2 (en) | 2014-08-05 | 2018-04-24 | Tobii Ab | Dynamic determination of hardware |
US9778628B2 (en) | 2014-08-07 | 2017-10-03 | Goodrich Corporation | Optimization of human supervisors and cyber-physical systems |
US9934507B2 (en) * | 2014-08-11 | 2018-04-03 | International Business Machines Corporation | Mapping user actions to historical paths to determine a predicted endpoint |
US10460342B1 (en) * | 2014-08-12 | 2019-10-29 | Benko, LLC | Methods and software for providing targeted advertising to a product program |
US9953646B2 (en) | 2014-09-02 | 2018-04-24 | Belleau Technologies | Method and system for dynamic speech recognition and tracking of prewritten script |
US11283924B2 (en) * | 2014-09-15 | 2022-03-22 | Avaya Inc. | System and method for guiding agents in an enterprise over aggregated interactions |
US9058563B1 (en) * | 2014-10-15 | 2015-06-16 | Blackwerks LLC | Suggesting activities |
CN104317402B (en) * | 2014-10-24 | 2020-07-03 | 小米科技有限责任公司 | Description information display method and device and electronic equipment |
CN105893385B (en) * | 2015-01-04 | 2020-10-23 | 伊姆西Ip控股有限责任公司 | Method and apparatus for analyzing user behavior |
US9854049B2 (en) | 2015-01-30 | 2017-12-26 | Rovi Guides, Inc. | Systems and methods for resolving ambiguous terms in social chatter based on a user profile |
US10134394B2 (en) | 2015-03-20 | 2018-11-20 | Google Llc | Speech recognition using log-linear model |
US10326768B2 (en) * | 2015-05-28 | 2019-06-18 | Google Llc | Access control for enterprise knowledge |
CN104866102B (en) * | 2015-05-29 | 2018-10-23 | 中山大学 | A kind of the operation intelligence control system and method for blind person's computer |
AU2016270937B2 (en) | 2015-06-02 | 2021-07-29 | Liveperson, Inc. | Dynamic communication routing based on consistency weighting and routing rules |
US10409443B2 (en) | 2015-06-24 | 2019-09-10 | Microsoft Technology Licensing, Llc | Contextual cursor display based on hand tracking |
US10621499B1 (en) | 2015-08-03 | 2020-04-14 | Marca Research & Development International, Llc | Systems and methods for semantic understanding of digital information |
US10073890B1 (en) | 2015-08-03 | 2018-09-11 | Marca Research & Development International, Llc | Systems and methods for patent reference comparison in a combined semantical-probabilistic algorithm |
CN105047198B (en) * | 2015-08-24 | 2020-09-22 | 百度在线网络技术(北京)有限公司 | Voice error correction processing method and device |
US10331464B2 (en) * | 2015-09-17 | 2019-06-25 | Dropbox, Inc. | Method and system for an adaptive contextual instruction tool |
US20170115838A1 (en) * | 2015-10-27 | 2017-04-27 | Ayori Zena Selassie | Method and apparatus for personal introspection tool |
US10956666B2 (en) | 2015-11-09 | 2021-03-23 | Apple Inc. | Unconventional virtual assistant interactions |
US10146397B2 (en) | 2015-11-27 | 2018-12-04 | International Business Machines Corporation | User experience steering |
US9654639B1 (en) | 2015-12-10 | 2017-05-16 | Microsoft Technology Licensing, Llc | Resource partitioning for routing on-demand services |
US10275775B2 (en) | 2015-12-10 | 2019-04-30 | Microsoft Technology Licensing, Llc | Context generation for routing on-demand services |
US10223174B2 (en) | 2015-12-10 | 2019-03-05 | Microsoft Technology Licensing, Llc | Tenant engagement signal acquisition and exposure |
US9686406B1 (en) | 2015-12-10 | 2017-06-20 | Microsoft Technology Licensing, Llc | Issue detection for routing assistance requests |
US10353564B2 (en) | 2015-12-21 | 2019-07-16 | Sap Se | Graphical user interface with virtual extension areas |
US10606618B2 (en) | 2016-01-19 | 2020-03-31 | Adp, Llc | Contextual assistance system |
US20170212886A1 (en) * | 2016-01-22 | 2017-07-27 | Microsoft Technology Licensing, Llc | Configurable Generic Language Understanding Models |
US20170212650A1 (en) * | 2016-01-22 | 2017-07-27 | Microsoft Technology Licensing, Llc | Dynamically optimizing user engagement |
US9922057B2 (en) * | 2016-02-09 | 2018-03-20 | International Business Machines Corporation | Dynamic construction of knowledge base by usage pattern recognition |
US9978367B2 (en) | 2016-03-16 | 2018-05-22 | Google Llc | Determining dialog states for language models |
US9928106B2 (en) | 2016-03-29 | 2018-03-27 | Wipro Limited | System and methods for dynamically assigning control to one or more BOTs |
US10212464B2 (en) * | 2016-04-15 | 2019-02-19 | Hulu, LLC | Generation, ranking, and delivery of actions for entities in a video delivery system |
US10540439B2 (en) | 2016-04-15 | 2020-01-21 | Marca Research & Development International, Llc | Systems and methods for identifying evidentiary information |
US10579238B2 (en) | 2016-05-13 | 2020-03-03 | Sap Se | Flexible screen layout across multiple platforms |
US10353534B2 (en) | 2016-05-13 | 2019-07-16 | Sap Se | Overview page in multi application user interface |
US20170329505A1 (en) | 2016-05-13 | 2017-11-16 | Sap Se | Transferring parameters in applications and interfaces across multiple platforms |
US10318253B2 (en) | 2016-05-13 | 2019-06-11 | Sap Se | Smart templates for use in multiple platforms |
US11093834B2 (en) * | 2016-07-06 | 2021-08-17 | Palo Alto Research Center Incorporated | Computer-implemented system and method for predicting activity outcome based on user attention |
US11222270B2 (en) | 2016-07-28 | 2022-01-11 | International Business Machiness Corporation | Using learned application flow to predict outcomes and identify trouble spots in network business transactions |
US11030673B2 (en) * | 2016-07-28 | 2021-06-08 | International Business Machines Corporation | Using learned application flow to assist users in network business transaction based apps |
US10278065B2 (en) | 2016-08-14 | 2019-04-30 | Liveperson, Inc. | Systems and methods for real-time remote control of mobile applications |
US10832664B2 (en) | 2016-08-19 | 2020-11-10 | Google Llc | Automated speech recognition using language models that selectively use domain-specific model components |
CN106297478A (en) * | 2016-08-23 | 2017-01-04 | 郑州师范学院 | A kind of piano intelligent tutoring system |
US11200026B2 (en) * | 2016-08-26 | 2021-12-14 | Bragi GmbH | Wireless earpiece with a passive virtual assistant |
US20180075128A1 (en) * | 2016-09-13 | 2018-03-15 | Adobe Systems Incorporated | Identifying Key Terms Related to an Entity |
US10579400B2 (en) * | 2016-11-11 | 2020-03-03 | International Business Machines Corporation | Path-sensitive contextual help system |
CN108073600B (en) * | 2016-11-11 | 2022-06-03 | 阿里巴巴集团控股有限公司 | Intelligent question-answer interaction method and device and electronic equipment |
US10963642B2 (en) * | 2016-11-28 | 2021-03-30 | Microsoft Technology Licensing, Llc | Intelligent assistant help system |
US10810380B2 (en) * | 2016-12-21 | 2020-10-20 | Facebook, Inc. | Transliteration using machine translation pipeline |
US10878192B2 (en) * | 2017-01-06 | 2020-12-29 | Microsoft Technology Licensing, Llc | Contextual document recall |
US10915303B2 (en) | 2017-01-26 | 2021-02-09 | Sap Se | Run time integrated development and modification system |
JP2018128829A (en) * | 2017-02-08 | 2018-08-16 | 富士ゼロックス株式会社 | Information processing apparatus and information processing program |
US10311860B2 (en) | 2017-02-14 | 2019-06-04 | Google Llc | Language model biasing system |
WO2018205084A1 (en) * | 2017-05-08 | 2018-11-15 | Microsoft Technology Licensing, Llc | Providing local service information in automated chatting |
CN108874812B (en) * | 2017-05-10 | 2021-12-10 | 腾讯科技(北京)有限公司 | Data processing method, server and computer storage medium |
US10540339B2 (en) | 2017-05-19 | 2020-01-21 | Microsoft Technology Licensing, Llc | Matching a resource with a user for a predicted user need |
US10528228B2 (en) | 2017-06-21 | 2020-01-07 | Microsoft Technology Licensing, Llc | Interaction with notifications across devices with a digital assistant |
CN109308896B (en) * | 2017-07-28 | 2022-04-15 | 江苏汇通金科数据股份有限公司 | Voice processing method and device, storage medium and processor |
US10885808B2 (en) | 2017-09-28 | 2021-01-05 | International Business Machines Corporation | Curating tutorials based on historic user data |
US20190286968A1 (en) * | 2018-03-16 | 2019-09-19 | International Business Machines Corporation | Cognitive adaption of recommendation system |
DK201870360A1 (en) * | 2018-06-03 | 2019-12-20 | Apple Inc. | Accelerated task performance |
US11200811B2 (en) * | 2018-08-03 | 2021-12-14 | International Business Machines Corporation | Intelligent recommendation of guidance instructions |
CN109801625A (en) * | 2018-12-29 | 2019-05-24 | 百度在线网络技术(北京)有限公司 | Control method, device, user equipment and the storage medium of virtual speech assistant |
US11211055B2 (en) * | 2019-01-14 | 2021-12-28 | Microsoft Technology Licensing, Llc | Utilizing rule specificity in conversational AI |
CN109857787B (en) * | 2019-01-18 | 2022-04-15 | 维沃移动通信有限公司 | Display method and terminal |
JP7331395B2 (en) * | 2019-03-20 | 2023-08-23 | 富士フイルムビジネスイノベーション株式会社 | Process extractor and program |
US11204994B2 (en) | 2019-05-09 | 2021-12-21 | International Business Machines Corporation | Injection attack identification and mitigation |
US11311958B1 (en) * | 2019-05-13 | 2022-04-26 | Airgas, Inc. | Digital welding and cutting efficiency analysis, process evaluation and response feedback system for process optimization |
US11397858B2 (en) * | 2019-08-15 | 2022-07-26 | Kyndryl, Inc. | Utilizing widget content by virtual agent to initiate conversation |
US11150923B2 (en) * | 2019-09-16 | 2021-10-19 | Samsung Electronics Co., Ltd. | Electronic apparatus and method for providing manual thereof |
US11023220B2 (en) | 2019-09-26 | 2021-06-01 | Dell Products L.P. | Firmware update with integrated smart sequence and action engine |
JP2021117729A (en) * | 2020-01-27 | 2021-08-10 | 富士フイルムビジネスイノベーション株式会社 | Notification device and program |
CN111461901B (en) * | 2020-03-31 | 2023-05-12 | 德联易控科技(北京)有限公司 | Method and device for outputting vehicle insurance claim information |
US11789759B2 (en) * | 2020-08-12 | 2023-10-17 | Baton Simulations | Method, system and apparatus assisting a user of a virtual environment |
US11308287B1 (en) * | 2020-10-01 | 2022-04-19 | International Business Machines Corporation | Background conversation analysis for providing a real-time feedback |
CN115203172B (en) * | 2022-06-30 | 2023-11-07 | 北京亚控科技发展有限公司 | Model construction and model data subscription method and device, electronic equipment and medium |
Family Cites Families (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4964077A (en) * | 1987-10-06 | 1990-10-16 | International Business Machines Corporation | Method for automatically adjusting help information displayed in an online interactive system |
US4905163A (en) * | 1988-10-03 | 1990-02-27 | Minnesota Mining & Manufacturing Company | Intelligent optical navigator dynamic information presentation and navigation system |
US5115501A (en) * | 1988-11-04 | 1992-05-19 | International Business Machines Corporation | Procedure for automatically customizing the user interface of application programs |
US5058008A (en) * | 1989-10-03 | 1991-10-15 | Pitney Bowes Inc. | Mail system with personalized training for users |
US5103798A (en) * | 1989-10-16 | 1992-04-14 | Mcgraw Michael F | Archery arm guard |
US5239617A (en) * | 1990-01-05 | 1993-08-24 | International Business Machines Corporation | Method and apparatus providing an intelligent help explanation paradigm paralleling computer user activity |
JP3268529B2 (en) * | 1990-03-14 | 2002-03-25 | 株式会社日立製作所 | Knowledge database processing system and expert system |
US5311422A (en) * | 1990-06-28 | 1994-05-10 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | General purpose architecture for intelligent computer-aided training |
US5103498A (en) * | 1990-08-02 | 1992-04-07 | Tandy Corporation | Intelligent help system |
US5535321A (en) * | 1991-02-14 | 1996-07-09 | International Business Machines Corporation | Method and apparatus for variable complexity user interface in a data processing system |
US5901246A (en) * | 1995-06-06 | 1999-05-04 | Hoffberg; Steven M. | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US5903454A (en) * | 1991-12-23 | 1999-05-11 | Hoffberg; Linda Irene | Human-factored interface corporating adaptive pattern recognition based controller apparatus |
US5377103A (en) * | 1992-05-15 | 1994-12-27 | International Business Machines Corporation | Constrained natural language interface for a computer that employs a browse function |
US5369575A (en) * | 1992-05-15 | 1994-11-29 | International Business Machines Corporation | Constrained natural language interface for a computer system |
US5477447A (en) * | 1992-05-27 | 1995-12-19 | Apple Computer, Incorporated | Method and apparatus for providing computer-implemented assistance |
US5390281A (en) * | 1992-05-27 | 1995-02-14 | Apple Computer, Inc. | Method and apparatus for deducing user intent and providing computer implemented services |
US5432940A (en) * | 1992-11-02 | 1995-07-11 | Borland International, Inc. | System and methods for improved computer-based training |
US5983179A (en) * | 1992-11-13 | 1999-11-09 | Dragon Systems, Inc. | Speech recognition system which turns its voice response on for confirmation when it has been turned off without confirmation |
US5546502A (en) * | 1993-03-19 | 1996-08-13 | Ricoh Company, Ltd. | Automatic invocation of computational resources without user intervention |
US5701399A (en) * | 1993-06-09 | 1997-12-23 | Inference Corporation | Integration of case-based search engine into help database |
US5799292A (en) * | 1994-04-29 | 1998-08-25 | International Business Machines Corporation | Adaptive hypermedia presentation method and system |
US5644686A (en) * | 1994-04-29 | 1997-07-01 | International Business Machines Corporation | Expert system and method employing hierarchical knowledge base, and interactive multimedia/hypermedia applications |
US5597312A (en) * | 1994-05-04 | 1997-01-28 | U S West Technologies, Inc. | Intelligent tutoring method and system |
US5715374A (en) * | 1994-06-29 | 1998-02-03 | Microsoft Corporation | Method and system for case-based reasoning utilizing a belief network |
US5682469A (en) * | 1994-07-08 | 1997-10-28 | Microsoft Corporation | Software platform having a real world interface with animated characters |
JP3488525B2 (en) * | 1994-12-13 | 2004-01-19 | 富士通株式会社 | Help screen display method and help screen display device |
US5694559A (en) * | 1995-03-07 | 1997-12-02 | Microsoft Corporation | On-line help method and system utilizing free text query |
US5778402A (en) * | 1995-06-07 | 1998-07-07 | Microsoft Corporation | Method and system for auto-formatting a document using an event-based rule engine to format a document as the user types |
US5696964A (en) * | 1996-04-16 | 1997-12-09 | Nec Research Institute, Inc. | Multimedia database retrieval system which maintains a posterior probability distribution that each item in the database is a target of a search |
US5828999A (en) * | 1996-05-06 | 1998-10-27 | Apple Computer, Inc. | Method and system for deriving a large-span semantic language model for large-vocabulary recognition systems |
US5809145A (en) * | 1996-06-28 | 1998-09-15 | Paradata Systems Inc. | System for distributing digital information |
US5689619A (en) * | 1996-08-09 | 1997-11-18 | The United States Of America As Represented By The Secretary Of The Army | Eyetracker control of heads-up displays |
US5864848A (en) * | 1997-01-31 | 1999-01-26 | Microsoft Corporation | Goal-driven information interpretation and extraction system |
-
1996
- 1996-07-19 US US08/684,003 patent/US6021403A/en not_active Expired - Lifetime
-
1997
- 1997-07-02 EP EP06076448A patent/EP1742174A2/en not_active Withdrawn
- 1997-07-02 CN CNB97196520XA patent/CN1280715C/en not_active Expired - Lifetime
- 1997-07-02 WO PCT/US1997/010856 patent/WO1998003907A2/en active IP Right Grant
- 1997-07-02 DE DE69736552T patent/DE69736552T2/en not_active Expired - Lifetime
- 1997-07-02 EP EP97941319A patent/EP0912932B1/en not_active Expired - Lifetime
- 1997-07-02 CN CNA2004100749685A patent/CN1632749A/en active Pending
- 1997-07-02 JP JP50692898A patent/JP4676037B2/en not_active Expired - Lifetime
- 1997-07-16 CA CA002210601A patent/CA2210601C/en not_active Expired - Lifetime
-
1998
- 1998-11-20 US US09/197,160 patent/US6233570B1/en not_active Expired - Lifetime
- 1998-11-20 US US09/197,159 patent/US6260035B1/en not_active Expired - Lifetime
- 1998-11-20 US US09/197,158 patent/US6262730B1/en not_active Expired - Lifetime
-
2007
- 2007-09-25 JP JP2007248286A patent/JP4750771B2/en not_active Expired - Lifetime
-
2009
- 2009-04-13 JP JP2009097499A patent/JP2009157951A/en active Pending
Also Published As
Publication number | Publication date |
---|---|
DE69736552T2 (en) | 2007-05-10 |
JP4750771B2 (en) | 2011-08-17 |
US6021403A (en) | 2000-02-01 |
US6262730B1 (en) | 2001-07-17 |
CN1280715C (en) | 2006-10-18 |
US6233570B1 (en) | 2001-05-15 |
JP2009157951A (en) | 2009-07-16 |
DE69736552D1 (en) | 2006-10-05 |
JP4676037B2 (en) | 2011-04-27 |
EP0912932B1 (en) | 2006-08-23 |
EP0912932A2 (en) | 1999-05-06 |
CN1632749A (en) | 2005-06-29 |
WO1998003907A3 (en) | 1998-09-11 |
EP1742174A2 (en) | 2007-01-10 |
JP2008084321A (en) | 2008-04-10 |
CA2210601A1 (en) | 1998-01-19 |
WO1998003907A2 (en) | 1998-01-29 |
US6260035B1 (en) | 2001-07-10 |
JP2001510599A (en) | 2001-07-31 |
CN1231742A (en) | 1999-10-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2210601C (en) | Intelligent user assistance facility | |
WO1998003907A9 (en) | Intelligent user assistance facility | |
Gerlach et al. | Understanding human-computer interaction for information systems design | |
Benyon et al. | Applying user modeling to human-computer interaction design | |
Benyon et al. | Adaptive systems: from intelligent tutoring to autonomous agents | |
US5241621A (en) | Management issue recognition and resolution knowledge processor | |
US4972328A (en) | Interactive knowledge base end user interface driven maintenance and acquisition system | |
US20070003914A1 (en) | Consultative system | |
Farooq et al. | A survey of formal tools and models for developing user interfaces | |
Ross | Intelligent user interfaces: Survey and research directions | |
JPH05265680A (en) | Apparatus and method for graphical interface control | |
Shen et al. | A scenario-driven decision support system for serious crime investigation | |
Pracht | Model visualization: Graphical support for DSS problem structuring and knowledge organization | |
Purchase et al. | An empirical study of on-line help design: features and principles | |
Crowley et al. | SlideTutor: A model-tracing Intelligent Tutoring System for teaching microscopic diagnosis | |
Malinowski et al. | A taxonomy of adaptive user interfaces | |
Forde et al. | Knowledge-based control for finite element analysis | |
Kitajima et al. | LICAI+: A comprehension-based model of the recall of action sequences | |
Hoschka | Methodological and Tool Projects | |
Masarakal | Improving expertise-sensitive help systems | |
Chu et al. | Robust interactive decision-analysis (RID): concepts, methodology, and system principles | |
Halbrugge | Predicting User Performance and Errors | |
Kitajima et al. | Measuring the gulf of evaluation in display-based HCI | |
Yoder | The role of human-computer interaction in medical information systems: Principles and implementation of MEDIGATE | |
Harrington | Utilizing bayesian techniques for user interface intelligence |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
EEER | Examination request | ||
MKEX | Expiry |
Effective date: 20170717 |