CN1678994A - System and method for providing access to user interface information - Google Patents

System and method for providing access to user interface information Download PDF

Info

Publication number
CN1678994A
CN1678994A CNA038209381A CN03820938A CN1678994A CN 1678994 A CN1678994 A CN 1678994A CN A038209381 A CNA038209381 A CN A038209381A CN 03820938 A CN03820938 A CN 03820938A CN 1678994 A CN1678994 A CN 1678994A
Authority
CN
China
Prior art keywords
user interface
client computer
information
accessibility
interface
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CNA038209381A
Other languages
Chinese (zh)
Other versions
CN1678994B (en
Inventor
R·辛克莱尔
P·M·瓦格纳
B·麦克科恩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Publication of CN1678994A publication Critical patent/CN1678994A/en
Application granted granted Critical
Publication of CN1678994B publication Critical patent/CN1678994B/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/541—Interprogram communication via adapters, e.g. between incompatible applications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/542—Event management; Broadcasting; Multicasting; Notifications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/133—Protocols for remote procedure calls [RPC]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99931—Database or file accessing
    • Y10S707/99933—Query processing, i.e. searching

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)
  • User Interface Of Digital Computer (AREA)
  • Computer And Data Communications (AREA)

Abstract

A method and system providing a client with user interface information are described. An accessibility system for providing user interface information to a client. The accessibility system comprises an accessibility system core including user interface automation services and APIs. The user interface automation tools filter information based on whether the user interface information is interesting to the client. The accessibility system additionally comprises a client side interface including a logical tree for revealing user interface information that is interesting to the client and for hiding user interface information that is not interesting to the client. The accessibility system also comprises a server side interface for facilitating information transfer from a server side regardless of the server side technology.

Description

Be used to provide the system and method for calling party interface information
Technical field
The present invention relates to ancillary technique, automatic test product and these products and the mutual field of user interface information.
Background technology
The having of ancillary technique (AT) product help study, communication and visit comprise by computer software and the field of the information that shows in the computer user that wants help.The information at these product needed relevant calculation machine interfaces.Similarly, existing automatic test product and user interface instructs utility routine (commandingutilities) also need the information of relevant user interface.At present, these products also lack enough user interfaces (UI) information source.This three series products (client computer) need have from the required support in other places so that they can: (1) collects the information of relevant application program user interface; Which kind of technology what (2) no matter be used to set up UI is, can both find able to programmely and inquire the UI element; (3) produce the input of keyboard and positioning equipment; And (4) understand current available be any class behavior or function.Currently can provide all these functions to the AT product without any a kind of monotechnics.In addition, current AT product also not all is applicable to all Graphic Operating Systems (OS) technology, and they all lack the ability of filtering and coordinate redundancy or error notification with concentrated form.Therefore another shortcoming is that current robotization and accessibility (accessibility) lower floor facility can not be expanded, and needs other change of OS level new functional to add.
In addition, at present in order to collect the information of relevant application program user interface, the AT product must be write the application-specific code and obtain information for the user.The process of writing this application-specific code is consuming time, and needs continual maintenance.Current robotization lower floor facility also lacks the ability of filtering and coordinate redundancy or error notification with consistent form.Thereby, need the independent filtering information of event consumer.
Current system can notify the AT product with three magnitude request events: all things on (1) desktop; (2) in the specific process (for example opening word processor); Perhaps in the thread in (3) specific process (a plurality of objects of in process, working).Current, when client computer received an incident, its reception was used for the certain window of processing events generation and the window of other bit information where the indication incident takes place.Client computer can be intersected process transfer, to obtain the UI object of relevant this incident mutually.This object has been arranged, and client computer also can be carried out other intersection process transfer, with the information of relevant this object of request.If client computer needs five information, then client computer just must be carried out the intersection process transfer five times.The process transfer that intersects is extremely slow, and it is very high therefore to use accessibility lower floor facility to collect the performance cost of UI information.This type of known sight as shown in Figure 8.Server application 12 firing events 6.Kernel 14 determines to notify which client computer, and sends event notice 18 to relevant client computer 10.Client computer 10 is crossed over processing border 2 and is made request 16 to server application 12, asks the object of relevant event notice 18.Server application 12 returns object 20, and client computer 10 can begin to send the information request 16 to the UI control of relevant firing event then.Server application 12 is crossed over processing border 2 and is returned the information of being requested 20 to client computer 10.
Another current selection is to allow client code to be written in the process as dynamic link library (.DLL).This selection exists some shortcomings.At first, it needs coherent system that client code is written into process.Secondly, it also exists safety problem, in case because client code is written into program process, just be difficult to limit the information of its collection.Once more, be as a kind of effective technology to client computer, it must be loaded into each process in the system.Under the optimal cases, only should be being subjected to trusting clients to be written into the process of Another Application program.
In addition, need a kind of system, it can give client computer and specify it to want the ability of the event notice that receives.In known system, client computer may need to do numerous intersection process transfers, and whether analytical information is interested in the incident to determine it then.This just needs a kind of mechanism, and it can carry out this event filtering in more executable mode, and it can be convenient to upgrade to support new system or application event.In addition, need only use the system of trusted components so that alleviate potential safety hazard.
At present, when the information of the relevant user interface of search, need the AT product visit specific UI framework intrinsic tree.Therefore, need a plurality of trees to transmit the user interface information of a plurality of UI frameworks.These discrepant trees may comprise that the user loses interest in or invisible information, implicit such as the visible UI control of the management that the terminal user handled.Therefore, need only have those users the independent unified tree of node interested.
The solution that needs a kind of demand at AT product, automated test tool and instruction utility routine.This scheme must be applicable to all figure OS technology, but and should make the access that becomes of the UI of form of ownership and UI assembly.
Summary of the invention
The present invention relates to be used for providing the method and the computer applied algorithm of user interface information to client computer.In one aspect of the invention, provide the accessibility system that user interface information is provided to client computer that is applicable to.This accessibility system comprises kernel in the accessibility system with user interface robotization service, and whether this service is interested in the client computer based on user interface information comes filtering information.The accessibility system also comprises the client-side interface with logic tree, and this interface is used for disclosing client computer users interest interface information and hides the uninterested user interface information of client computer.The accessibility system also comprises the server side interface, and user interface information is convenient to transmit from the server side interface in this interface, and irrelevant with the user interface engine that is used to set up server side application.
In another aspect of this invention, provide to be applicable to the computer implemented method that user interface information is provided to client computer.This method comprises that the robotization service of employing accessibility system comes the monitoring user interface information, and transmits user interface information by the server side interface, and irrelevant with the server side technology.This method also comprises uses the logical elements that forms a client interface part to set to determine the interested particular user interface information of client computer.
Other advantage of the present invention and new feature will state in the following description, and partly will be obviously in instructions, or know by the practice of invention.
Description of drawings
The present invention carries out as detailed below with reference to accompanying drawing, wherein:
Fig. 1 is the block diagram that is suitable for the computingasystem environment used in realizing the present invention;
Fig. 2 is a block diagram mutual between accessibility system, client environment and the server environment;
Fig. 3 is the block diagram that the assembly of accessibility system kernel is shown;
Fig. 4 (A)-(D) shows based on the logic tree of intrinsic element and creates;
Fig. 5 is the process flow diagram that the procedural order that is used for setting up logic tree is shown;
Fig. 6 shows dialog box and the assembly thereof that forms logical elements;
Fig. 7 illustrates the process flow diagram that relates to the process that activates case mechanism of the present invention;
Fig. 8 shows the known system that is used for event notice.
Embodiment
The exemplary operation environment
Fig. 1 has set forth the example of the suitable computingasystem environment 100 that the present invention can realize therein.This computingasystem environment 100 only is an example of suitable computing environment, is not to be intended to propose usable range of the present invention or functional any restriction.Computing environment 100 should not be interpreted as that the arbitrary assembly shown in the exemplary operation environment 100 or its combination are had any dependence or any demand yet.
The present invention can computer executable instructions general context describe, such as the program module of carrying out by computing machine.Usually, program module comprises the routine carrying out specific tasks or realize concrete abstract data structure, program, object, assembly, data structure or the like.In addition, it will be appreciated by those skilled in the art that the present invention can put into practice in other computer system configurations, comprise handheld apparatus, multicomputer system, based on the system of microprocessor or the electrical equipment consumer goods able to programme, mini-computer, mainframe computer or the like.The present invention also can put into practice in the distributed computing environment that task is carried out by the teleprocessing equipment that connects through communication network.In distributed computing environment, program module can place the local and remote computer-readable storage medium that comprises memory storage device.
With reference to Fig. 1, realize that example system 100 of the present invention comprises the universal computing device of computing machine 110 forms, computing machine 110 comprises that processing unit 120, system storage 130 and handle comprise that the various system components of system storage are coupled to the system bus 121 of processing unit 120.
Computing machine 110 generally includes various computer-readable mediums.As example, and unrestricted, computer-readable medium can comprise computer-readable storage medium and communication media.System storage 130 comprises such as the easy mistake of ROM (read-only memory) (ROM) 131 and random-access memory (ram) 132 and/or the computer-readable medium of nonvolatile memory form.Including when helping computing machine 110 as starting the basic input/output (BIOS) 133 that transmits the basic routine of information at interelement is stored among the ROM131 usually.RAM132 comprises usually can 120 zero accesses of processed unit and/or current data and/or program module of operating.As example, and unrestricted, Fig. 1 shows operating system 134, application program 135, other program module 136 and routine data 137.
Computing machine 110 also can comprise other removable/not removable, easy mistake/nonvolatile computer storage media.As example, Fig. 1 illustrates and reads and write hard disk drive 141 not removable, non-volatile magnetic medium, read and write the disc driver 151 of removable, non-volatile disk 152, read and write removable, nonvolatile optical disk 156, such as the CD drive 155 of CD-ROM or other optical medium.Other removable/not removable, easy mistake/nonvolatile computer storage media that also is used in the example calculation environment includes, but not limited to as tape, flash card, digital versatile dish, digitizing video-tape, solid-state RAM, solid-state ROM or the like.Hard disk drive 141 is connected with system bus 121 by the not removable memory interface such as interface 140 usually, and disc driver 151 is connected with system bus 121 by the removable memory interface such as interface 150 usually with CD drive 155.
As mentioned above and disk drive as shown in Figure 1 and the computer-readable storage medium that is associated thereof the storage of computer-readable instruction, data structure, program module and other data is provided for computing machine 110.In Fig. 1, for example, hard disk drive 141 is illustrated as storage operating system 144, application program 145, other program module 146 and routine data 147.Notice that these assemblies can be identical or different with operating system 134, application program 135, other program module 136 and routine data 137.This different being tagged to that gives operating system 144, application program 145, other program module 146 and routine data 147 save your breath bright they are different copies.The user can and be often referred to locating device 161 into mouse, tracking ball or touch pad etc. to computing machine 110 input commands and information by input media such as keyboard 162.Other input media (not shown) can comprise microphone, joystick, game mat, satellite receiver, scanner or the like.These and other input equipment usually links to each other with processing unit 120 by the user's input interface 160 with the system bus coupling, but also can be connected with bus structure by other interface such as parallel port, game port or USB (universal serial bus) (USB).The display device of monitor 191 or other type also can link to each other with system bus 121 by the interface such as video interface 190.Except monitor, computing machine also can comprise other output device such as loudspeaker 197 and printer 196, and they link to each other by output peripheral interface 190.
Computing machine 110 among the present invention can use and one or more remote computer, moves in the networked environment that connects such as the logic of remote computer 180.Remote computer 180 can be a personal computer, and generally includes the relevant many or all components of above-mentioned and personal computer 110, although memory storage apparatus 181 only is shown in Fig. 1.Logic depicted in figure 1 connects and comprises Local Area Network 171 and wide area network (WAN) 173, but also can comprise other network.
When being used for the lan network environment, computing machine 110 is connected with LAN (Local Area Network) 171 by network interface or adapter 170.When being used for the WAN network environment, computing machine 110 generally includes modulator-demodular unit 172 or other is used at wide area network 173, such as setting up communicating devices in the Internet.The modulator-demodular unit 172 that can be built-in or external is connected by user's input interface 160 or other suitable mechanism with system bus 121.In networked environment, program module or its part relevant with computing machine 110 can be stored in the remote memory storage device.As example, and unrestricted, Fig. 1 shows the remote application 185 that resides in the remote computer 180.It is exemplary that network shown in should be appreciated that connects, and other technology that is used for establishing a communications link at intercomputer also can be used.
Although many other intrawares of not shown computing machine 110, it is well-known it will be appreciated by those skilled in the art that this assembly and interconnecting.Therefore, do not need to disclose other details of relevant calculation machine 110 internal frameworks in conjunction with the present invention.
The accessibility system architecture
As shown in Figure 2, accessibility system 200 is mutual with client environment 300 and server environment 400.The accessibility system can realize in the aforesaid computer environment 100 of reference Fig. 1.Accessibility system 200 comprises and is used for promoting the client-side accessibility interface 220 mutual with client computer 300, is convenient to server side accessibility interface 230 and the accessibility system kernel 201 mutual with server 400.Accessibility of the present invention system 200 provides and is used for new application programming interfaces (API), interface, and the metaphor (metaphor) of accessing user interface able to programme (UI).Accessibility system 200 makes application program can allow own and employed any assembly becomes addressable.
Preferably, client environment 300 comprises ancillary technique (AT) product or robotization UI testing tool.Server side 400 can be realized various different technologies, as shown in Figure 2.Server system 410 comprises adapter 412 and kernel 414 (can find) in first kind UI.Acting server assembly 422 and control 424 (finding among the second class UI of the Win32 UI that can be provided at the microsoft operation system product such as State of Washington Redmond Microsoft) are provided server system 420.Server system 430 comprises adapter 432 and inner OM 434 (can find) in the 3rd class UI.
As shown in Figure 3, included case mechanism 210 in accessibility system 200 depends on to be used for promoting client environment 300 and automatic client computer 202 of 400 mutual UI of server environment and UI automatic serving device 204.Automatic client computer 202 of UI and UI automatic serving device 204 are described in greater detail below with reference to case mechanism 210 of the present invention.The ability that accessibility of the present invention system 200 gives client computer (AT product) 300 is: (1) collects the information of relevant application program user interface; (2) no matter the UI element can both be found able to programmely and inquire to which kind of technology of being used to set up UI; (3) produce the input of keyboard and positioning equipment; And (4) understand current available be any class behavior or function.Accessibility system 200 makes application program can allow own and assembly becomes addressable.Structure shown in Fig. 2 and 3 is enabled five main aspects of accessibility system 200, comprising: (1) logic UI tree; (2) control model; (3) case mechanism; (4) input API (not being included in this document); And (5) attribute.
UI access logic tree 222
The important component of accessibility system 200 is logic trees 222, and its example is shown in Fig. 4 (D).Tree 222 can be included in the client-side accessibility interface 220.
Logic tree 222 is views after filtration of the understructure level of UI element, rather than must be by the lone tree of control or application developer realization.On the contrary, it has cut off some attributes interested and uninterested that clearly define, and whether these attributes have indicated a structuring element should represent in logic tree 222.Accessibility system kernel 201 uses these information producing UI logic tree 222 after filtration, and this tree can represent subsequently to AT product or test script.
Logic tree 222 is trees of element, and clauses and subclauses or packet configuration in each element representation control, the control can be dialog box, sash (pane) or framework (frame).The structure of logic tree 222 should be represented the UI (even in fact control is to use different understructures to realize) of application program by user's perception.It is stable that this tree is passed in time.As long as application program looks like equally the user,, represent that the logic tree of this application program must keep identical even the realization details on application program backstage changes.Former with realization because of structuring thereby exist intrinsic element, " ShDocView " form such as shell in the OS of the Microsoft product should not appear in this tree, because the user can not perceive them.
Logic tree 222 is independent trees that a plurality of segments are set up, thereby can integrated a plurality of different processes to make it be identical for client computer for it.Logic tree 222 enable block retrievals, and value that can the getattr tabulation.When the user has been intersected process transfer when asking numerical value usually, accessibility system 200 will obtain these numerical value by using logic tree 222.
Different with the result who only makes up with a step in the known system, logic tree 222 is constructed by the segment that is used to set up original tree.As shown in Figure 5, there are three main processes to set up logic tree 222.In process 72, the intrinsic element of lower floor of lower floor technology is located by accessibility system 200, and arrives at the proper tree shown in Fig. 4 (A).In process 74, the accessibility system intrinsic element of 200 combinations is to form the original tree shown in Fig. 4 (B).At last, in process 76, obtain logic tree 222 by the assembly of hiding in the original tree 20 of loseing interest in, shown in Fig. 4 (D).
Fig. 4 (A) shows two proper trees 10 and 14, and they make up from the intrinsic element of lower floor's technology, such as Win32 UI or any other available UI.Proper tree 10 comprises father node 11 and a plurality of descendants's node 12 that multiple relation is arranged between mutually.Similarly, proper tree 14 comprises the father node 15 with a plurality of child nodes 16.Child node 16 can be described as siblings each other.
Shown in Fig. 4 (B), proper tree 10 capable of being combined and 14 is to form original tree 20.Original tree 20 comprises father node 21, and it has two child nodes 22 and 30.Child node 22 has descendants's node 23-29, and child node 30 has descendants's node 31-33.The combination that this original tree 20 is proper tree 10 and 14, wherein the node of proper tree 10 forms node 22-29, and the node of proper tree 14 forms node 30-33.
By the method for expressing as Fig. 4 (C) and 4 (D), original tree 20 is convertible into logic tree 222.Be transformed into logic tree 222 from original tree 20, the developer can insert prompting on original tree.The developer can be the vertex ticks in the original tree 20 " hiding oneself " or " hiding own and child node " or " child node of concealed nodes " or the like.The developer also can move on to the next door to node, perhaps node is placed on before the child node.Can use the change on these " promptings " and the original tree 20 to form logic tree 222.For example, in Fig. 4 (C), the developer is labeled as the node 24-26 and 33 of original tree 20 uninterested, and is indicated as frame 40 and 41.Usually, the vertex ticks that comprises the element that can not be seen by the user for loseing interest in.The node relevant with seeing UI is considered to be interested usually, and will be included on the logic tree 222 to be used by AT client computer 300.Shown in Fig. 4 (D), on logic tree 222, do not comprise being marked as uninterested node.
Accessibility system 200 uses logic tree 222 to find out the information of relevant incident, system state, object's position and the information of relevant controlling.Known system does not have the ability of scrutinizing as yet on its tree.Logic tree 222 can be based on preferably the navigating of client computer 300, and no matter employed be which kind of server side application can both provide information.
In the operation, if client computer 300 need be obtained the information of relevant application program for the user, client computer 300 is sought the button that will press, and observes the literal on the button.Client computer 300 will be called API and " find out element ".API will return the numerical value of position in the logic tree 222 that points to client-side interface 220.By logic tree 222, accessibility system 200 all gives the abstract view of 300 1 UI of client computer, and irrelevant with employed application program.This abstract model comprises that list box, button or other UI assembly can expect mutually the same structure, attribute, incident and functional.
Logic tree 222 is single integrated trees of UI logical expressions, and it forms the form that only comprises client computer 300 interested elements.Therefore, show terminal user's model opposite with the structuring level and the conjecture that force the AT product to filter the UI element, logic tree 222 has represented the layer of structure that a kind of proper mapping shows the terminal user.This has greatly simplified the AT product and has described the task of UI to the user, and helps mutual between user and application program.
Because the underclad portion that this logic tree 222 is accessibility systems 200, other assembly of all of accessibility system 200 can design according to logic tree 222 work.For example, Fig. 6 has shown a kind of simple dialog frame 60 with utmost point simple structure.Yet when current available accessibility technical standpoint was observed, the structure of this dialog box 60 was shockingly complicated unexpectedly.It comprises 264 objects, and the AT product must filter to find which has meaning to the terminal user one by one to it.Accessibility system 200 has been arranged and to the support of logic UI tree 222, the developer who has this dialog box 60 can be provided with some attributes, with the following structure of showing as shown in Figure 6 to AT product 300.
As shown in Figure 6, for " Run " dialog box, the developer can be expressed as " name of key entry program, file, file or internet resource, and window can be opened it for you " of the window Figure 62 that rolls and 63 interested.The developer can be the combo box that comprises notepad (notepad), word (Word Word), calculator (counter) or the like and OK (being) button 65, cancel (denying) button 66 and browse (browsing) button indication interested also.This offers a kind of low-cost mechanism of developer, with its element of mark by different level and therefore produce the logical expressions of its application program UI by UI accessibility system 200.Attribute shown in each can by with logic tree 222 on each other node have a node of particular kind of relationship to represent.Provide instant benefit to test group and AT product or client computer 300 for logical expressions.
One group of API can make client computer 300 can obtain tree.Function comprises: the logical elements of (1) point-to-point; (2) logical elements of incident; (3) current is the logical elements of focus.As mentioned above, logical elements represent the UI assembly, also may be control, control a part or container or logical combination (for example dialog box, sash or framework).Control is very big according to its functional variation.Therefore, use functional that different interfaces represents to be associated with specific control types.These control special interfaces are derived from expression all are controlled all general basic interfaces of identical functions.This general basic interface comprises: (1) is used for the method for navigational logic tree 222; (2) be used for the universal method of getattr value; And (3) are with visiting the method that is supported to control special interface.In the navigation of logic tree 222, the application program UI of certain lower floor technology all provides its oneself technology for navigation.
Although logic tree 222 is the most interested for the user, primitive element tree 20 also provides some critical function.Logic tree 222 only comprises user-dependent element, primitive element tree 20 then comprise expression lower floor framework implementation structure such as 22 node.For example, for WIN32 UI segment, this tree will comprise the node of representing HWND.In some aspects, primitive element tree 20 is between logical elements tree 222 and the intrinsic element-tree of lower floor's framework oneself " intermediary agency ".Primitive element tree 20 is used as the lower floor of construction logic element-tree, and it is the at first place of insertion system of intrinsic element.
Also can use primitive element tree 20 to do debugging and test.The place of finding out or describing the particular problem node is useful.Function on the basic primitive element node comprises: the method for the primitive element tree that is used for navigating; Be used for jumping to the respective logic element method of (if existence); Attribute one " the HWND 0x483FE " of HWND node for example that comprises ' the debugging character string ' of this element; And other ' infrastructure behind the scenes ' method.These other methods enable to hit test and position; Incident; And represent attribute that framework can easily provide (for example, focusing, enable).
Primitive element tree 20 comprises expression from the various node 22-33 that describe the element of engine (rendering engine).Primitive element tree is used as describes engine with the own starting point of inserting accessibility system 200, and builds up integrated logic tree 222 from suitable such as the lightweight adapter object from the intrinsic element of the HWND of Win32.It also is used to handle transfaunation (hosting transition), and wherein a kind of technology is the host of another kind of technology.Because primitive element tree 20 is lower floors of construction logic tree 222, can use it check logic tree whether complete sum link to each other, and check unaccounted element.Also can use primitive element tree 20 for other class lower floor facility task: such as some fundamental element ID and the element property that provides some basic framework to provide are provided, such as focus on, enable and position.
Primitive element tree 20 is not the main source of the information of AT product or client computer 300, is not used to logical navigation, is not represented to the terminal user yet.It can be returned at certain time point in the future thereby primitive element tree 20 can not be used to catch element position in the tree.Logical elements tree 222 is carried out these all functions.
Primitive element tree 20 need not the knowledge of representative logical elements usually, just can describe primitive element (HWND, Elements) structure mechanically of technology from lower floor.Therefore can use it to seek the not primitive element of explanation on logic tree 222.Primitive element tree 20 is useful debugging and diagnostic tools, because the class ' stack dump ' (stack dump) that it allows to catch node location is described.In addition, known system based on the code specific criteria, and is difficult to their tree to realize with different technologies.This method is used a kind of general abstract ' primitive element ' type, and it can be described technology by (or representative) any lower floor and realize.
In order to obtain the primitive element tree, call the primitive element root and will obtain the desktop element, this is by guaranteeing that its father node is that empty (NULL) and all other nodes all obtain check with it as final ancestors.In order to obtain other element, call method will use effective screen coordinate to return element to obtain primitive element from specified point.Obtain after the primitive element tree, can check and check it by inspection element (father node, sister's node and child node).
In the operation, client computer 300 can be used such as the relation of: father node, back sister's node, preceding sister's node, first child node, last child node or the like the primitive element tree 20 of navigating.Client computer 300 can jump to respective logic element on the logic tree 222 from primitive element.
Case mechanism
When client computer wanted to keep the notice of the incident that obtains, client computer 300 can be by automatic client computer 202 registrations of UI as shown in Figure 3 to obtain this information.Client computer 300 is specified the attribute list of wanting object information, the information whereabouts that receives and wanting to fetch.Client requests mails to the automatic client computer 202 of UI.The automatic client computer 202 of UI can monitor any process on the desktop.UI automatic serving device 204 is followed the tracks of in that client computer 300 of listening to, and knows how to return the automatic client computer 202 of UI.The automatic client computer 202 of UI is to the interest of UI engine 206 notice client computer, so UI engine 206 knows when tell UI automatic serving device 204 incident.The UI engine needn't utilize client computer notice, but can select on the contrary always to UI automatic serving device 204 notification events, perhaps only notifies the UI automatic serving device when client computer when listening to any incident.If only want to open UI automatic serving device notice as client computer UI engine when listening to incident, notice is useful.The UI engine can be done like this, with avoid UI speed may descend and avoid being written into its may and unwanted code module.
UI engine 206 is informed the UI incident to UI automatic serving device 204 then.UI automatic serving device 204 returns the logical elements of being asked to client computer 300, and sends the information that comprises 300 request event attributes of client computer to client computer 300.Which information UI automatic serving device 204 judges within the scope of client requests, and if only if information just forms logical elements when appealing to.Its set of properties that will when processing events, use that forms that logical elements is included in that UI automatic serving device side looks ahead that client computer indicated.For example, UI automatic serving device 204 can be found logical elements for combo box.Scope is this combo box and sub-frame thereof.Client computer 300 can ask son/father/relatives' element in the registration phase range of definition.
After UI automatic serving device 204 determined whether information is being requested in the scope, it set up logical elements.The automatic client computer 202 of UI provides from UI automatic serving device 204 to client computer 300 by the dialogue with destination application and receives the information that is requested, and object is routed to the suitably service in space on the client computer 300.
When client computer 300 registrations will receive event notice, create UI automatic serving device 204.As example, the UI engine 206 operation Word of Microsoft word-processing applications.The client computer 300 registered attribute of name of doing change.The registration of client computer makes UI automatic serving device 204 be created.The registration of client computer also is notified to the UI automatic serving device to attribute of name to the 206 notice beginnings of UI engine.UI engine 206 does not obtain range information.UI engine 206 calls one of API for server side.The UI engine specifies (1) which attribute to change; (2) the new value of attribute; And (3) possible old value.UI automatic serving device 204 is created based on client computer 300 interested incidents, and therefore knows interested incident, attribute, client computer and scope, and 300 pairs of logical elements of having created are interested thereby it knows whether any client computer.If an above client computer 300 has been registered incident to specific UI automatic serving device 204, and similar events as that client computer 300 is registered has also been asked to obtain in large quantities attribute and has been returned logical elements, when UI automatic serving device 204 returns to client computer 300 to incident, each client computer 300 will obtain the association that is requested a large amount of attributes that returns with logical elements.
For each client computer 300 in reception, UI automatic serving device 204 is usually notified client computer 300 by the logic basis that is associated with incident to client transfers.UI automatic serving device 204 is only created a logical elements.This is the improvement to the prior art of their all object copies as Event origin of each client requests of needs.
If UI engine 206 does not utilize the notice of the automatic client computer of UI when the client computer registering events, whether UI engine 206 can have any accessibility client computer 300 receiving to 204 inquiries of UI automatic serving device, if and do not have client computer receiving, then could avoid establishment information and send it to the work of UI automatic serving device 204.For example, screen reader is a client computer 300, and specify information whereabouts that it wants, the focus that receives incident changes object and the tabulation of interested particular community.Then can notify UI engine 206, and it know should be to UI automatic serving device 204 transmission incidents.After detecting the focus change, UI engine 206 notice UI automatic serving devices 204.UI automatic serving device 204 converts well-known interface to, and incident and object are sent to the automatic client computer 202 of UI.The automatic client computer 202 of UI is routed to suitable space on the client computer 300 to object.
By removing the central repositories of incident in the kernel, just can be improved based on the said modules of known system.On the contrary, to know that 300 pairs of all client computer are obtained the information of the relevant environment of operation therein interested for UI automatic serving device 204.The removal of kernel thesaurus has also been created how point-to-point mutual because UI automatic serving device 204 has been finished the function of before carrying out in kernel.Accessibility of the present invention system 200 gives client computer 300 and specifies and want the ability of viewing content, thereby can use UI automatic serving device 204 to finish filtration at server side.
Fig. 7 is the process flow diagram that the process that relates to event registration and Notification Method is shown.In step 80, client computer 300 request events notice.In step 82, the automatic client computer 202 of UI sends request to UI automatic serving device 204.In step 84, the automatic client computer of UI notifies it to want the UI engine 206 that obtains notifying.In step 86, UI automatic serving device 204 is from UI engine 206 reception notifications.In step 88, UI automatic serving device 204 filters the information that has received.If it is irrelevant with the user to find to have received information in step 90, UI automatic serving device 204 is at step 92 removal information and wait order such as continuation grade.Perhaps, information-related in step 90 discovery, UI automatic serving device 204 is created logical elements and is sent it to the automatic client computer 202 of UI in step 94.In step 96, the automatic client computer 202 of UI will reception information be placed on the appropriate location on the client computer 300.
The case mechanism 210 of accessibility system 200 makes client computer 300 can register the attribute change of reception UI, the event notice of tree change, multi-media events and relevant information in the control structure.Do not have these abilities, client computer 300 has to select constantly UI elements all in the system to find whether have any information, structure or state to change.The case mechanism 210 of accessibility system 200 also make client computer 300 can the outer incident of receiving process, the request property set that will return with event notice, and register for the incident on a plurality of elements.
Case mechanism 210 represents: AT product or test application are used for the interface of registering events; The interface that is used to receive event notice that the AT product is realized on object; And control realizes the interface that it is used to notify the event engine of UI incident.Case mechanism 210 is used to make AT product and test application to receive to be independent of the UI engine in order to describing the incident of UI, and no matter which kind of lower floor's technology can both be followed the trail of the upper level applications window to make AT product and test application.
It is how mutual with accessibility system 200 that Fig. 2 shows client computer 300.Do not cross over the general management of process.On the contrary, each accessibility system users machine 300 or server 400 all have the environment of oneself.Client applications only receives the incident of the application program of supporting accessibility system 200.In order to begin in application program to use the accessibility system event, client computer 300 can be done one of following four things: (1) use " AddTopLevelWindow Listener " API find to occur on the table and handling this incident new UI, register other incident and by the incident of this device reception from any process; (2) use the interested UI in one of " Find " API location, and lock a specific UI; (3) use some other device to find interested UI, such as on screen, finding window processor or point, and use this processor or point to obtain the logical elements of guiding as the reception incident; Perhaps (4) are used " AddFocus ChangedListener " API to follow the trail of input focus and are registered UI and go up current incident as focus.
Parent window is that his father is the window of desktop.The use indication that term " is opened " shows new parent window to the user.The use indication parent window that term " is closed " disappears.
Accessibility system server side interface 230 comprises and is used for adding and two API of the incident removed to accessibility system 200 notice.When cancelling registration at the client computer registering events with to it, these API of the automatic client calls of UI.This makes UI engine 206 obtain relevant information of asking which accessibility incident in its background.
In the time can using, incident is associated with logical elements from application logic element-tree 222.When logical elements can not be used, on behalf of the well-known object of Event origin, incident be associated with human readable character string or other.Case mechanism 210 provides filtration based on the user preferences that provides during event registration.By using the filtration hobby of client computer on the server, before sending it to client computer, the quantity that case mechanism 210 calls by the minimizing striding course is improved the outer performance of process inherently with creating event related data and striding course.Case mechanism 210 is provided at the method for specifying during the event registration the attribute of the interested logical elements of incident.This has further reduced the quantity that striding course calls.Need not to change master operating system (OS) with regard to extensible event mechanism 210.Although can use code to come realization event mechanism 210, still can visit it by the COM interoperability without the application program of management through management.
AT client computer 300 processing events and two tasks carrying out are: (1) event registration; (2) processing events in recovery.To the requirement of arbitrary task through management with all be available in without management code.In one embodiment of this invention, client computer 300 can be transmitted the interface when registering events.Client computer 300 is transmitted the object of realizing well-known interface when registration, and UI engine 206 reclaims these interfaces with notification receiver.In logical elements tree 222, have in the situation of source object, reclaim all a reception sources element and an additional parameter at every turn.
Being can be used as by the attainable well-known interface of AT product is the method for returning incident.Following chapters and sections show the incident of some type, register the appearance that looks and how to receive incident.
1. parent window incident
The parent window incident comprise relevant menu and drop-down combo box incident or any be father's feature with the desktop.In one embodiment of this invention, use the AddTopLevelWindowListener method to receive the notice that parent window is opened and closed.Call AddTopLevelWindowListener and will obtain on front desktop, the notice that new parent window is opened or parent window is closed is being arranged.The RemoveTopLevelWindowListener method provides and stops receiving the notice that parent window is opened or closed on the desktop.This method uses recycle object to identify this recipient.Therefore, the object that is delivered in the RemoveTopLevelWindowListener method is identical with the object that passes to AddTopLevelWindowListener.For each new parent window of opening, call the OnTopLevelWindowOpened method once by accessibility system 200.Similarly, when parent window is closed, call the OnTopLevelWindowClosed method once by accessibility system 200.In order to receive these notices, client computer 300 is called the AddTopLevelWindowListener method.
2. focus incident
Client computer 300 usually needs to be used for following the tracks of the method for focus.It is difficult doing like this in Microsft Windows OS.For example, when menu is drop-down (for example File menu among the Word of Microsoft), along with the user moves down cursor one by one, the clauses and subclauses in the menu become focus.(for example the user presses esc key) do not have the focus incident to send when menu is closed.On the contrary, focusing changes interested client computer must receive numerous incidents, and points out that wherein which has represented the change of focus in logic.In one embodiment of this invention, can use the AddFocusChangedListener method to come to notify focus to change incident to the recipient.Client computer 300 can be specified the set of properties that will return.Appointment is the part of all event registration API with this feature of the attribute that logical elements is returned.Client computer 300 can be called the RemoveFocusChangedListener method and stop receiving the notice that focus changes.This method can use recycle object to identify the recipient, and the object that is delivered in this case in the RemoveFocusChangedListener method is identical with the object that passes to AddFocusChangedListener.The OnFocusChanged method is called by accessibility system 200 when focus changes.The FocusEventArgs parameter of using in the OnFocusChanged method represents the information that relevant focus changes.If it is can obtain the information of the logical elements of relevant last focus, then available in the incident args parameter that this element will use in the OnFocusChanged element.It is up to added focus clearly on an element UI element focus to be arranged just that situation is also arranged.
3. attribute changes incident
When the attribute of logical elements changes, can cause attribute and change incident.In one embodiment of this invention, client computer 300 is called the AddPropertyChangedListener method, to receive the notice that attribute changes.When the logical elements tree goes up the property value change of the logical elements of appointment in AddPropertyChangedListener, call the OnPropertyChanged method by accessibility system 200.Which element what range parameter indication incident should cause is.For example, transmitting the root logical elements of window and the request restriction of descendants's element changed the attribute of this window reclaims.If, then omit scope, and send any change that takes place on the table to specified attribute to all elements setting range parameter on the tree.Client computer 300 can repeatedly be called AddPropertyChangeListener with different attribute group and/or different recycle object.Indicate by the notice that accessibility system 200 provides: the attribute that has changed; New property value; And old property value (as obtaining).
Client computer 300 can be called the RemovePropertyChangeListener method to stop receiving the notice that attribute is changed.But this method usable range element and recycle object are identifying this recipient, and the object that is transmitted in this case must be identical with the object that passes to AddPropertyChangeListener.
4. from the incident of control model
The incident that is caused by control model must be extendible, and therefore identifies these incidents by GUID.When registration, can accept any GUID value.For any new control model, the incident that it writes down must be unique GUID.May need to change the AT product to receive new control model incident.The recipient also must be able to determine the scope of these incidents.For example, oriented test may be wanted the control of these time limit in application-specific or application program.What control model defines is the source, and event consumer will it may be noted that part of file and how use source element and event variable object so that know.
The AddEventListener method will make client computer 300 can receive the incident that is used to control.Can the usable range parameter be which element with what indicate firing event.For example, transmit the root logical elements of window and ask descendants's element will be time limit at this window.If want to set all elements, then send incidents all on the desktop.Can use the RemoveEventListener method to stop receiving the notice that attribute is changed.But this method usable range element recycle object and event ID are identifying this recipient, and the object that is transmitted in this case must be identical with the object that passes to AddEventListener.
When application program is modulated when having used the AddEventListener method, when the control particular event is initiated, call the OnEvent method by accessibility system 200, and the source of incident is that the logical elements tree goes up the logical elements of appointment among the AddEventListener.When causing the incident of control, usually can obtain event-specific information.
Can call the RemoveAllListener method to stop any incident.This is the fast method of putting in order before client applications is closed.When end application, use the removal method alternatively.
5. logical organization changes incident
When the logical elements tree construction changes, caused logical organization and changed incident.Can realize the notice of AddLogicalStructureChangedListener method with structural change on the receive logic element-tree.When adding, removing logical elements or it was lost efficacy, call the method for specifying on the recycle object.Range parameter is constraint element as described above.Can use the RemoveLogicalStructureChangedListener method to stop receiving the incident that the logical elements tree is changed.This method can use recycle object and scope element identifying this recipient, and the object that is transmitted in this case must be identical with the object that passes to AddEventListener.
When adding daughter element, and father's element can call the OnChildAdded method by accessibility system 200 when being logical elements on the logical elements tree 222 of appointment in AddLogicalStructureChangedListener.When removing daughter element, and during the logical elements on its old father's element logical elements tree that is appointment in AddLogicalStructureChangedListener, can call the OnChildRemoved method by accessibility system 200.
When in stage blink, adding a large amount of daughter element, (for example surpass 20 daughter elements), by accessibility system call OnChildrenBulkAdded method.When in stage blink, removing a large amount of daughter element, by accessibility system call OnChildrenBulkRemoved method.When in stage blink, adding simultaneously and removing a large amount of daughter element, (for example surpass 20 daughter elements), by accessibility system call OnChildrenInvalidated method.
6. multi-media events
Another kind of incident is a multi-media events.Multimedia can comprise sound, video and animation.Method can be supported multi-media events and to the client computer notification action, comprise " termination ", " time-out ", " F.F. ", " rewinding " and " quiet ".Can realize being similar to the aforesaid method that is used for adding and removing the multimedia recipient.
7. simple sound incident
The simple sound incident can with the multi-media events separate processes.The representative of simple sound incident is used for transmitting to the user simple, the of short duration sound of some incident that is different from sound itself.The simple sound incident can comprise: the sound of playing when new mail arrives at; The sound that when the Notebook Battery energy is low, produces; The sound of playing when perhaps mailbox shows with the IconExclamation type.Can call the AddSoundListener method receiving the notice of in progress simple sound, and can realize that also the RemoveSoundListener method is to stop receiving the notice of simple sound incident.
When playing simple sound, call the OnSound method by accessibility system 200.In order to receive this notice, the application call AddSoundListener that is receiving.The OnSound method obtains following information: the name of sound; The source of sound; And indication sound is to the Alert Level value of user's importance.Possible Alert Level comprises: ' the unknown ', the unknown of indication importance; ' report ', indication information are represented; ' warning ', indication warning situation; ' inquiry ', indication needs the user to respond; ' admiration ', to indicate this incident be not crucial but may be important; And ' key ', the generation of indication critical event.
8. soft focus incident
The soft focus incident occurs on the table but is retained in the background.Some example of soft focus incident is: at the balloon-like help window of notification area indication " can obtain new renewal "; Go on the taskbar focus background applications flash icon; And the printer icon that when printing beginning and finishing, occurs or disappear at notification frame.These incidents seem and some overlapping of other kind of event (multimedia can relate to animation event as soft focus).Yet, come incident is classified based on the method that incident will send user's content to rather than transmit incident.
Can realize that the AddSoftFocusListener method is to receive the notice do not become input focus and to attempt obtaining the incident that the user notes.The RemoveSoftFocusListener method stops notice.This method can use recycle object to identify this recipient, and the object that is therefore transmitted should with pass to the identical of AddSoftFocusListener.
When the soft focus incident takes place, can call the OnSoftFocus method by accessibility system 200.In order to receive this notice, call AddSoftFocusListener in the application program or the client computer 300 that receive.Can use source element (as obtaining) to obtain the more information of relevant incident.The exemplary source element can be the logic root element by the balloon-like help window of one of notification application use in system's frame.The OnSoftFocus method is returned: event name; Event origin; And indication is to the alert level of user's importance.
When client computer 300 uses AddTopLevelWindowListener API to receive incident from particular procedure, the action of following caption client computer 300 and accessibility system 200.
Client computer Accessibility system and target UI window
Call AddTopLevelWindowListener Accessibility system users machine begins a thread and occurs to observe at the new UI of the upper level applications window of creating and eliminate, and accessibility system users machine calls the OnTopLevelWindow Opened method of this client computer
Occur in other incident of target UI window with registration from other API of OnTopLevelWindowOpened method call Accessibility system users machine transmits incident Id so that notification event optionally to target UI window.Accessibility system users machine sends incident Id and filtering information to the accessibility system server, so its further filter event.Target UI window uses accessibility system server API to notify interested thing to accessibility system server side
Part.
Handle the incident in the recycle object The accessibility system server is passed incident back accessibility system users machine.Accessibility system users machine reclaims the object that client applications provides registration
Call RemoveListener to stop the reception incident Accessibility system users machine transmits no longer interested incident to accessibility system server and target UI window.Target app stops transmitting incident to the accessibility system server
Event notice
Use the accessibility system event of events corresponding Notification Method by server 400 or the UI of lower floor engine to enumerate more than supporting.UI automatic serving device API comprise this server or the UI of lower floor engine may call with finish method.For example, the NotifyPropertyChanged method of a server calls is arranged when the particular community on the logical elements changes, to notify.When UI changes, produce suitable parameter and call these Notification Methods and depend on server 400 or the UI of lower floor engine.
The server method
By the automatic client calls AdviseEventAdded method of UI and AdviseEventRemoved method so that when client computer 300 request events announcement server 400.This makes the server 400 can be not to accessibility system 200 communication eventses when the neither one incident appeals to.
Control model
The unique method that the accessibility system provides specific UI element or control to support in order to classify and to represent function.With opposite in specific control types (for example button, edit box or list box) in this area function association, the accessibility model definition general controls set of patterns, the one side of each mode-definition UI behavior wherein.Because these patterns are independently of one another, they can be made up to describe the function complete or collected works that specific UI element is supported.
For example, with the class name according to element, such as button, come descriptive element opposite, accessibility system 200 is described as it to support to call control model.Structure, attribute, incident and method that the control model definition is supported by element.Therefore, these patterns not only make the behavior of client's function inquiry control, also make it handle control by using to the interface of AD HOC design able to programmely.For example, the SelectionContainer pattern provides all methods in order to inquiry selected entry, selection particular items or cancellation whether its selection or definite this control to be supported single or single preference pattern.
Control model by 300 definition of accessibility system comprises at present: (1) selects container; (2) level; (3) can call; (4) simple mesh; (5) text; (6) value; (7) request object; (8) can roll; (9) can sort; (10) towing; And (11) other container.
This technology make the control developer can realize new control types and still have simultaneously to AT product and test script represent its behavior through clear and definite define method.If introduced new behavior type, the new control model of definable is to express required function.
Note technical products and test script be can write now and every kind of pattern rather than every kind of UI Control work how used to understand.Because control model is than control class much less, this technology makes that essential code is minimum.This method is also encouraged a kind of more flexible framework (as long as they support the known control pattern) that can inquire and handle new control effectively.
Following form provides some example of general controls and they pattern with supporting.
Control Relevant control model
Button Can call
Check box, radio button Value
List box Select container, can roll
Combo box Select container, can roll, be worth
Tree view Select container, can roll, level
The tabulation diagrammatic sketch Select container, can roll, can sort
Text box, editor Value, text, can roll
Can use more specific interfaces to represent the function that is associated with the general controls pattern.The example of these patterns comprises: (1) selectes Manage Containers; (2) grid layout container; (3) comprise the UI value of value; (4) icon of representative object (file, form or the like); And (5) UI element that can call.Generally speaking, these patterns not strictness are bound to specific control, and different control can realize identical pattern.For example, list box, combo box and tree-shaped diagrammatic sketch are all realized ' selection Manage Containers ' pattern.If some control can realize a plurality of patterns in the time of suitably: select grid will realize ' grid layout container ' pattern and ' selection Manage Containers ' pattern.
Do not resemble independent ' role ' (task) attribute in the previous application program.On the contrary, two kinds of different mechanism have been used.Control model is determined the available function of control, and the human-readable localizable attribute control class name that provides the user to understand, such as ' button ', ' list box ' or the like.
Attribute
Accessibility system 200 will be to general GetProperty method feature in addition.Attribute is preferably to be represented by GUID, and it has and is used for converting to or change from not localizable memory form (useful to script and configuration file) and can obtains the practical approach that localization is described.Two key advantage that general GetProperty method is compared with single method be (a) it allow not change the interface and just can Add New Attribute as time goes by; And (b) it allows to obtain the technology that can not realize-obtain such as the piece attribute of array driving when using single method.Each attribute must have the intention of clear definition.Must clear defined attribute whether be intended to as the mankind or machine consumption or whether attribute localizable, or the like.
The present invention is described at specific embodiment, and these embodiment are illustrative and non-limiting in all respects.For a person skilled in the art, the optional embodiment that does not deviate from its scope that the present invention comprises will be conspicuous.
Can be seen from the foregoing, the present invention can adapt to well reaching above-described all results and object, and explains for this system and method and to show and other intrinsic advantage.Be appreciated that some feature and sub-portfolio are practical, and can not adopt with reference to further feature and sub-portfolio.This is expected, and within claims scope.

Claims (18)

1. accessibility system that is used for providing user interface information to client computer, it is characterized in that described accessibility system comprises: have and be used for based on described client computer whether to kernel in the accessibility system of the described user interface information user interface robotization service that comes filtering information interested; Client-side interface with logic tree, this logic tree are used for representing described client computer institute's users interest interface information and hide the uninterested user interface information of described client computer; And the server side interface, this interface is convenient to transmit from the information of this application program, and irrelevant with the user interface engine that is used to set up server side application.
2. accessibility as claimed in claim 1 system, it is characterized in that, described user interface robotization service forms a case mechanism, described case mechanism comprises by the automatic client computer of the user interface of described client-side interface operation, and the user interface automated server by described server side interface operation.
3. accessibility as claimed in claim 1 system is characterized in that, the automatic client actions of described user interface so that receive the event registration request from described client computer, and notify described event registration request to described user interface automated server.
4. accessibility as claimed in claim 1 system is characterized in that described user interface automated server is sought event information from the user interface engine of server side, and filters described event information to remove the information outside the event registration request scope.
5. accessibility as claimed in claim 1 system is characterized in that described logic tree comprises a plurality of elements, wherein in clauses and subclauses in each element representation control, the control or the packet configuration.
6. accessibility as claimed in claim 1 system is characterized in that, lower floor of described lower floor application program technology determine to be used for the to navigate proper technology of described logic tree.
7. accessibility as claimed in claim 1 system is characterized in that described logic tree is unified the user interface of different application.
8. accessibility as claimed in claim 1 system is characterized in that, also comprises the interface that is used for representing the relevant controlling mode capabilities.
9. accessibility as claimed in claim 8 system is characterized in that the described accessibility API of system returns the logic tree position to provide the information of relevant application program to described client computer.
10. computer implemented method that is used for providing user interface information to client computer, described method comprises step: with accessibility system automation services monitoring user interface information; Transmit user interface information through the server side interface, and irrelevant with the employed technology of server side; And use the logical elements that forms a described client-side interface part to set to determine the interested particular user interface information of described client computer.
11. method as claimed in claim 10 is characterized in that, also comprises by the automatic client computer of the user interface of described client-side interface operation, and the user interface automated server by described server side interface operation.
12. method as claimed in claim 11 is characterized in that, also is included in the event registration request that receives on the automatic client computer of described user interface from described client computer, and notifies described event registration request to described user interface automated server.
13. method as claimed in claim 12 is characterized in that, also comprises from the user interface engine of server side seeking event information, and filters described event information to remove the information outside the event registration request scope.
14. method as claimed in claim 10 is characterized in that, also comprises in clauses and subclauses in usually representing to control, control with each yuan in the described logic tree or the packet configuration.
15. method as claimed in claim 10 is characterized in that, also comprises the proper technology of the described logic tree of determining to be used for to navigate.
16. method as claimed in claim 10 is characterized in that, also is included in the user interface of unified different application in the described logic tree.
17. method as claimed in claim 10 is characterized in that, also comprising provides the interface that is used for representing the relevant controlling mode capabilities.
18. method as claimed in claim 10 is characterized in that, also comprises using the accessibility system to return the logic tree position to provide the information of relevant application program to described client computer.
CN038209381A 2002-09-30 2003-05-16 System and method for providing access to user interface information Expired - Fee Related CN1678994B (en)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US41533902P 2002-09-30 2002-09-30
US60/415,339 2002-09-30
US10/367,226 US7434167B2 (en) 2002-09-30 2003-02-14 Accessibility system and method
US10/367,226 2003-02-14
PCT/US2003/015456 WO2004031947A1 (en) 2002-09-30 2003-05-16 System and method for providing access to user interface information

Publications (2)

Publication Number Publication Date
CN1678994A true CN1678994A (en) 2005-10-05
CN1678994B CN1678994B (en) 2010-05-26

Family

ID=32033352

Family Applications (1)

Application Number Title Priority Date Filing Date
CN038209381A Expired - Fee Related CN1678994B (en) 2002-09-30 2003-05-16 System and method for providing access to user interface information

Country Status (7)

Country Link
US (1) US7434167B2 (en)
EP (1) EP1546871A1 (en)
JP (2) JP4942932B2 (en)
KR (1) KR100965708B1 (en)
CN (1) CN1678994B (en)
AU (1) AU2003299163A1 (en)
WO (1) WO2004031947A1 (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101835101A (en) * 2009-02-27 2010-09-15 捷讯研究有限公司 The mobile radio communications system and the correlation technique of appliance icon notification indicia framing are provided
CN102804161A (en) * 2009-04-03 2012-11-28 社会传播公司 Application sharing
US9411489B2 (en) 2007-10-24 2016-08-09 Sococo, Inc. Interfacing with a spatial virtual communication environment
US9411490B2 (en) 2007-10-24 2016-08-09 Sococo, Inc. Shared virtual area communication environment based apparatus and methods
US9755966B2 (en) 2007-10-24 2017-09-05 Sococo, Inc. Routing virtual area based communications
US9762641B2 (en) 2007-10-24 2017-09-12 Sococo, Inc. Automated real-time data stream switching in a shared virtual area communication environment
US9853922B2 (en) 2012-02-24 2017-12-26 Sococo, Inc. Virtual area communications
US10003624B2 (en) 2009-01-15 2018-06-19 Sococo, Inc. Realtime communications and network browsing client
US10158689B2 (en) 2007-10-24 2018-12-18 Sococo, Inc. Realtime kernel
CN109542644A (en) * 2018-11-19 2019-03-29 北京小米移动软件有限公司 Application programming interface call method and device
US10366514B2 (en) 2008-04-05 2019-07-30 Sococo, Inc. Locating communicants in a multi-location virtual communications environment

Families Citing this family (140)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8645137B2 (en) 2000-03-16 2014-02-04 Apple Inc. Fast, language-independent method for user authentication by voice
US7647564B2 (en) * 2003-02-28 2010-01-12 Bea Systems, Inc. System and method for dynamically generating a graphical user interface
US20040230328A1 (en) * 2003-03-21 2004-11-18 Steve Armstrong Remote data visualization within an asset data system for a process plant
US8127252B2 (en) * 2003-11-07 2012-02-28 Microsoft Corporation Method and system for presenting user interface (UI) information
US7644367B2 (en) 2003-05-16 2010-01-05 Microsoft Corporation User interface automation framework classes and interfaces
EP1776647A4 (en) * 2004-03-16 2009-12-30 Freedom Scientific Inc Multimodal xml delivery system and method
US8744852B1 (en) 2004-10-01 2014-06-03 Apple Inc. Spoken interfaces
US7818755B1 (en) * 2004-12-22 2010-10-19 Apple Inc. Window server event taps
US7676549B2 (en) * 2005-05-27 2010-03-09 Microsoft Corporation Techniques for providing accessibility options in remote terminal sessions
US8677377B2 (en) 2005-09-08 2014-03-18 Apple Inc. Method and apparatus for building an intelligent automated assistant
US8863010B2 (en) * 2005-12-27 2014-10-14 Sap Ag Linking user interface design environment and application development environment
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
JP5172156B2 (en) * 2007-01-15 2013-03-27 富士通株式会社 Information processing apparatus, display method, and display program
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US20090106764A1 (en) * 2007-10-22 2009-04-23 Microsoft Corporation Support for globalization in test automation
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US8996376B2 (en) 2008-04-05 2015-03-31 Apple Inc. Intelligent text-to-speech conversion
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
JP2011521383A (en) 2008-05-20 2011-07-21 ザ・フィードルーム， インコーポレイテッド System and method for real-time creation and modification of a video player for users with disabilities
US20100030549A1 (en) 2008-07-31 2010-02-04 Lee Michael M Mobile device having human language translation capability with positional feedback
US20100082733A1 (en) * 2008-09-30 2010-04-01 Microsoft Corporation Extensible remote programmatic access to user interface
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US9241062B2 (en) * 2009-05-20 2016-01-19 Citrix Systems, Inc. Methods and systems for using external display devices with a mobile computing device
US10706373B2 (en) 2011-06-03 2020-07-07 Apple Inc. Performing actions associated with task items that represent tasks to perform
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
WO2011089450A2 (en) 2010-01-25 2011-07-28 Andrew Peter Nelson Jerram Apparatuses, methods and systems for a digital conversation management platform
US8682667B2 (en) 2010-02-25 2014-03-25 Apple Inc. User profiling for selecting user specific voice input processing information
US8433828B2 (en) 2010-02-26 2013-04-30 Apple Inc. Accessory protocol for touch screen device accessibility
US20110307831A1 (en) * 2010-06-10 2011-12-15 Microsoft Corporation User-Controlled Application Access to Resources
US8966447B2 (en) * 2010-06-21 2015-02-24 Apple Inc. Capturing and displaying state of automated user-level testing of a graphical user interface application
US10762293B2 (en) 2010-12-22 2020-09-01 Apple Inc. Using parts-of-speech tagging and named entity recognition for spelling correction
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US8994660B2 (en) 2011-08-29 2015-03-31 Apple Inc. Text correction processing
US8607251B1 (en) * 2011-09-21 2013-12-10 Google Inc. Dynamic accessibility behavior
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9280610B2 (en) 2012-05-14 2016-03-08 Apple Inc. Crowd sourcing information to fulfill user requests
US9721563B2 (en) 2012-06-08 2017-08-01 Apple Inc. Name recognition system
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9547647B2 (en) 2012-09-19 2017-01-17 Apple Inc. Voice-based media searching
EP2954514B1 (en) 2013-02-07 2021-03-31 Apple Inc. Voice trigger for a digital assistant
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
WO2014144579A1 (en) 2013-03-15 2014-09-18 Apple Inc. System and method for updating an adaptive speech recognition model
AU2014233517B2 (en) 2013-03-15 2017-05-25 Apple Inc. Training an at least partial voice command system
WO2014197334A2 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
WO2014197336A1 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
WO2014197335A1 (en) 2013-06-08 2014-12-11 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
DE112014002747T5 (en) 2013-06-09 2016-03-03 Apple Inc. Apparatus, method and graphical user interface for enabling conversation persistence over two or more instances of a digital assistant
KR101809808B1 (en) 2013-06-13 2017-12-15 애플 인크. System and method for emergency calls initiated by voice command
US9979638B2 (en) * 2013-06-19 2018-05-22 Hcl Technologies Limited Systems and methods to construct engineering environment supporting API enablement for software defined networking
AU2014306221B2 (en) 2013-08-06 2017-04-06 Apple Inc. Auto-activating smart responses based on activities from remote devices
US9703684B1 (en) * 2013-10-01 2017-07-11 Amazon Technologies, Inc. Unified user element information provisioning
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US10592095B2 (en) 2014-05-23 2020-03-17 Apple Inc. Instantaneous speaking of content on touch devices
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
AU2015266863B2 (en) 2014-05-30 2018-03-15 Apple Inc. Multi-command single utterance input method
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US10140102B2 (en) * 2014-08-21 2018-11-27 International Business Machines Corporation Evaluating accessibility compliance of a hybrid user interface design
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US10671235B2 (en) 2015-03-03 2020-06-02 Soroco Private Limited Software robots for programmatically controlling computer programs to perform tasks
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US9578173B2 (en) 2015-06-05 2017-02-21 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US10459600B2 (en) 2015-06-24 2019-10-29 Microsoft Technology Licensing, Llc Conversion of platform-independent accessibility logic into platform-specific accessibility functionality
US9734312B1 (en) * 2015-08-12 2017-08-15 Symantec Corporation Systems and methods for detecting when users are uninstalling applications
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
DK179309B1 (en) 2016-06-09 2018-04-23 Apple Inc Intelligent automated assistant in a home environment
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
DK179343B1 (en) 2016-06-11 2018-05-14 Apple Inc Intelligent task discovery
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
DK179049B1 (en) 2016-06-11 2017-09-18 Apple Inc Data driven natural language event detection and classification
DK179415B1 (en) 2016-06-11 2018-06-14 Apple Inc Intelligent device arbitration and control
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
EP3340072B1 (en) * 2016-12-23 2021-02-17 Solute GmbH Method for performing an automatic request for data and program product
DK201770439A1 (en) 2017-05-11 2018-12-13 Apple Inc. Offline personal assistant
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
DK179745B1 (en) 2017-05-12 2019-05-01 Apple Inc. SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT
DK201770431A1 (en) 2017-05-15 2018-12-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
DK201770432A1 (en) 2017-05-15 2018-12-21 Apple Inc. Hierarchical belief states for digital assistants
DK179549B1 (en) 2017-05-16 2019-02-12 Apple Inc. Far-field extension for digital assistant services
US11520687B2 (en) 2020-11-20 2022-12-06 Bank Of America Corporation System and method for automated testing of an accessibility feature of a user device by emulating actions of users

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5327529A (en) * 1990-09-24 1994-07-05 Geoworks Process of designing user's interfaces for application programs
US5613122A (en) 1994-11-14 1997-03-18 Object Technology Licensing Corp. Object-oriented operating system
EP0941515A1 (en) 1995-10-31 1999-09-15 Frederick S.M. Herz System for customized electronic identification of desirable objects
US6161126A (en) 1995-12-13 2000-12-12 Immersion Corporation Implementing force feedback over the World Wide Web and other computer networks
AU2759297A (en) * 1996-05-20 1997-12-09 Cognos Incorporated Application services builder
CH690875A5 (en) 1996-05-21 2001-02-15 Hts High Technology Systems Ag Home and building automation system.
US5923328A (en) 1996-08-07 1999-07-13 Microsoft Corporation Method and system for displaying a hierarchical sub-tree by selection of a user interface element in a sub-tree bar control
US6144377A (en) * 1997-03-11 2000-11-07 Microsoft Corporation Providing access to user interface elements of legacy application programs
US6341280B1 (en) 1998-10-30 2002-01-22 Netscape Communications Corporation Inline tree filters
KR20010082529A (en) * 1998-11-19 2001-08-30 추후제출 Unified computing and communication architecture(ucca)
EP1069500A1 (en) * 1999-07-12 2001-01-17 International Business Machines Corporation Downloadable user-interface
US6532023B1 (en) * 1999-08-12 2003-03-11 International Business Machines Corporation Recording selected applet events of a user interaction sequence
AU2002239297A1 (en) 2000-11-16 2002-06-03 Mydtv, Inc. System and methods for determining the desirability of video programming events

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9755966B2 (en) 2007-10-24 2017-09-05 Sococo, Inc. Routing virtual area based communications
US9411489B2 (en) 2007-10-24 2016-08-09 Sococo, Inc. Interfacing with a spatial virtual communication environment
US9411490B2 (en) 2007-10-24 2016-08-09 Sococo, Inc. Shared virtual area communication environment based apparatus and methods
US9483157B2 (en) 2007-10-24 2016-11-01 Sococo, Inc. Interfacing with a spatial virtual communication environment
USRE46309E1 (en) 2007-10-24 2017-02-14 Sococo, Inc. Application sharing
US9762641B2 (en) 2007-10-24 2017-09-12 Sococo, Inc. Automated real-time data stream switching in a shared virtual area communication environment
US10158689B2 (en) 2007-10-24 2018-12-18 Sococo, Inc. Realtime kernel
US10366514B2 (en) 2008-04-05 2019-07-30 Sococo, Inc. Locating communicants in a multi-location virtual communications environment
US10003624B2 (en) 2009-01-15 2018-06-19 Sococo, Inc. Realtime communications and network browsing client
CN101835101A (en) * 2009-02-27 2010-09-15 捷讯研究有限公司 The mobile radio communications system and the correlation technique of appliance icon notification indicia framing are provided
CN102804161A (en) * 2009-04-03 2012-11-28 社会传播公司 Application sharing
US9853922B2 (en) 2012-02-24 2017-12-26 Sococo, Inc. Virtual area communications
CN109542644A (en) * 2018-11-19 2019-03-29 北京小米移动软件有限公司 Application programming interface call method and device
CN109542644B (en) * 2018-11-19 2022-12-09 北京小米移动软件有限公司 Application programming interface calling method and device

Also Published As

Publication number Publication date
JP2009199618A (en) 2009-09-03
AU2003299163A1 (en) 2004-04-23
JP4942932B2 (en) 2012-05-30
US7434167B2 (en) 2008-10-07
WO2004031947A1 (en) 2004-04-15
KR100965708B1 (en) 2010-06-24
WO2004031947A8 (en) 2004-07-08
KR20050056191A (en) 2005-06-14
EP1546871A1 (en) 2005-06-29
US20040064593A1 (en) 2004-04-01
CN1678994B (en) 2010-05-26
JP2006501566A (en) 2006-01-12

Similar Documents

Publication Publication Date Title
CN1678994A (en) System and method for providing access to user interface information
US7471646B2 (en) System and methods for inline property editing in tree view based editors
CA2481552C (en) Programming interface for a computer platform
JP5026415B2 (en) Data centric workflow
CN101615122B (en) Method and system for providing information related to elements of a user interface by an access system
CN1534949A (en) Method and equipment used for obtaining state information in network
CN1811704A (en) System and method for a context-awareness platform
CN1855041A (en) Method for implementing a virtual push-to-talk function and multimodal interface
CN1813233A (en) User interface for controlling access to computer objects
CN1509442A (en) Automated management of internet and/or web contert
CN101821993A (en) Method and system for handling failover in distributed environment that uses session affinity
CN1639697A (en) Network management system, display method, and program
CN1487391A (en) Accessible system event mechanism and method
CN1856784A (en) System and method for locking and isolation in a storage platform
WO2013106355A1 (en) Declarative dynamic control flow in continuation-based runtime
US20040061714A1 (en) Logical element tree and method
CN1620655A (en) System and method for controlling user interface attribute by data
JP2007538314A (en) General user interface command architecture
CN1975691A (en) Method and apparatus for a database workload simulator
CN1459738A (en) System and method of roaming user in wide area network
US10223457B2 (en) Identification of distributed user interface (DUI) elements
Kaufman et al. Understanding the Architecture
NZ524858A (en) Receiving user interface information from server for keeping track of changes to user interface depending on client's registration

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
ASS Succession or assignment of patent right

Owner name: MICROSOFT TECHNOLOGY LICENSING LLC

Free format text: FORMER OWNER: MICROSOFT CORP.

Effective date: 20150429

C41 Transfer of patent application or patent right or utility model
TR01 Transfer of patent right

Effective date of registration: 20150429

Address after: Washington State

Patentee after: Micro soft technique license Co., Ltd

Address before: Washington State

Patentee before: Microsoft Corp.

CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20100526

Termination date: 20190516