US20030001854A1 - Capturing graphics primitives associated with any display object rendered to a graphical user interface - Google Patents

Capturing graphics primitives associated with any display object rendered to a graphical user interface Download PDF

Info

Publication number
US20030001854A1
US20030001854A1 US09/892,989 US89298901A US2003001854A1 US 20030001854 A1 US20030001854 A1 US 20030001854A1 US 89298901 A US89298901 A US 89298901A US 2003001854 A1 US2003001854 A1 US 2003001854A1
Authority
US
United States
Prior art keywords
target process
component
graphics primitives
primitives
spy
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US09/892,989
Other versions
US6802055B2 (en
Inventor
Shashidhar Jade
Peter Thompson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Priority to US09/892,989 priority Critical patent/US6802055B2/en
Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: JADE, SHASHIDHAR P., THOMPSON, PETER A.
Publication of US20030001854A1 publication Critical patent/US20030001854A1/en
Application granted granted Critical
Publication of US6802055B2 publication Critical patent/US6802055B2/en
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Adjusted expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/451Execution arrangements for user interfaces
    • G06F9/454Multi-language systems; Localisation; Internationalisation

Definitions

  • This invention relates generally to methods and systems for retrieving information descriptive of graphic elements that are displayed to a user interface and, more particularly, relates to methods and systems for retrieving graphics primitives and associated attributes of such graphics primitives of a display object rendered to a user interface.
  • GUI graphical user interface
  • Each graphics primitive has its own set of attributes that define its appearance and/or style within the user interface. These attributes include visual and stylistic characteristics such as the text style or font, line length and style, arc length, etc.
  • attributes include visual and stylistic characteristics such as the text style or font, line length and style, arc length, etc.
  • multiple graphics primitives are combined to create the various display objects (e.g. buttons, menus, dialogue boxes, etc.) that are displayed to the user when they are using the application.
  • a display object such as a dialog box can include a data entry field, user buttons containing text characters or strings, and/or other graphics primitives.
  • the localization tool must be able to access all of the text primitives for the dialog box.
  • the specific attributes of the button such as the length and shape of the button, must be known in order to account for changes in the length of a string due to translation.
  • the graphics primitives that comprise the various display objects within an application can be accessed by conventional means.
  • the graphics primitives are indicated by a resource file (*.res) that is stored within the application's executable (*.exe).
  • Resource files are simply plain text scripts that indicate the various resources required for the application to run successfully, and can be viewed with a standard resource editor/viewer tool.
  • the resource files are converted into a binary representation at compile time, and then merged into the executable image of the application during runtime.
  • Resource types include text string tables, which contain the various text strings that are displayed by the application during runtime.
  • resource types often required by an application include menus, dialog boxes, cursors, icons, toolbars, bitmaps and other display objects composed of one or more graphics primitives.
  • the resource files provide access to all of the display objects, and consequently the graphics primitives associated with the application.
  • the localization tool must also be able to know where and how the various text strings indicated in the resource file are used within the application.
  • the resource file indicates all of the graphics primitives relative to the executable application, and includes a text string table that contains the various literal strings and text characters displayed by the application during runtime. While the strings within the table can be easily accessed and localized accordingly, the table does not explicitly indicate the display object that a particular string corresponds to. The actual usage, or context of the string cannot be determined until it is displayed by the application during runtime.
  • Context refers specifically to any information that allows the localization tool to account for the differences in meaning that occur when the same string or phrase is displayed in different ways within the application.
  • the term ‘O.K.’ may have a different meaning as it appears in a dialogue box than in a pull-down menu.
  • the localized product In addition to having accurately translated strings that are used in the correct context, the localized product must also maintain the same font properties as the original application. For instance, a button within the original application having text that reads “EXIT” should read as “SALIDA” when localized for Spanish speaking users. The literal meaning of the text as well as the font properties, which in this case are Times New Roman, bold and italicize to name a few, should be maintained from one version to the next. However, if the localized button reads as “SALIDA”, the intentional emphasis placed on the original text is lost. This can cause problems in applications where varying font sizes, typeface, and styles are required to effectively convey information to a user of the application.
  • font properties of text strings are generally stored within a temporary data structure allocated for the string known as a device context.
  • the information maintained within this data structure, including the font properties, is discarded by the application from memory after the text is drawn to the screen. Again, this information can only be determined during the actual runtime of the application.
  • the present invention provides a mechanism for capturing the one or more graphics primitives associated with an application as it is in execution. Moreover, the invention allows for the detection and retrieval of the unique attributes of the one or more graphics primitives as they are drawn to a graphical user interface. These graphic capturing techniques can be applied directly to any controls, buttons, windows and/or any other display objects that can be invoked by an application, including those that are custom drawn or non-standard with respect to the operating system.
  • a calling process such as a localization tool, utilizes the invention to capture graphics primitives, such as text strings, that are displayed to the screen by a target process during runtime.
  • the target process is any computer implemented process or software application that requires a graphical user interface (GUI) to display visual information to a computer user.
  • GUI graphical user interface
  • the calling process invokes an injection DLL (Dynamic Link Library) to inject a spy DLL into the executable code of the target process.
  • the spy DLL installs patches and hook functions into the operating system API's (Application Programming Interfaces) that have routines for displaying text to a graphical user interface.
  • the hook functions monitor the operating system messages generated during the execution of the target application, while the patches allow for the capture of the various graphics primitives and associated attributes of the primitives that are drawn to the user interface.
  • a display object is rendered to the GUI by the target application as a result of an invoked action (e.g. mouse-clicking, function key)
  • the hook functions are called to capture the operating system messages passed and the patches capture the graphics primitives of the object.
  • the patches also capture the unique attributes of the graphics primitives, including the font properties of a displayed text.
  • This captured information is then packaged and delivered to the calling process for processing. Because the graphics primitives are captured in connection with the operating system messages passed during runtime, the calling process obtains complete information about any viewable or executable objects displayed by the target process.
  • the operating system messages provide a context for a captured graphics primitive, which allows the calling process to better associate a captured primitive with a specific display object.
  • a text string primitive can be easily associated with a specific dialogue box that is called by the application as a result of a user action.
  • the invention allows the graphics primitives and associated attributes of custom/user drawn objects to be captured. This overcomes the limitations imposed by the operating system on allowing the unique attributes of non-standard objects to be exposed by the resource file for the application.
  • FIG. 1 is a block diagram generally illustrating an exemplary computer system on which the present invention resides;
  • FIG. 2 is a diagram illustrating the target application within a user interface having a dialog box spawned as a result of an invoked action within the target application;
  • FIG. 3 is a functional block diagram illustrating the major components of the invention.
  • FIG. 4 is a flow chart illustrating the method executed by the calling process for capturing graphics primitives and system messages generated by the target process
  • FIG. 5 is a diagram illustrating a data structure containing information descriptive of the graphics primitives that comprise a display object.
  • program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
  • program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
  • program modules may be practiced with other computer system configurations, including hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like.
  • the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.
  • program modules may be located in both local and remote memory storage devices.
  • FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented.
  • the computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100 .
  • the invention relates to the capture and retrieval of the one or more graphics primitives that comprise a display object rendered to a graphical user interface by an application executing on the operating system 100 .
  • a display object is any drawing element that can be viewed by a user of a computer from a user interface screen (monitor) 191 . More specifically, a display object refers to any drawing element that displays lines, text, images and other visible information to a graphical user interface (GUI). This includes, but is not limited to control and dialogue boxes, functional buttons, menu screens, combo boxes, and any graphical windows that are capable of being executed or invoked by an application running on the operating system 144 .
  • the graphics primitives that comprise the display object can be easily captured by practicing the methods of the invention, and subsequently returned to a calling process such as a localization tool or text-to-speech application for processing.
  • a “calling process” is the process that utilizes the invention to capture the one or more graphics primitives of a display object that can be invoked by the various application programs 145 on the computer.
  • the graphics primitives include drawing elements such as text characters or strings, lines, arcs, polygons, etc., and have associated attributes that define its visual appearance such as font size, line length, and arc length.
  • the calling process can be any computer executable process, such as a localization tool or text-to-speech application that requires access to the drawing elements that are drawn to a graphical user interface by an application.
  • a “target application” or “target process” refers to the particular process, executable procedure, or application that a particular text string is to be captured from.
  • the calling process captures the one or more graphic primitives and associated attributes of the primitives that are displayed to a graphical user interface by the target application/process.
  • Display objects are commonly used within operating systems that support a graphical user interface.
  • Windows operating system a standard set of display objects are available for use by the various application programs 145 that execute on the computer 110 .
  • a dialog box or menu display used by Microsoft Word for example can also be called upon by Microsoft Excel to provide the same functional purpose.
  • Standard display objects have a mode of operation that is defined according to the functions and mechanisms of the operating system, while the appearance of the display object is defined according to one or more graphics primitives that comprise the object. For instance, multiple lines, curves, and text characters can be combined to yield a user button.
  • the appearance of the button is further determined by the specific attributes of the graphics primitives such as the font type, text alignment and placement, line thickness and shading, style, etc.
  • graphics primitives and associated attributes can be easily determined through conventional methods, such as by accessing the resource file of the target application.
  • access to the unique attributes of objects that are non-standard with respect to the operating system cannot be determined through conventional means.
  • the present invention allows for the capture of the graphics primitives and associated attributes of both standard and non-standard display objects.
  • Non-standard display objects typically fall into two categories, namely owner-draw controls or custom class objects.
  • Owner-draw controls are display objects that have a customized style or appearance and perform tasks that differ from that of the standard objects.
  • custom class objects consist of unique functions or class names that are not recognized by the OS. Because these types of objects are not standard with respect to the operating system, their unique attributes associated with these objects cannot be determined by conventional methods. For instance, the resource file of an application having an owner-draw or custom class control does not reveal the specific graphics primitives or attributes of the object. Customized features cannot be determined directly from the static (non-executed) resource file.
  • context information is information that is descriptive of the display object in which the text appears. This descriptive information can include parameters such as the type of display object (dialog box, menu, window, etc.) and its current state (active/inactive). Context information also includes system information such as the API calls and/or function calls made by the target application to render the display object to a user interface, the object handle or resource ID, the specific location of files called during execution of the display object, and any other information that provides a general context for the text that is displayed to the user interface screen 191 during the execution of the target process or application.
  • Context information is obtained by intercepting the system messages that are passed between the target application and the operating system during the execution of the application. Capturing the text in connection with the operating system messages provides a general context for the text, and allows the information to be associated with a specific task or process within the target application.
  • the computing system environment 100 includes a general purpose computing device in the form of a computer 110 .
  • Components of computer 110 may include, but are not limited to, a processing unit 120 , a system memory 130 , and a system bus 121 that couples various system components including the system memory to the processing unit 120 .
  • the system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
  • such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
  • ISA Industry Standard Architecture
  • MCA Micro Channel Architecture
  • EISA Enhanced ISA
  • VESA Video Electronics Standards Association
  • PCI Peripheral Component Interconnect
  • Computer 110 typically includes a variety of computer readable media.
  • Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media.
  • Computer readable media may comprise computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 110 .
  • Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
  • modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
  • the system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132 .
  • ROM read only memory
  • RAM random access memory
  • BIOS basic input/output system
  • RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120 .
  • FIG. 1 illustrates operating system 134 , application programs 135 , other program modules 136 , and program data 137 .
  • the computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media.
  • FIG. 1 illustrates a hard disk drive 140 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152 , and an optical disk drive 155 that reads from or writes to a removable, nonvolatile optical disk 156 such as a CD ROM or other optical media.
  • removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
  • the hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140
  • magnetic disk drive 151 and optical disk drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150 .
  • the drives and their associated computer storage media discussed above and illustrated in FIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the computer 110 .
  • hard disk drive 141 is illustrated as storing operating system 144 , application programs 145 , other program modules 146 , and program data 147 .
  • operating system 144 application programs 145 , other program modules 146 , and program data 147 are given different numbers here to illustrate that, at a minimum, they are different copies.
  • a user may enter commands and information into the computer 20 through input devices such as a keyboard 162 and pointing device 161 , commonly referred to as a mouse, trackball or touch pad.
  • Other input devices may include a microphone, joystick, game pad, satellite dish, scanner, or the like.
  • These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB).
  • a monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190 .
  • computers may also include other peripheral output devices such as speakers 197 and printer 196 , which may be connected through an output peripheral interface 190 .
  • the computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180 .
  • the remote computer 180 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 110 , although only a memory storage device 181 has been illustrated in FIG. 1.
  • the logical connections depicted in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173 , but may also include other networks.
  • LAN local area network
  • WAN wide area network
  • Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
  • the personal computer 110 When used in a LAN networking environment, the personal computer 110 is connected to the LAN 171 through a network interface or adapter 170 .
  • the computer 110 When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173 , such as the Internet.
  • the modem 172 which may be internal or external, may be connected to the system bus 121 via the user input interface 160 , or other appropriate mechanism.
  • program modules depicted relative to the personal computer 110 may be stored in the remote memory storage device.
  • FIG. 1 illustrates remote application programs 185 as residing on memory device 181 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
  • a target application 202 with a display object 204 is shown.
  • the display object is spawned as a result of an invoked action within the target application, such as from clicking one of the menu items 212 with the mouse 161 .
  • Operating systems that support GUIs such as Windows, contain numerous types of display objects that are presented to a user interface 200 , including user buttons 206 , menu items 210 and dialogue boxes 204 .
  • the display object is a dialogue box 204 consisting of three user option buttons 206 , where one of them 210 is owner drawn.
  • the dialogue box 204 also consists of a text field 208 for displaying text characters, having defined font properties to the GUI.
  • buttons 206 shown within the dialogue box and the associated text field 208 are child controls of the dialogue box.
  • a child control can be best described as a display object that is initiated by a parent object.
  • the dialogue box 204 is the parent object, and has four children associated with it.
  • Each of the objects has its own handle, which is a unique identification number assigned by the operating system 144 that distinguishes one object from another.
  • the handles of the various display objects are also indicated in the resource file associated with the executable application.
  • the dialogue box has its own unique handle, and each of its children has its own handle, which in many operating systems is inherited directly from the parent object.
  • This data structure is not indicated within the resource file, and not maintained within memory after the text it relates to is drawn to the screen. Therefore, the only way to access the font properties of a text string or character is to instantly capture the font information from the data structure as it is drawn to the graphical user interface.
  • the invention allows the font properties to be captured during runtime of the application.
  • a calling process 300 injects DLLs into a target application 302 to “spy” on or monitor the operating system messages generated during target application execution. This technique is useful for determining the internal state of an application as it is in operation.
  • a spy component 308 is used to perform the spying. It is an executable program module, such as a DLL, consisting of various functions and data for monitoring an application or process as it is in execution. The spy component 308 is injected into the target process or application 302 via an injection component 306 (event 406 ).
  • This injection component 306 is invoked by the calling process 300 (event 404 ), which is an executable program module, such as a DLL, having executable instructions for injecting source code and/or program modules into a target process 302 .
  • the spy component installs function patches into the operating system's application programming interfaces (APIs) that have executable instructions and routines for outputting graphics primitives to a user interface (event 410 ).
  • APIs application programming interfaces
  • the primary API for Windows known as Win32, provides a complete set of functions to format and draw text primitives in a target application's 302 user interface 200 (e.g.).
  • DrawText( ), TextOut( ), and PolyTextOut( ) are responsible for outputting individual characters, symbols, or entire strings of text to the screen.
  • these functions are the functions that are patched by the spy component 308 after injection (event 410 ).
  • the function patches serve the specific purpose of capturing the text that is rendered to the user interface screen 191 during the execution of the target process or application 302 .
  • the spy component installs one or more hook functions into the operating system's APIs to monitor system messages passed during execution of the target application/process 302 (event 412 ).
  • Hook functions intercept system messages that are passed between the operating system and system threads during process execution. Whenever an action such as the pressing of a function key, click of a mouse, or activation of a dialogue box occurs, the OS generates a message that passes through a chain of hook procedures before reaching the target process.
  • Standard hook functions are utilized to monitor the system messages that pertain specifically to the output of display objects to a user interface screen 191 .
  • the WH_MSGFILTER and WH_SYSMSGFILTER hooks are used to monitor messages that are to be processed by a menu, scroll bar, message box, or dialog box.
  • the hook functions are installed and uninstalled accordingly by a hook management component 304 , which is called upon by the spy component after it is injected 310 into the target process.
  • the hook management component is a separate process from the calling process and target process, and has executable instructions for installing and uninstalling hook functions within a process or application designated by the spy component.
  • the designated process is the target process or application. Because the hook management component operates as an independent process, one or more calling processes can use the hook management component to install hooks within one or more target processes. In this way, the system messages of multiple independently running processes or applications can be monitored during runtime.
  • the generated message is captured by the one or more hook functions (event 414 ).
  • This action results in the release of a trigger, or flag, that activates the function patches to capture the graphics primitives that are drawn to the screen to render the display object (event 416 ).
  • the system messages captured by the one or more hook functions are then used to invalidate the display object under execution (event 418 ).
  • the invalidation of a display object is a process whereby a selected or active display object is redrawn due to a user or system invoked change to the object. For instance, a display object is invalidated each time a user resizes the display object or moves it to a different position within the user interface 200 . When such an action occurs, the display object is redrawn by calling the same API functions and routines that rendered it to the interface screen the first time.
  • the appropriate API function calls are made, which in turn enumerates the display objects to the screen.
  • the installed function patches capture the graphics primitives, such as the text and other drawing elements that are drawn to the interface screen 200 (event 420 ).
  • the graphics primitives and associated attributes of the graphics primitives related to the display object are captured in association with the context information provided by the system messages.
  • FIG. 5 is illustrative of the data structure containing the captured graphics primitives, associated attributes and context information related to the dialogue box 204 of FIG. 2. While this information is captured and retrieved in a similar format as shown in the figure, those skilled in the art will recognize that the actual data, and format of the data is dependent upon the type of action invoked within the target process. Moreover, the information contained within the data structure will differ from one type of display object to another, as each object can consist of various buttons, text fields, and other drawing elements. Therefore, the information that is captured and stored within the data structure will vary from one process to another.
  • the captured information includes data and parameters that are descriptive of the display object 204 both functionally and graphically.
  • the object type as shown is a DIALOGUE_BOX 500 , which further consists of other data and parameters that define its appearance and operation. These parameters include the object's four child controls, namely TEXT_FIELD 502 , OWNER_DRAWN BUTTON 504 , BUTTON — 1 506 , and BUTTON — 2 508 (these names were assigned for illustrative purposes only).
  • each of the controls is a user button, and one of the buttons is owner drawn 210 .
  • Each button is a display object that is composed of one or more graphics primitives, such as text characters and lines that define its shape.
  • the primitives have unique attributes, such as font type, text size, line length and other stylistic characteristics.
  • the sequential format of the information as shown in FIG. 5 is due to the function patches that capture the graphics primitives and associated attributes.
  • the graphics primitives and attributes of such primitives are instantly captured, stored in the data structure in the order in which the various attributes are drawn to the screen, and returned to the calling process.
  • the invention as described herein can be incorporated into the source code of a calling process directly, or called upon by the calling process to capture and retrieve the graphics primitives and associated context information of a display object.
  • the calling process can be any computer executable application where the capture of the various graphics primitives that are output to a user interface by a target process is required. Once captured, the calling process can process this information accordingly. Examples of applications that can practice the capturing techniques disclosed include, but are not limited to localization tools, language processing applications and text-to-speech applications. Also, while the methods disclosed are applicable to various operating systems and platforms, the ability to capture graphics primitives and associated attributes in connection with context information can be particularly useful within the Windows OS to better support Active Accessibility applications.

Abstract

The present invention provides a mechanism for capturing the one or more graphics primitives drawn to a user interface by an application in execution is presented. A calling process invokes an injection component to inject a spy component into a target process. Once the spy component is injected, it installs patches and hook functions into the executable code of the operating system API's that relate to the output of graphics primitives to the graphical user interface. The hook functions monitor operating system messages generated during the execution of the target process/application, while the patches allow for the capture of graphics primitives and attributes associated with such graphics primitives. Whenever a display object is rendered to an interface screen as a result of an invoked action within the target process, the hook functions are called to capture the operating system messages passed, and the patches capture the graphics primitives that are drawn to the screen to render the display object. This information is then packaged and delivered to the calling process for processing. Because the graphics primitives and attributes associated with such graphics primitives are captured in connection with the operating system messages passed during process execution, the calling process is able to obtain complete information about any viewable or executable object displayed by the target process.

Description

    TECHNICAL FIELD
  • This invention relates generally to methods and systems for retrieving information descriptive of graphic elements that are displayed to a user interface and, more particularly, relates to methods and systems for retrieving graphics primitives and associated attributes of such graphics primitives of a display object rendered to a user interface. [0001]
  • BACKGROUND OF THE INVENTION
  • To successfully compete in the global market, a company's advertising literature and products must be easily understood by everyone, regardless of language or cultural differences. This requirement is perhaps most apparent within the vast software technology market, where software tools such as Microsoft Word, are commonplace on a global scale and must be comprehensible to users of any culture. The need for the accurate representation of language within software products marketed and sold worldwide is the essence of the localization industry. Localization of a product is the accurate translation and adaptation of any software or executable/viewable code into the language of the locality into which the product is being marketed and sold. [0002]
  • In order to work effectively across geographic and cultural borders, a localized software product must have the highest quality translation from a source language to a native or local language while retaining the functionality of the original product. The layout of text within a graphical user interface (GUI) is one of the biggest obstacles to overcome when localizing a product because the localized version of code must have the same general appearance and meaning as the original. This requires that the localization tool used to perform the translation be able to completely access and receive as input all of the graphics primitives that are rendered to the user interface screen during the execution of the software application. A graphics primitive is a drawing element, such as a text character, line, arc, polygon, etc., that is drawn to a user interface according to the specific function calls and mechanisms of the operating system. Each graphics primitive has its own set of attributes that define its appearance and/or style within the user interface. These attributes include visual and stylistic characteristics such as the text style or font, line length and style, arc length, etc. In GUI based software applications, multiple graphics primitives are combined to create the various display objects (e.g. buttons, menus, dialogue boxes, etc.) that are displayed to the user when they are using the application. [0003]
  • Because a typical application can include many different display objects, the graphics primitives that comprise the objects provide the primary information and data to be localized. For instance, a display object such as a dialog box can include a data entry field, user buttons containing text characters or strings, and/or other graphics primitives. To properly translate the text strings, the localization tool must be able to access all of the text primitives for the dialog box. Likewise, the specific attributes of the button, such as the length and shape of the button, must be known in order to account for changes in the length of a string due to translation. Once the graphics primitives that comprise the various display objects are determined, they can be localized accordingly, and the user interface of the application as a whole can be modified to suit the intended locality. [0004]
  • The graphics primitives that comprise the various display objects within an application can be accessed by conventional means. In Windows based applications for example, the graphics primitives are indicated by a resource file (*.res) that is stored within the application's executable (*.exe). Resource files are simply plain text scripts that indicate the various resources required for the application to run successfully, and can be viewed with a standard resource editor/viewer tool. The resource files are converted into a binary representation at compile time, and then merged into the executable image of the application during runtime. Resource types include text string tables, which contain the various text strings that are displayed by the application during runtime. Other resource types often required by an application include menus, dialog boxes, cursors, icons, toolbars, bitmaps and other display objects composed of one or more graphics primitives. The resource files provide access to all of the display objects, and consequently the graphics primitives associated with the application. [0005]
  • Despite the extensive information provided from the resource files, however, many localization errors still occur because one or more text strings are missed during the localization process. This is because standard methods of capturing graphics primitives and associated attributes of such graphics primitives are limited to only those display objects that are standard objects of the operating system (OS). Yet, there are many GUI based software applications that contain “custom-class” or “owner-draw” controls. These types of controls represent customized display objects that perform special functions or that have attributes that differ from the standard set of objects provided by the OS. So, while these customized objects are indicated as resources of the application within the resource file, the specific graphics primitives and associated attributes of the primitives that comprise the objects cannot be obtained directly from the resource file for localization. Rather, the primitives of customized display objects cannot be revealed until the object is invoked by the application during runtime. [0006]
  • Access to the graphics primitives that comprise the various display objects within the application, however, is still not enough to ensure a literal translation of a software product. The localization tool must also be able to know where and how the various text strings indicated in the resource file are used within the application. As described, the resource file indicates all of the graphics primitives relative to the executable application, and includes a text string table that contains the various literal strings and text characters displayed by the application during runtime. While the strings within the table can be easily accessed and localized accordingly, the table does not explicitly indicate the display object that a particular string corresponds to. The actual usage, or context of the string cannot be determined until it is displayed by the application during runtime. Context refers specifically to any information that allows the localization tool to account for the differences in meaning that occur when the same string or phrase is displayed in different ways within the application. For example, the term ‘O.K.’ may have a different meaning as it appears in a dialogue box than in a pull-down menu. [0007]
  • In addition to having accurately translated strings that are used in the correct context, the localized product must also maintain the same font properties as the original application. For instance, a button within the original application having text that reads “EXIT” should read as “SALIDA” when localized for Spanish speaking users. The literal meaning of the text as well as the font properties, which in this case are Times New Roman, bold and italicize to name a few, should be maintained from one version to the next. However, if the localized button reads as “SALIDA”, the intentional emphasis placed on the original text is lost. This can cause problems in applications where varying font sizes, typeface, and styles are required to effectively convey information to a user of the application. Unfortunately, there is no convenient way for the font properties of text strings to be captured during the localization process, such as from the resource file. This is because the font properties (which are attributes of a text primitive) are generally stored within a temporary data structure allocated for the string known as a device context. The information maintained within this data structure, including the font properties, is discarded by the application from memory after the text is drawn to the screen. Again, this information can only be determined during the actual runtime of the application. [0008]
  • To overcome the limitations discussed above, a way is needed to easily access the graphics primitives and associated attributes of any display object (standard and non-standard) called during the execution of an application. Likewise, a convenient means of determining the context of the text strings that get displayed to a user during the execution of the application is necessary to ensure that the text strings are associated with the correct display object. A way to capture the font properties of a text string or character is also needed so that this information is made available with the other attributes of a graphics primitive. [0009]
  • SUMMARY OF THE INVENTION
  • The present invention provides a mechanism for capturing the one or more graphics primitives associated with an application as it is in execution. Moreover, the invention allows for the detection and retrieval of the unique attributes of the one or more graphics primitives as they are drawn to a graphical user interface. These graphic capturing techniques can be applied directly to any controls, buttons, windows and/or any other display objects that can be invoked by an application, including those that are custom drawn or non-standard with respect to the operating system. [0010]
  • A calling process, such as a localization tool, utilizes the invention to capture graphics primitives, such as text strings, that are displayed to the screen by a target process during runtime. The target process is any computer implemented process or software application that requires a graphical user interface (GUI) to display visual information to a computer user. In operation, the calling process invokes an injection DLL (Dynamic Link Library) to inject a spy DLL into the executable code of the target process. Once the spy DLL is injected, it installs patches and hook functions into the operating system API's (Application Programming Interfaces) that have routines for displaying text to a graphical user interface. The hook functions monitor the operating system messages generated during the execution of the target application, while the patches allow for the capture of the various graphics primitives and associated attributes of the primitives that are drawn to the user interface. [0011]
  • Whenever a display object is rendered to the GUI by the target application as a result of an invoked action (e.g. mouse-clicking, function key), the hook functions are called to capture the operating system messages passed and the patches capture the graphics primitives of the object. The patches also capture the unique attributes of the graphics primitives, including the font properties of a displayed text. This captured information is then packaged and delivered to the calling process for processing. Because the graphics primitives are captured in connection with the operating system messages passed during runtime, the calling process obtains complete information about any viewable or executable objects displayed by the target process. The operating system messages provide a context for a captured graphics primitive, which allows the calling process to better associate a captured primitive with a specific display object. As an example, a text string primitive can be easily associated with a specific dialogue box that is called by the application as a result of a user action. Furthermore, the invention allows the graphics primitives and associated attributes of custom/user drawn objects to be captured. This overcomes the limitations imposed by the operating system on allowing the unique attributes of non-standard objects to be exposed by the resource file for the application. [0012]
  • Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments that proceeds with reference to the accompanying figures. [0013]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • While the appended claims set forth the features of the present invention with particularity, the invention, together with its objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which: [0014]
  • FIG. 1 is a block diagram generally illustrating an exemplary computer system on which the present invention resides; [0015]
  • FIG. 2 is a diagram illustrating the target application within a user interface having a dialog box spawned as a result of an invoked action within the target application; [0016]
  • FIG. 3 is a functional block diagram illustrating the major components of the invention; [0017]
  • FIG. 4 is a flow chart illustrating the method executed by the calling process for capturing graphics primitives and system messages generated by the target process; and [0018]
  • FIG. 5 is a diagram illustrating a data structure containing information descriptive of the graphics primitives that comprise a display object.[0019]
  • DETAILED DESCRIPTION OF THE INVENTION
  • Turning to the drawings, wherein like reference numerals refer to like elements, the invention is illustrated as being implemented in a suitable computing environment. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a personal computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. [0020]
  • FIG. 1 illustrates an example of a suitable [0021] computing system environment 100 on which the invention may be implemented. The computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100.
  • The invention relates to the capture and retrieval of the one or more graphics primitives that comprise a display object rendered to a graphical user interface by an application executing on the [0022] operating system 100. In the context of a graphical user interface, a display object is any drawing element that can be viewed by a user of a computer from a user interface screen (monitor) 191. More specifically, a display object refers to any drawing element that displays lines, text, images and other visible information to a graphical user interface (GUI). This includes, but is not limited to control and dialogue boxes, functional buttons, menu screens, combo boxes, and any graphical windows that are capable of being executed or invoked by an application running on the operating system 144. The graphics primitives that comprise the display object can be easily captured by practicing the methods of the invention, and subsequently returned to a calling process such as a localization tool or text-to-speech application for processing.
  • As described herein, the term “process” refers to an executable procedure, computing task, or part of a program that is executed by the [0023] computer 110. More specifically, a “calling process” is the process that utilizes the invention to capture the one or more graphics primitives of a display object that can be invoked by the various application programs 145 on the computer. The graphics primitives include drawing elements such as text characters or strings, lines, arcs, polygons, etc., and have associated attributes that define its visual appearance such as font size, line length, and arc length. The calling process can be any computer executable process, such as a localization tool or text-to-speech application that requires access to the drawing elements that are drawn to a graphical user interface by an application. Similarly, a “target application” or “target process” refers to the particular process, executable procedure, or application that a particular text string is to be captured from. In accordance with the invention, the calling process captures the one or more graphic primitives and associated attributes of the primitives that are displayed to a graphical user interface by the target application/process.
  • Display objects are commonly used within operating systems that support a graphical user interface. For example, in the Windows operating system, a standard set of display objects are available for use by the [0024] various application programs 145 that execute on the computer 110. In this way, a dialog box or menu display used by Microsoft Word for example, can also be called upon by Microsoft Excel to provide the same functional purpose. Standard display objects have a mode of operation that is defined according to the functions and mechanisms of the operating system, while the appearance of the display object is defined according to one or more graphics primitives that comprise the object. For instance, multiple lines, curves, and text characters can be combined to yield a user button. The appearance of the button is further determined by the specific attributes of the graphics primitives such as the font type, text alignment and placement, line thickness and shading, style, etc. Such graphics primitives and associated attributes can be easily determined through conventional methods, such as by accessing the resource file of the target application. However, access to the unique attributes of objects that are non-standard with respect to the operating system cannot be determined through conventional means.
  • In contrast, the present invention allows for the capture of the graphics primitives and associated attributes of both standard and non-standard display objects. Non-standard display objects typically fall into two categories, namely owner-draw controls or custom class objects. Owner-draw controls are display objects that have a customized style or appearance and perform tasks that differ from that of the standard objects. Similarly, custom class objects consist of unique functions or class names that are not recognized by the OS. Because these types of objects are not standard with respect to the operating system, their unique attributes associated with these objects cannot be determined by conventional methods. For instance, the resource file of an application having an owner-draw or custom class control does not reveal the specific graphics primitives or attributes of the object. Customized features cannot be determined directly from the static (non-executed) resource file. Rather, they can only be realized once the display object is invoked during the runtime of the application. Current methods of capturing graphics primitives are limited to the information contained within the “static” resource file, and do not support the capture of “dynamic” (runtime) information. The invention overcomes this limitation by allowing the graphics primitives and unique attributes of customized display objects to be captured during runtime execution of the application. [0025]
  • Still further, the invention also relates to a method and system for capturing “context information” associated with any text that is displayed during the execution of a process or application. In general, context information is information that is descriptive of the display object in which the text appears. This descriptive information can include parameters such as the type of display object (dialog box, menu, window, etc.) and its current state (active/inactive). Context information also includes system information such as the API calls and/or function calls made by the target application to render the display object to a user interface, the object handle or resource ID, the specific location of files called during execution of the display object, and any other information that provides a general context for the text that is displayed to the [0026] user interface screen 191 during the execution of the target process or application. Context information is obtained by intercepting the system messages that are passed between the target application and the operating system during the execution of the application. Capturing the text in connection with the operating system messages provides a general context for the text, and allows the information to be associated with a specific task or process within the target application.
  • The [0027] computing system environment 100 includes a general purpose computing device in the form of a computer 110. Components of computer 110 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
  • [0028] Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
  • The [0029] system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation, FIG. 1 illustrates operating system 134, application programs 135, other program modules 136, and program data 137.
  • The [0030] computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive 140 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152, and an optical disk drive 155 that reads from or writes to a removable, nonvolatile optical disk 156 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140, and magnetic disk drive 151 and optical disk drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150.
  • The drives and their associated computer storage media discussed above and illustrated in FIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the [0031] computer 110. In FIG. 1, for example, hard disk drive 141 is illustrated as storing operating system 144, application programs 145, other program modules 146, and program data 147. Note that these components can either be the same as or different from operating system 134, application programs 135, other program modules 136, and program data 137. Operating system 144, application programs 145, other program modules 146, and program data 147 are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 20 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 190.
  • The [0032] computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 110, although only a memory storage device 181 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
  • When used in a LAN networking environment, the [0033] personal computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the personal computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs 185 as residing on memory device 181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
  • With reference now to FIG. 2, a [0034] target application 202 with a display object 204 is shown. The display object is spawned as a result of an invoked action within the target application, such as from clicking one of the menu items 212 with the mouse 161. Operating systems that support GUIs such as Windows, contain numerous types of display objects that are presented to a user interface 200, including user buttons 206, menu items 210 and dialogue boxes 204. In FIG. 2, the display object is a dialogue box 204 consisting of three user option buttons 206, where one of them 210 is owner drawn. The dialogue box 204 also consists of a text field 208 for displaying text characters, having defined font properties to the GUI. All of the buttons 206 shown within the dialogue box and the associated text field 208 are child controls of the dialogue box. A child control can be best described as a display object that is initiated by a parent object. In the illustration, the dialogue box 204 is the parent object, and has four children associated with it. Each of the objects has its own handle, which is a unique identification number assigned by the operating system 144 that distinguishes one object from another. The handles of the various display objects are also indicated in the resource file associated with the executable application. Hence, the dialogue box has its own unique handle, and each of its children has its own handle, which in many operating systems is inherited directly from the parent object.
  • It is common practice amongst software developers to obtain information related to a particular display object from the resource file, or by accessing the handle of the object directly. Once the handle of the object is known, the developer can view the graphics primitives and associated attributes of the object directly from the resource file. Examples of information that can be viewed include the various controls within the display objects, the placement of the object within the target application or parent window, the line properties, etc. However, while the graphics primitives of a display object may be obtained in this way, it is not possible to ascertain the font properties of display objects directly. This is because the font properties, which are attributes of a text primitive, are maintained within a temporary data structure provided by a device context. This data structure is not indicated within the resource file, and not maintained within memory after the text it relates to is drawn to the screen. Therefore, the only way to access the font properties of a text string or character is to instantly capture the font information from the data structure as it is drawn to the graphical user interface. The invention allows the font properties to be captured during runtime of the application. [0035]
  • Specifically, with reference to FIG. 3 and the flowchart of FIG. 4, a [0036] calling process 300 injects DLLs into a target application 302 to “spy” on or monitor the operating system messages generated during target application execution. This technique is useful for determining the internal state of an application as it is in operation. A spy component 308 is used to perform the spying. It is an executable program module, such as a DLL, consisting of various functions and data for monitoring an application or process as it is in execution. The spy component 308 is injected into the target process or application 302 via an injection component 306 (event 406). This injection component 306 is invoked by the calling process 300 (event 404), which is an executable program module, such as a DLL, having executable instructions for injecting source code and/or program modules into a target process 302. Once the spy component is injected 310, it installs function patches into the operating system's application programming interfaces (APIs) that have executable instructions and routines for outputting graphics primitives to a user interface (event 410). For example, the primary API for Windows, known as Win32, provides a complete set of functions to format and draw text primitives in a target application's 302 user interface 200 (e.g.). These functions, such as DrawText( ), TextOut( ), and PolyTextOut( ) are responsible for outputting individual characters, symbols, or entire strings of text to the screen. Within the Windows environment, these are the functions that are patched by the spy component 308 after injection (event 410). The function patches serve the specific purpose of capturing the text that is rendered to the user interface screen 191 during the execution of the target process or application 302.
  • In addition to installing the function patches, the spy component installs one or more hook functions into the operating system's APIs to monitor system messages passed during execution of the target application/process [0037] 302 (event 412). Hook functions intercept system messages that are passed between the operating system and system threads during process execution. Whenever an action such as the pressing of a function key, click of a mouse, or activation of a dialogue box occurs, the OS generates a message that passes through a chain of hook procedures before reaching the target process. Standard hook functions are utilized to monitor the system messages that pertain specifically to the output of display objects to a user interface screen 191. In the Windows operating system for example, the WH_MSGFILTER and WH_SYSMSGFILTER hooks are used to monitor messages that are to be processed by a menu, scroll bar, message box, or dialog box.
  • The hook functions are installed and uninstalled accordingly by a [0038] hook management component 304, which is called upon by the spy component after it is injected 310 into the target process. The hook management component is a separate process from the calling process and target process, and has executable instructions for installing and uninstalling hook functions within a process or application designated by the spy component. In accordance with the invention, the designated process is the target process or application. Because the hook management component operates as an independent process, one or more calling processes can use the hook management component to install hooks within one or more target processes. In this way, the system messages of multiple independently running processes or applications can be monitored during runtime. As soon as an action is invoked within the target process 302 that results in the invocation of a display object, the generated message is captured by the one or more hook functions (event 414). This action results in the release of a trigger, or flag, that activates the function patches to capture the graphics primitives that are drawn to the screen to render the display object (event 416).
  • After enabling the function patches, the system messages captured by the one or more hook functions are then used to invalidate the display object under execution (event [0039] 418). The invalidation of a display object is a process whereby a selected or active display object is redrawn due to a user or system invoked change to the object. For instance, a display object is invalidated each time a user resizes the display object or moves it to a different position within the user interface 200. When such an action occurs, the display object is redrawn by calling the same API functions and routines that rendered it to the interface screen the first time. By using the system messages (which provide a context for the captured graphics primitives) captured by the hook functions, the appropriate API function calls are made, which in turn enumerates the display objects to the screen. As the display box is redrawn/invalidated, the installed function patches capture the graphics primitives, such as the text and other drawing elements that are drawn to the interface screen 200 (event 420). Thus, the graphics primitives and associated attributes of the graphics primitives related to the display object are captured in association with the context information provided by the system messages.
  • Finally, after the hook functions and function patches capture the graphics primitives including the font, and context information accordingly, this information is packaged into a data structure and sent to the [0040] calling process 300 as a system message (event 422 and 424). The process of capturing the graphics primitives and context information is continued for each invoked action within the target process, until execution is terminated. Because the hook functions and patches capture the runtime resource information on demand, the calling process receives this packaged information in an enumerated format.
  • FIG. 5 is illustrative of the data structure containing the captured graphics primitives, associated attributes and context information related to the [0041] dialogue box 204 of FIG. 2. While this information is captured and retrieved in a similar format as shown in the figure, those skilled in the art will recognize that the actual data, and format of the data is dependent upon the type of action invoked within the target process. Moreover, the information contained within the data structure will differ from one type of display object to another, as each object can consist of various buttons, text fields, and other drawing elements. Therefore, the information that is captured and stored within the data structure will vary from one process to another.
  • In FIG. 5, the captured information includes data and parameters that are descriptive of the [0042] display object 204 both functionally and graphically. The object type as shown is a DIALOGUE_BOX 500, which further consists of other data and parameters that define its appearance and operation. These parameters include the object's four child controls, namely TEXT_FIELD 502, OWNER_DRAWN BUTTON 504, BUTTON 1 506, and BUTTON 2 508 (these names were assigned for illustrative purposes only). As shown in FIG. 2, each of the controls is a user button, and one of the buttons is owner drawn 210. Each button is a display object that is composed of one or more graphics primitives, such as text characters and lines that define its shape. Likewise, the primitives have unique attributes, such as font type, text size, line length and other stylistic characteristics. The sequential format of the information as shown in FIG. 5 is due to the function patches that capture the graphics primitives and associated attributes. As the display objects of the target application are enumerated to the screen, the graphics primitives and attributes of such primitives are instantly captured, stored in the data structure in the order in which the various attributes are drawn to the screen, and returned to the calling process. The ability to capture the graphics primitives and associated attributes of the graphics primitives in connection with the operating system messages passed during process execution (context information), results in the return of complete graphic information to the calling process.
  • The invention as described herein can be incorporated into the source code of a calling process directly, or called upon by the calling process to capture and retrieve the graphics primitives and associated context information of a display object. The calling process can be any computer executable application where the capture of the various graphics primitives that are output to a user interface by a target process is required. Once captured, the calling process can process this information accordingly. Examples of applications that can practice the capturing techniques disclosed include, but are not limited to localization tools, language processing applications and text-to-speech applications. Also, while the methods disclosed are applicable to various operating systems and platforms, the ability to capture graphics primitives and associated attributes in connection with context information can be particularly useful within the Windows OS to better support Active Accessibility applications. These types of applications are commonly used to make computer applications accommodating for people with physical disabilities, such as blindness or restricted mobility. By integrating the ability to capture graphics primitives and attributes of such primitives from a target process, Active Accessibility can be better supported within applications having display objects that are not native to the operating system itself (e.g. owner-draw controls). Indeed, the invention can be practiced in any system that requires or desires instant access to typographical or visual information from any display object that can be rendered to a user interface. [0043]
  • In this description, the invention is described with reference to acts and symbolic representations of operations that are performed by one or more computers such as the [0044] computer 110, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processing unit of the computer of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the computer in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while the invention is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that the various acts and operations described hereinafter may also be implemented in hardware.
  • All of the references cited herein, including patents, patent applications, and publications, are hereby incorporated in their entireties by reference. In view of the many possible embodiments to which the principles of this invention may be applied, it should be recognized that the embodiment described herein with respect to the drawing figures is meant to be illustrative only and should not be taken as limiting the scope of invention. For example, those having skill in the art will recognize that the elements of the illustrated embodiment shown in software may be implemented in hardware and vice versa or that the illustrated embodiment can be modified in arrangement and detail without departing from the spirit of the invention. Therefore, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof. [0045]

Claims (31)

We claim:
1. A method for capturing one or more graphic primitives and attributes associated with such graphic primitives of a display object, the method comprising the steps of:
injecting a spy component into a target process by a calling process, wherein the spy component is an executable program module;
capturing the one or more graphic primitives and attributes associated with such graphic primitives during the execution of the target process; and
returning the graphic primitives and attributes associated with such graphic primitives to the calling process.
2. The method of claim 1, wherein the step of injecting the spy component is performed by an injection component that is invoked by the calling process.
3. The method of claim 2, wherein the injection component is an executable program module comprising executable routines for injecting source code into a target process.
4. The method of claim 1, wherein the step of injecting includes inserting one or more function patches into one or more executable program modules that correspond to an operating system upon which the target process is being executed, the executable program modules having instructions for rendering graphics primitives to a graphical user interface.
5. The method of claim 4, wherein the step of inserting the function patches is performed by the spy component.
6. The method of claim 1, wherein the step of injecting includes installing one or more hook functions into the operating system APIs that generate system messages in the event of a display object being output to a user interface screen during the execution of the target process.
7. The method of claim 6, wherein the step of installing is performed by the spy component.
8. The method of claim 6, wherein the system messages provide context information that is descriptive of an invoked action within the target process.
9. The method of claim 6, wherein the one or more hook functions are installed by a hook management component that is called upon by the spy component after injection into the target process.
10. The method of claim 9, wherein the hook management component is responsible for uninstalling the one or more hook functions upon termination of the target process.
11. The method of claim 1, wherein the step of capturing includes calling the one or more hook functions to intercept any system messages generated as a result of an invoked action within the target process.
12. The method of claim 11, wherein the one or more hook functions set a flag to activate the one or more function patches installed by the spy component.
13. The method of claim 1, wherein the step of capturing includes invalidating a display object that is output to a user interface as a result of the invoked action within the target process.
14. The method of claim 13, wherein the step of invalidating includes calling the function patches to capture the graphics primitives and attributes associated with such graphics primitives as they are drawn to the user interface to render the display object.
15. The method of claim 1, wherein the step of returning includes sending the context information captured by the one or more hook functions to the calling process as a system message.
16. The method of claim 1, wherein the step of returning further includes sending the graphics primitives and attributes associated with such graphics primitives that are captured by the one or more function patches to the calling process as a system message.
17. A computer-readable medium having computer-executable instructions for capturing one or more graphic primitives and attributes associated with such graphic primitives of a display object, the computer-executable instructions performing steps comprising:
injecting a spy component into a target process by a calling process, wherein the spy component is an executable program module;
capturing the one or more graphic primitives and attributes associated with such graphic primitives during the execution of the target process; and
returning the graphic primitives and attributes associated with such graphic primitives to the calling process.
18. A system for capturing one or more graphic primitives and attributes associated with such graphic primitives of a display object, the system comprising:
an injection component for injecting a spy component into a target process residing on a computer;
a spy component for capturing graphics primitives and attributes associated with such graphics primitives in connection with system messages that are generated by the target process as a result of an invoked action within the target process; and
a hook management component for installing and uninstalling one or more hook functions into one or more program modules that are executed by an operating system residing on the computer, the program modules having instructions for generating system messages during the execution of the target process.
19. The system of claim 18, wherein the injection component is an executable program module consisting of executable instructions for injecting the spy component into the executable code of the target process.
20. The system of claim 18, wherein the injection component injects the spy component into the target process on behalf of a calling process.
21. The system of claim 20, wherein the calling process is a computer executable application.
22. The system of claim 18, wherein the spy component inserts one or more function patches into one or more executable program modules that correspond to the operating system upon which the target process is being executed, the executable program modules having instructions for rendering graphics primitives to a graphical user interface.
23. The system of claim 22, wherein the function patches capture graphics primitives and associated attributes of such graphics primitives that are rendered to a user interface by a display object as a result of an action invoked within the target process.
24. The system of claim 23, wherein the spy component packages the graphics primitives and attributes associated with such graphics primitives and sends it to the calling process as a system message.
25. The system of claim 18, wherein the spy component calls the hook management component to insert one or more hook functions into one or more executable program modules that correspond to the operating system upon which the target process is being executed, the executable program modules having instructions for generating system messages in the event of a display object being output to a user interface during the execution of the target process.
26. The system of claim 25, wherein the one or more hook functions set a flag to activate the one or more function patches installed by the spy component.
27. The system of claim 25, wherein the one or more hook functions intercept any system messages generated as a result of an invoked action within the target process.
28. The method of claim 27, wherein the system messages contain context information that is descriptive of an invoked action within the target process.
29. The system of claim 18, wherein the spy component packages the context information and sends it to the calling process as an OS message.
30. The system of claim 18, wherein the hook management component is responsible for installing and uninstalling hook functions on behalf of the spy component.
31. The system of claim 30, wherein the hook functions are uninstalled by the hook management component upon termination of the target process.
US09/892,989 2001-06-27 2001-06-27 Capturing graphics primitives associated with any display object rendered to a graphical user interface Expired - Lifetime US6802055B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US09/892,989 US6802055B2 (en) 2001-06-27 2001-06-27 Capturing graphics primitives associated with any display object rendered to a graphical user interface

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US09/892,989 US6802055B2 (en) 2001-06-27 2001-06-27 Capturing graphics primitives associated with any display object rendered to a graphical user interface

Publications (2)

Publication Number Publication Date
US20030001854A1 true US20030001854A1 (en) 2003-01-02
US6802055B2 US6802055B2 (en) 2004-10-05

Family

ID=25400842

Family Applications (1)

Application Number Title Priority Date Filing Date
US09/892,989 Expired - Lifetime US6802055B2 (en) 2001-06-27 2001-06-27 Capturing graphics primitives associated with any display object rendered to a graphical user interface

Country Status (1)

Country Link
US (1) US6802055B2 (en)

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020083217A1 (en) * 1997-07-25 2002-06-27 Ward Alan D. System and method asynchronously accessing a graphics system for graphics application evaluation and control
US20050149498A1 (en) * 2003-12-31 2005-07-07 Stephen Lawrence Methods and systems for improving a search ranking using article information
US20050223061A1 (en) * 2004-03-31 2005-10-06 Auerbach David B Methods and systems for processing email messages
US20050234875A1 (en) * 2004-03-31 2005-10-20 Auerbach David B Methods and systems for processing media files
US20050246588A1 (en) * 2004-03-31 2005-11-03 Google, Inc. Profile based capture component
US20060123414A1 (en) * 2004-12-03 2006-06-08 International Business Machines Corporation Method and apparatus for creation of customized install packages for installation of software
US20060248401A1 (en) * 2005-04-15 2006-11-02 Microsoft Corporation Method and apparatus for performance analysis on a software program
US20060294463A1 (en) * 2005-06-24 2006-12-28 International Business Machines Corporation Localizing a Java GUI Application Without Source Code Modification and Rebuilding
US20070005682A1 (en) * 2005-06-30 2007-01-04 Yuki Ishibashi Information processing apparatus, information processing system, message control method, and program product, storage medium
US20070126749A1 (en) * 2005-12-01 2007-06-07 Exent Technologies, Ltd. System, method and computer program product for dynamically identifying, selecting and extracting graphical and media objects in frames or scenes rendered by a software application
US20070132753A1 (en) * 2005-12-12 2007-06-14 Microsoft Corporation Alternative graphics pipe
US20070214461A1 (en) * 2005-06-08 2007-09-13 Logitech Europe S.A. System and method for transparently processing multimedia data
US20070296718A1 (en) * 2005-12-01 2007-12-27 Exent Technologies, Ltd. Dynamic resizing of graphics content rendered by an application to facilitate rendering of additional graphics content
US7333976B1 (en) 2004-03-31 2008-02-19 Google Inc. Methods and systems for processing contact information
US20080042923A1 (en) * 2006-08-16 2008-02-21 Rick De Laet Systems, methods, and apparatus for recording of graphical display
US7412708B1 (en) 2004-03-31 2008-08-12 Google Inc. Methods and systems for capturing information
US20090083753A1 (en) * 2007-09-25 2009-03-26 Exent Technologies, Ltd. Dynamic thread generation and management for improved computer program performance
US20090182753A1 (en) * 2008-01-15 2009-07-16 Microsoft Corporation Recording user-driven events withi a computing system
US20090189890A1 (en) * 2008-01-27 2009-07-30 Tim Corbett Methods and systems for improving resource utilization by delaying rendering of three dimensional graphics
US7581227B1 (en) 2004-03-31 2009-08-25 Google Inc. Systems and methods of synchronizing indexes
US7680888B1 (en) 2004-03-31 2010-03-16 Google Inc. Methods and systems for processing instant messenger messages
US7725508B2 (en) 2004-03-31 2010-05-25 Google Inc. Methods and systems for information capture and retrieval
US20110099632A1 (en) * 2005-07-15 2011-04-28 Microsoft Corporation Detecting user-mode rootkits
US20110225569A1 (en) * 2010-03-10 2011-09-15 International Business Machines Corporation Automated desktop benchmarking
US8161053B1 (en) 2004-03-31 2012-04-17 Google Inc. Methods and systems for eliminating duplicate events
US8346777B1 (en) 2004-03-31 2013-01-01 Google Inc. Systems and methods for selectively storing event data
US8386728B1 (en) 2004-03-31 2013-02-26 Google Inc. Methods and systems for prioritizing a crawl
US8631076B1 (en) 2004-03-31 2014-01-14 Google Inc. Methods and systems for associating instant messenger events
US20140171190A1 (en) * 2012-12-14 2014-06-19 Nvidia Corporation Implementing a remote gaming server on a desktop computer
US8892495B2 (en) 1991-12-23 2014-11-18 Blanding Hovenweep, Llc Adaptive pattern recognition based controller apparatus and method and human-interface therefore
US8954420B1 (en) 2003-12-31 2015-02-10 Google Inc. Methods and systems for improving a search ranking using article information
CN104636132A (en) * 2014-12-23 2015-05-20 用友软件股份有限公司 Application program operation method and system and application program
US9262446B1 (en) 2005-12-29 2016-02-16 Google Inc. Dynamically ranking entries in a personal data book
US20160283204A1 (en) * 2015-03-25 2016-09-29 Ca, Inc. Editing software products using text mapping files
US9535563B2 (en) 1999-02-01 2017-01-03 Blanding Hovenweep, Llc Internet appliance system and method
EP3312719A4 (en) * 2015-07-21 2018-07-04 ZTE Corporation Method and device for identifying java window control
US20190294422A1 (en) * 2018-03-22 2019-09-26 Sick Ag Method of Carrying Out Modifications to a Software Application
CN111198723A (en) * 2018-11-19 2020-05-26 深圳市优必选科技有限公司 Process injection method, terminal equipment and computer readable storage medium

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2003010684A1 (en) 2001-07-26 2003-02-06 Irise, Inc. System and process for gathering, recording and validating requirements for computer applications
US20040098703A1 (en) * 2002-11-15 2004-05-20 Kondur Sankar Narayana Integration of multiple software tools under a single site
JP4281414B2 (en) * 2003-05-22 2009-06-17 セイコーエプソン株式会社 Barcode or two-dimensional code printing system, barcode or two-dimensional code font setting method, and barcode or two-dimensional code font setting program
US7788606B2 (en) * 2004-06-14 2010-08-31 Sas Institute Inc. Computer-implemented system and method for defining graphics primitives
US20060156397A1 (en) * 2005-01-13 2006-07-13 Steven Dai A New Anti-spy method without using scan
CN100489866C (en) * 2005-09-30 2009-05-20 鸿富锦精密工业(深圳)有限公司 Workpiece discriminating system and method
US7596536B2 (en) * 2005-12-01 2009-09-29 Exent Technologies, Ltd. System, method and computer program product for dynamically measuring properties of objects rendered and/or referenced by an application executing on a computing device
US20070168309A1 (en) * 2005-12-01 2007-07-19 Exent Technologies, Ltd. System, method and computer program product for dynamically extracting and sharing event information from an executing software application
US7596540B2 (en) * 2005-12-01 2009-09-29 Exent Technologies, Ltd. System, method and computer program product for dynamically enhancing an application executing on a computing device
US7975259B2 (en) * 2007-06-01 2011-07-05 Sap Ag Verification of customization results
US20080319777A1 (en) * 2007-06-20 2008-12-25 Roland Hoff Business transaction issue manager
US9472167B2 (en) * 2009-04-17 2016-10-18 International Business Machines Corporation Video capture through hardware
US8413117B1 (en) * 2009-08-07 2013-04-02 Symantec Corporation Systems and methods for focusing product testing based on areas of change within the product between product builds
EP2625606A4 (en) 2010-10-08 2014-11-26 Irise System and method for extending a visualization platform
US9235783B1 (en) 2012-04-20 2016-01-12 The Boeing Company Highlighting an object in a display using a highlight object
US9214136B1 (en) * 2012-04-26 2015-12-15 The Boeing Company Highlighting an object in a display using a dynamically generated highlight object

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6064381A (en) * 1996-12-03 2000-05-16 Ergolight Ltd. Apparatus and methods for analyzing software systems
US6091422A (en) * 1998-04-03 2000-07-18 Avid Technology, Inc. System for editing complex visual data providing a continuously updated rendering
US6314470B1 (en) * 1997-07-25 2001-11-06 Hewlett Packard Company System and method for asynchronously accessing a graphics system for graphics application evaluation and control

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6064381A (en) * 1996-12-03 2000-05-16 Ergolight Ltd. Apparatus and methods for analyzing software systems
US6314470B1 (en) * 1997-07-25 2001-11-06 Hewlett Packard Company System and method for asynchronously accessing a graphics system for graphics application evaluation and control
US6091422A (en) * 1998-04-03 2000-07-18 Avid Technology, Inc. System for editing complex visual data providing a continuously updated rendering

Cited By (81)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8892495B2 (en) 1991-12-23 2014-11-18 Blanding Hovenweep, Llc Adaptive pattern recognition based controller apparatus and method and human-interface therefore
US20020083217A1 (en) * 1997-07-25 2002-06-27 Ward Alan D. System and method asynchronously accessing a graphics system for graphics application evaluation and control
US7028309B2 (en) * 1997-07-25 2006-04-11 Hewlett-Packard Development Company, L.P. Accessing a graphics system for graphics application evaluation and control
US9535563B2 (en) 1999-02-01 2017-01-03 Blanding Hovenweep, Llc Internet appliance system and method
US20050149498A1 (en) * 2003-12-31 2005-07-07 Stephen Lawrence Methods and systems for improving a search ranking using article information
US10423679B2 (en) 2003-12-31 2019-09-24 Google Llc Methods and systems for improving a search ranking using article information
US8954420B1 (en) 2003-12-31 2015-02-10 Google Inc. Methods and systems for improving a search ranking using article information
US8812515B1 (en) 2004-03-31 2014-08-19 Google Inc. Processing contact information
US7680809B2 (en) * 2004-03-31 2010-03-16 Google Inc. Profile based capture component
US8631076B1 (en) 2004-03-31 2014-01-14 Google Inc. Methods and systems for associating instant messenger events
US8275839B2 (en) 2004-03-31 2012-09-25 Google Inc. Methods and systems for processing email messages
EP1741025A2 (en) * 2004-03-31 2007-01-10 Google, Inc. Methods and systems for processing media files
US20050246588A1 (en) * 2004-03-31 2005-11-03 Google, Inc. Profile based capture component
US8386728B1 (en) 2004-03-31 2013-02-26 Google Inc. Methods and systems for prioritizing a crawl
US8346777B1 (en) 2004-03-31 2013-01-01 Google Inc. Systems and methods for selectively storing event data
WO2005103900A1 (en) 2004-03-31 2005-11-03 Google, Inc. Profile based capture component for monitoring events in applications
US9189553B2 (en) 2004-03-31 2015-11-17 Google Inc. Methods and systems for prioritizing a crawl
US7333976B1 (en) 2004-03-31 2008-02-19 Google Inc. Methods and systems for processing contact information
US8099407B2 (en) 2004-03-31 2012-01-17 Google Inc. Methods and systems for processing media files
US7412708B1 (en) 2004-03-31 2008-08-12 Google Inc. Methods and systems for capturing information
US9311408B2 (en) 2004-03-31 2016-04-12 Google, Inc. Methods and systems for processing media files
CN100428182C (en) * 2004-03-31 2008-10-22 咕果公司 Profile based capture component for monitoring events in applications
US7941439B1 (en) * 2004-03-31 2011-05-10 Google Inc. Methods and systems for information capture
EP1741025A4 (en) * 2004-03-31 2009-07-01 Google Inc Methods and systems for processing media files
US20050234875A1 (en) * 2004-03-31 2005-10-20 Auerbach David B Methods and systems for processing media files
US8161053B1 (en) 2004-03-31 2012-04-17 Google Inc. Methods and systems for eliminating duplicate events
US9836544B2 (en) 2004-03-31 2017-12-05 Google Inc. Methods and systems for prioritizing a crawl
US20050223061A1 (en) * 2004-03-31 2005-10-06 Auerbach David B Methods and systems for processing email messages
US10180980B2 (en) 2004-03-31 2019-01-15 Google Llc Methods and systems for eliminating duplicate events
US7581227B1 (en) 2004-03-31 2009-08-25 Google Inc. Systems and methods of synchronizing indexes
US7725508B2 (en) 2004-03-31 2010-05-25 Google Inc. Methods and systems for information capture and retrieval
US7680888B1 (en) 2004-03-31 2010-03-16 Google Inc. Methods and systems for processing instant messenger messages
KR100952379B1 (en) 2004-03-31 2010-04-14 구글 인코포레이티드 Profile based capture component for monitoring events in applications
US20060123414A1 (en) * 2004-12-03 2006-06-08 International Business Machines Corporation Method and apparatus for creation of customized install packages for installation of software
US10162618B2 (en) 2004-12-03 2018-12-25 International Business Machines Corporation Method and apparatus for creation of customized install packages for installation of software
WO2006113111A3 (en) * 2005-04-15 2007-11-29 Microsoft Corp Method and apparatus for performance analysis on a software program
US20060248401A1 (en) * 2005-04-15 2006-11-02 Microsoft Corporation Method and apparatus for performance analysis on a software program
US7698686B2 (en) * 2005-04-15 2010-04-13 Microsoft Corporation Method and apparatus for performance analysis on a software program
JP2008537222A (en) * 2005-04-15 2008-09-11 マイクロソフト コーポレーション Method and apparatus for performance analysis of software programs
US8606950B2 (en) * 2005-06-08 2013-12-10 Logitech Europe S.A. System and method for transparently processing multimedia data
US20070214461A1 (en) * 2005-06-08 2007-09-13 Logitech Europe S.A. System and method for transparently processing multimedia data
US8645928B2 (en) * 2005-06-24 2014-02-04 International Business Machines Corporation Localizing a java GUI application without source code modification and rebuilding
US20060294463A1 (en) * 2005-06-24 2006-12-28 International Business Machines Corporation Localizing a Java GUI Application Without Source Code Modification and Rebuilding
US20070005682A1 (en) * 2005-06-30 2007-01-04 Yuki Ishibashi Information processing apparatus, information processing system, message control method, and program product, storage medium
US9037744B2 (en) * 2005-06-30 2015-05-19 Ricoh Company, Ltd. Information processing apparatus, information processing system, message control method, and program product, storage medium
US8661541B2 (en) * 2005-07-15 2014-02-25 Microsoft Corporation Detecting user-mode rootkits
US20110099632A1 (en) * 2005-07-15 2011-04-28 Microsoft Corporation Detecting user-mode rootkits
US8629885B2 (en) 2005-12-01 2014-01-14 Exent Technologies, Ltd. System, method and computer program product for dynamically identifying, selecting and extracting graphical and media objects in frames or scenes rendered by a software application
US20070126749A1 (en) * 2005-12-01 2007-06-07 Exent Technologies, Ltd. System, method and computer program product for dynamically identifying, selecting and extracting graphical and media objects in frames or scenes rendered by a software application
US20070296718A1 (en) * 2005-12-01 2007-12-27 Exent Technologies, Ltd. Dynamic resizing of graphics content rendered by an application to facilitate rendering of additional graphics content
US20070132753A1 (en) * 2005-12-12 2007-06-14 Microsoft Corporation Alternative graphics pipe
US7773096B2 (en) * 2005-12-12 2010-08-10 Microsoft Corporation Alternative graphics pipe
US9262446B1 (en) 2005-12-29 2016-02-16 Google Inc. Dynamically ranking entries in a personal data book
US20080042923A1 (en) * 2006-08-16 2008-02-21 Rick De Laet Systems, methods, and apparatus for recording of graphical display
US8878833B2 (en) 2006-08-16 2014-11-04 Barco, Inc. Systems, methods, and apparatus for recording of graphical display
US20090083753A1 (en) * 2007-09-25 2009-03-26 Exent Technologies, Ltd. Dynamic thread generation and management for improved computer program performance
US8448138B2 (en) * 2008-01-15 2013-05-21 Microsoft Corporation Recording user-driven events within a computing system
US20090182753A1 (en) * 2008-01-15 2009-07-16 Microsoft Corporation Recording user-driven events withi a computing system
WO2009094673A2 (en) * 2008-01-27 2009-07-30 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
WO2009094673A3 (en) * 2008-01-27 2009-10-01 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
US8665265B2 (en) 2008-01-27 2014-03-04 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
US20090189890A1 (en) * 2008-01-27 2009-07-30 Tim Corbett Methods and systems for improving resource utilization by delaying rendering of three dimensional graphics
US8405654B2 (en) 2008-01-27 2013-03-26 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
US8350863B2 (en) 2008-01-27 2013-01-08 Citrix Systems, Inc. Methods and systems for improving resource utilization by delaying rendering of three dimensional graphics
US20090189892A1 (en) * 2008-01-27 2009-07-30 Nitin Desai Methods and systems for detecting a dirty region within a frame encompassing three dimensional graphics
US20090189893A1 (en) * 2008-01-27 2009-07-30 Petrov Julian Methods and systems for computing a hash from a three dimensional data set loaded into a resource
US8169436B2 (en) 2008-01-27 2012-05-01 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
EP2315124A3 (en) * 2008-01-27 2011-05-04 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
EP2293192A3 (en) * 2008-01-27 2011-05-04 Citrix Systems, Inc. Methods and systems for remoting three dimensional graphics
US20110225569A1 (en) * 2010-03-10 2011-09-15 International Business Machines Corporation Automated desktop benchmarking
US8495584B2 (en) 2010-03-10 2013-07-23 International Business Machines Corporation Automated desktop benchmarking
US10118095B2 (en) * 2012-12-14 2018-11-06 Nvidia Corporation Implementing a remote gaming server on a desktop computer
US20140171190A1 (en) * 2012-12-14 2014-06-19 Nvidia Corporation Implementing a remote gaming server on a desktop computer
CN104636132A (en) * 2014-12-23 2015-05-20 用友软件股份有限公司 Application program operation method and system and application program
US20160283204A1 (en) * 2015-03-25 2016-09-29 Ca, Inc. Editing software products using text mapping files
US9690549B2 (en) * 2015-03-25 2017-06-27 Ca, Inc. Editing software products using text mapping files
EP3312719A4 (en) * 2015-07-21 2018-07-04 ZTE Corporation Method and device for identifying java window control
US20190294422A1 (en) * 2018-03-22 2019-09-26 Sick Ag Method of Carrying Out Modifications to a Software Application
EP3543844B1 (en) * 2018-03-22 2020-04-29 Sick Ag Method for carrying out changes to a software application
US10949175B2 (en) * 2018-03-22 2021-03-16 Sick Ag Method of carrying out modifications to a software application
CN111198723A (en) * 2018-11-19 2020-05-26 深圳市优必选科技有限公司 Process injection method, terminal equipment and computer readable storage medium

Also Published As

Publication number Publication date
US6802055B2 (en) 2004-10-05

Similar Documents

Publication Publication Date Title
US6802055B2 (en) Capturing graphics primitives associated with any display object rendered to a graphical user interface
RU2371758C2 (en) Programming interface for computer platform
US10242181B2 (en) Isolating applications hosted by plug-in code
US6795089B2 (en) Dynamic, live surface and model elements for visualization and modeling
US7568165B2 (en) Sidebar engine, object model and schema
US7657844B2 (en) Providing accessibility compliance within advanced componentry
US6957439B1 (en) Method, system, and program for mapping objects in different language formats
US6567973B1 (en) Introspective editor system, program, and method for software translation using a facade class
US6275790B1 (en) Introspective editor system, program, and method for software translation
US6342907B1 (en) Specification language for defining user interface panels that are platform-independent
US7290245B2 (en) Methods and systems for navigating deterministically through a graphical user interface
US6976059B1 (en) System and method to provide applets using a server based virtual machine
US8904012B2 (en) Client-server input method editor architecture
US7506273B2 (en) Method and system for modifying properties of graphical user interface components
US7908560B2 (en) Method and system for cross-screen component communication in dynamically created composite applications
US7568196B2 (en) Initializing virtual machine that subsequently executes application
US20040205663A1 (en) Undo/redo architecture across multiple files
US6941520B1 (en) Method, system, and program for using a user interface program to generate a user interface for an application program
US20170277664A1 (en) Graphical data presented in code editor along with code
US8302070B2 (en) Output styling in an IDE console
US7698636B2 (en) System and method for in-context editing of components
US20030001892A1 (en) Property editor graphical user interface apparatus, method and computer program product
US7966562B1 (en) System and method for providing domain-sensitive help
JP2001521217A (en) A system for linking control to publications using a tag and drop interface
IL161651A (en) Enhancing resource adapters

Legal Events

Date Code Title Description
AS Assignment

Owner name: MICROSOFT CORPORATION, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JADE, SHASHIDHAR P.;THOMPSON, PETER A.;REEL/FRAME:011941/0391

Effective date: 20010419

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034541/0001

Effective date: 20141014

FPAY Fee payment

Year of fee payment: 12