CA2304066A1 - Method and system for interactively developing a graphical control-flow and associated application software for machine vision system - Google Patents
Method and system for interactively developing a graphical control-flow and associated application software for machine vision system Download PDFInfo
- Publication number
- CA2304066A1 CA2304066A1 CA002304066A CA2304066A CA2304066A1 CA 2304066 A1 CA2304066 A1 CA 2304066A1 CA 002304066 A CA002304066 A CA 002304066A CA 2304066 A CA2304066 A CA 2304066A CA 2304066 A1 CA2304066 A1 CA 2304066A1
- Authority
- CA
- Canada
- Prior art keywords
- machine vision
- control
- computer system
- operating parameters
- algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/20—Pc systems
- G05B2219/23—Pc programming
- G05B2219/23257—Grafcet
Abstract
A method and a system (26) as shown in the figure, are provided for interactively developing a graphical, control-flow structure and associated application software for use in a machine vision system (20) utilising a computer system (28), without the need for a user to write any code. The structure includes a control sequence having a plurality of steps or nodes, transitions, and links or flow lines which interconnect the nodes and transactions. The hardware operating parameters define a set of standard controls having properties. Graphical representations or icons are selected which correspond to the desired second control programs and inputs and outputs of the nodes. The first control program is linked to the second control program to form the application software. The step of linking includes the step of setting property of one standard control to be equal to one property of another standard control to form the application software in response to the commands without the user writing any of the application software.
Description
METHOD AND SYSTEM FOR INTERACTIVELY DEVELOPING A GRAPHICAL
CONTROL-FLOW AND ASSOCIATED APPLICATION SOFTWARE FOR MACHINE
VISION SYSTEM
Technical Field This invention relates to methods and systems for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system.
1 o Background Art Traditionally, vision tools are programmed together within a programming environment such as C++, Delphi, or Visual Basic wherein a set of tools can be created, used and configured together. Although some environments are easier to program than others (e. g.
Visual Basic is easier than C++), they still require knowledge of that programming environment.
The U.S. patent to Silver et al. (5,481,712) discloses a method and system for interactively generat-ing a computer program in "C" for machine vision analy-sis wherein the program only permits the operator (i.e.
programmer) to make only syntactically correct modifica-tions to the program. A menu element displays permissi-ble programming modifications for the location of interest. The menu element incorporates in its display of permissible programming modifications statements for machine vision analysis of an object image such as calls to machine vision, sub-routines and functions. The computer program is programmed via a closed or propri-etary package via predefined tools and operations.
The U.S. patent to Lipkis et al. (4,914,567) provides for a design system using visual language. The system uses a visual user presentation of a control system, including structure, specification and opera-tion. The visual element enables a user to interactive-ly design, modify and explore operating characteristics.
A Grafcet-based graphic control-flow language for setting and testing state variables is disclosed.
Additional features are: application-artificial vision systems; object-oriented applications platform; top-down design; and setting of variable values. The system links Grafcet and MAP technology.
The U.S. patent to Rumbaugh et al. (5,005,119) provides for user interactive control of computer programs and corresponding versions of input/output data flow. Programs for a CAD task and their respective data set requirements are visually displayed as a flowgraph with which the user interacts to select input data sets and initiate program executions.
The U.S. patent to Endo (5,327,350) provides for an interactive-type data input function for control-ling the operation of a machine tool by creating a machining program based on input data.
The U.S. patent to Fujita et al. (5,383,110) adds the feature of answering questions displayed on a display screen to generate a numerical control program.
The U.S. patent to Kahn (5,157,663) provides far an animated user interface for computer program creation, control and execution. The interface enables the user to create programs in an interactive visual manner. Animation is integrated with computer program ming to allow a user to visualize programming flow and operation.
The U.S. patent to Lane et al. (4,679,137) provides for a process control interface system for designer and operator. A dynamic menu element is used during the definition and selection of operating parame-ters used by a process control program. Each data structure includes value indicia defining the parameters of the data structures.
The U.S. patent to Iadipaolo et al.
(4,759,074) provides for a method for automatically inspecting parts utilizing machine vision and system utilizing same. The system is taught by a series of steps which features are to be inspected. Typical features, as well as tolerances, are defined by the programmer to develop reference data.
The U.S. patents to Hayward et al.
(5,574,828), Abdel-Malek (5,511,147), Kuwahara et al.
(4,833,624), Wright et al. (5,453,933), Spight (4,462,046), Gordon (5,314,055), each provide general teachings of user interactive programming techniques for robotic or CNC machine tool environments.
In the automation world (i.e. for PLC's), there exists open standards such as Grafcet and IEC 1131 to define the syntax for automation control.
CONTROL-FLOW AND ASSOCIATED APPLICATION SOFTWARE FOR MACHINE
VISION SYSTEM
Technical Field This invention relates to methods and systems for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system.
1 o Background Art Traditionally, vision tools are programmed together within a programming environment such as C++, Delphi, or Visual Basic wherein a set of tools can be created, used and configured together. Although some environments are easier to program than others (e. g.
Visual Basic is easier than C++), they still require knowledge of that programming environment.
The U.S. patent to Silver et al. (5,481,712) discloses a method and system for interactively generat-ing a computer program in "C" for machine vision analy-sis wherein the program only permits the operator (i.e.
programmer) to make only syntactically correct modifica-tions to the program. A menu element displays permissi-ble programming modifications for the location of interest. The menu element incorporates in its display of permissible programming modifications statements for machine vision analysis of an object image such as calls to machine vision, sub-routines and functions. The computer program is programmed via a closed or propri-etary package via predefined tools and operations.
The U.S. patent to Lipkis et al. (4,914,567) provides for a design system using visual language. The system uses a visual user presentation of a control system, including structure, specification and opera-tion. The visual element enables a user to interactive-ly design, modify and explore operating characteristics.
A Grafcet-based graphic control-flow language for setting and testing state variables is disclosed.
Additional features are: application-artificial vision systems; object-oriented applications platform; top-down design; and setting of variable values. The system links Grafcet and MAP technology.
The U.S. patent to Rumbaugh et al. (5,005,119) provides for user interactive control of computer programs and corresponding versions of input/output data flow. Programs for a CAD task and their respective data set requirements are visually displayed as a flowgraph with which the user interacts to select input data sets and initiate program executions.
The U.S. patent to Endo (5,327,350) provides for an interactive-type data input function for control-ling the operation of a machine tool by creating a machining program based on input data.
The U.S. patent to Fujita et al. (5,383,110) adds the feature of answering questions displayed on a display screen to generate a numerical control program.
The U.S. patent to Kahn (5,157,663) provides far an animated user interface for computer program creation, control and execution. The interface enables the user to create programs in an interactive visual manner. Animation is integrated with computer program ming to allow a user to visualize programming flow and operation.
The U.S. patent to Lane et al. (4,679,137) provides for a process control interface system for designer and operator. A dynamic menu element is used during the definition and selection of operating parame-ters used by a process control program. Each data structure includes value indicia defining the parameters of the data structures.
The U.S. patent to Iadipaolo et al.
(4,759,074) provides for a method for automatically inspecting parts utilizing machine vision and system utilizing same. The system is taught by a series of steps which features are to be inspected. Typical features, as well as tolerances, are defined by the programmer to develop reference data.
The U.S. patents to Hayward et al.
(5,574,828), Abdel-Malek (5,511,147), Kuwahara et al.
(4,833,624), Wright et al. (5,453,933), Spight (4,462,046), Gordon (5,314,055), each provide general teachings of user interactive programming techniques for robotic or CNC machine tool environments.
In the automation world (i.e. for PLC's), there exists open standards such as Grafcet and IEC 1131 to define the syntax for automation control.
~ummarv Of The Invention An object of the present invention is to provide a method and system for interactively developing application software for use in a machine vision system wherein the user teaches an imaging programming task without writing any code. Consequently, the user need not be a programmer. Typically, a programmed computer queries the user's intentions and guides the user's choices until the task is completely taught. In the process, the user does not write a single line of code, but rather sets variables that the machine vision tools require interactively. Certain tool's outputs typically will be another tool's input.
Another object of the present invention is to provide a method and system for interactively developing application software for use in a machine vision system wherein the application software may be developed in a hierarchy which allows bottom-up or top-down program development, thereby adapting to the user's thinking.
For example, to measure a distance a ruler tool requires two input geometries and, in turn, outputs a distance.
An edge tool may define an edge and feature find tools match a model to define a point. Both these can be the inputs to the ruler tool. The user may start top-down, i.e. he thinks measurement first and the system guides him to define the input geometries. Alternatively, the user may think bottom-up and define the geometries first and then decide to do a measurement between them.
Yet another object of the present invention is to provide a method and system for interactively devel-oping application software for use in a machine vision system wherein the method and system allow the easy integration of third party tools (by using standard controls, such as ActiveX controls). Not only can the standard properties of the controls be inputs or outputs to other tools, but also objects such as geometries can be used as inputs and outputs to other tools.
Yet still another object of the present invention is to provide a method and system for interac-tively developing application software for use in a machine vision system wherein, to represent the flow through the taught task, a modified Grafcet standard is utilized to accept standard controls and objects as part of the steps and transitions. In the Grafcet standard, transitions can access any control s properties or objects to make execution judgments.
In carrying out the above objects and other objects of the present invention, a method is provided for developing a graphical, control-flow structure and associated application software for use in a machine vision system utilizing a computer system. The struc-ture includes a control sequence having at least one node, at least one transition, and at least one link for interconnecting the at least one node and the at least one transition. The method includes the step of storing an application development program, including a first set of control programs representing possible components of a user interface for the machine vision system. The first set of control programs define a first set of standard controls. The method also includes the step of storing a second set of control programs representing possible machine vision algorithms for the machine vision system. The second set of control programs define a second set of standard controls. The method further includes the step of storing hardware operating parameters corresponding to possible hardware for use in the machine vision system. The hardware operating parameters define a third set of standard controls. The method further includes the step of displaying graphical representations of the possible components, the possible hardware and the possible machine vision algorithms.
Then, the method includes the step of receiving commands from a user of the computer system to select a first control program corresponding to a desired component of the user interface, desired hardware operating parame-ters corresponding to desired hardware and a machine vision graphical representation and its associated second control program corresponding to a desired machine vision algorithm. The method also includes the step of displaying the structure. The graphical repre-sentation is a node of the structure. The selected second control program is mapped into the structure at the node. The displayed structure also has at least one transition and at least one link for interconnecting the node and the at least one transition. Finally, the method includes the step of linking the first control program with the desired hardware operating parameters to the second control program to form the application software in response to the commands without the user writing any of the application software.
Further in carrying out the above objects and other objects of the present invention, a system is provided for carrying out the method steps.
The benef its accruing to the method and system of the present invention are numerous. For example, the method and system:
1) Interactively generate/"teach" a computer program.
2) The marriage of two standards ActiveX (i.e.
controls) and Grafcet and its derivative IEC
1131, i.e. the ActiveX standard and the Grafcet IEC 1131 have been linked together.
As a result of this, an ActiveX behavior is controlled via its methods. Its characteris-tics are controlled via setting its proper-ties. An ActiveX event can control the se-quencing in the Grafcet paradigm.
3) The Grafcet/IEC 1131 standard has now been redefined in that a transition can be either True (the default case), an event generated by some ActiveX control or a Boolean expression defined by an internal variable or properties of any ActiveX Component.
4) A step in the Grafcet standard is now either any third party ActiveX control method or a mathematical step that allows the definition of new internal variables.
5) The teach environment allows algorithms to be defined in a hierarchy that allows bottom-up or top-down teach approach.
6) Combine vision with control.
Another object of the present invention is to provide a method and system for interactively developing application software for use in a machine vision system wherein the application software may be developed in a hierarchy which allows bottom-up or top-down program development, thereby adapting to the user's thinking.
For example, to measure a distance a ruler tool requires two input geometries and, in turn, outputs a distance.
An edge tool may define an edge and feature find tools match a model to define a point. Both these can be the inputs to the ruler tool. The user may start top-down, i.e. he thinks measurement first and the system guides him to define the input geometries. Alternatively, the user may think bottom-up and define the geometries first and then decide to do a measurement between them.
Yet another object of the present invention is to provide a method and system for interactively devel-oping application software for use in a machine vision system wherein the method and system allow the easy integration of third party tools (by using standard controls, such as ActiveX controls). Not only can the standard properties of the controls be inputs or outputs to other tools, but also objects such as geometries can be used as inputs and outputs to other tools.
Yet still another object of the present invention is to provide a method and system for interac-tively developing application software for use in a machine vision system wherein, to represent the flow through the taught task, a modified Grafcet standard is utilized to accept standard controls and objects as part of the steps and transitions. In the Grafcet standard, transitions can access any control s properties or objects to make execution judgments.
In carrying out the above objects and other objects of the present invention, a method is provided for developing a graphical, control-flow structure and associated application software for use in a machine vision system utilizing a computer system. The struc-ture includes a control sequence having at least one node, at least one transition, and at least one link for interconnecting the at least one node and the at least one transition. The method includes the step of storing an application development program, including a first set of control programs representing possible components of a user interface for the machine vision system. The first set of control programs define a first set of standard controls. The method also includes the step of storing a second set of control programs representing possible machine vision algorithms for the machine vision system. The second set of control programs define a second set of standard controls. The method further includes the step of storing hardware operating parameters corresponding to possible hardware for use in the machine vision system. The hardware operating parameters define a third set of standard controls. The method further includes the step of displaying graphical representations of the possible components, the possible hardware and the possible machine vision algorithms.
Then, the method includes the step of receiving commands from a user of the computer system to select a first control program corresponding to a desired component of the user interface, desired hardware operating parame-ters corresponding to desired hardware and a machine vision graphical representation and its associated second control program corresponding to a desired machine vision algorithm. The method also includes the step of displaying the structure. The graphical repre-sentation is a node of the structure. The selected second control program is mapped into the structure at the node. The displayed structure also has at least one transition and at least one link for interconnecting the node and the at least one transition. Finally, the method includes the step of linking the first control program with the desired hardware operating parameters to the second control program to form the application software in response to the commands without the user writing any of the application software.
Further in carrying out the above objects and other objects of the present invention, a system is provided for carrying out the method steps.
The benef its accruing to the method and system of the present invention are numerous. For example, the method and system:
1) Interactively generate/"teach" a computer program.
2) The marriage of two standards ActiveX (i.e.
controls) and Grafcet and its derivative IEC
1131, i.e. the ActiveX standard and the Grafcet IEC 1131 have been linked together.
As a result of this, an ActiveX behavior is controlled via its methods. Its characteris-tics are controlled via setting its proper-ties. An ActiveX event can control the se-quencing in the Grafcet paradigm.
3) The Grafcet/IEC 1131 standard has now been redefined in that a transition can be either True (the default case), an event generated by some ActiveX control or a Boolean expression defined by an internal variable or properties of any ActiveX Component.
4) A step in the Grafcet standard is now either any third party ActiveX control method or a mathematical step that allows the definition of new internal variables.
5) The teach environment allows algorithms to be defined in a hierarchy that allows bottom-up or top-down teach approach.
6) Combine vision with control.
7) Allows the integration of any third party ActiveX controls.
The idea behind interactively generating a computer program is to ensure that the programmer cannot make syntax errors in producing his code. Hence, the environment guides the programmer via menus and appro-priate interface structure to produce code that is correct in syntax. The problem being addressed is that computer languages typically are tightly defined and intolerant to syntax errors.
_g_ The second idea is that the system operator does not need to be a skilled programmer. Rather, he can be an applications engineer who primarily under-stands the application and does not need to be a pro-s grammer. These days, automation engineers are used to using programming in ladder logic and Grafcet-type languages. Applications knowledgeable engineers tend to be familiar with these environments and the interactive environment being presented in this application is for these engineers.
The method and system of the present applica-tion is different from the examples mentioned above in that it takes a standard of Grafcet and modifies it to allow the integration of another standard, namely ActiveX controls which can be linked in.
Hence, a first unique feature of this inven-tion is that any third party tools in machine vision or automation or otherwise that are written as ActiveX
controls can be linked into this interactive environ-ment. To date, no interactive environment allows the interaction of third party components and definitely not the configuration of any ActiveX controls.
ActiveX controls are becoming widely accepted and many are being produced by many companies . These are able to be linked together and used without writing code that can produce syntax errors. A control sequence is set up triggered on events, timers and variable settings. This interactive modified Grafcet/IEC 1131 environment reads the properties of the ActiveX controls and allows these to be set or linked to other properties of other controls. The operator may select which methods) to run on the control and events fired by _g-controls are used to sequence the logic. Hence, the present invention sets how any third party ActiveX
control behaves by viewing the current standard inter-faces .
The second unique feature of the present invention is the menu structure that allows operators to think top-down or bottom-up in the way they program.
Hence, we are defining a methodology analogous to the human operator to define an inspection routine. This is addressing the type of person who needs to use the system. We want the operator to have applications knowledge and do not want to necessitate that he also be a programmer. Take the following example: An operator wishes to measure the distance between two points. He can think metrology first and start with a rule ActiveX
control. The interactive environment after interrogat-ing the ruler ActiveX control knows that two geometries are required. The interactive environment then queries all available ActiveX controls and determines which to present to the operator. Hence, the operator can choose between a line, point or other available geometry. The operator can select to define a new geometry or an existing geometry. Alternatively, the operator thinks points first and proceed to define the points. He then thinks metrology and selects the ruler ActiveX control to measure between already defined points. In this example, it was irrelevant whether the operator think geometry or metrology first. Both approaches work and the interactive environment create a program that runs and does not have any syntax errors. The whole program-ming sequence is done via menus and icons in a drag and drop environment.
WO 99114646 PCTlUS98/17066 The third feature of this invention is that predefined interfaces are supported to exchange data between tools. By default ActiveX controls expose a set of predefined variables such as floats, double string integers, etc. The invention exposes another interface (COMM) which allows the user to expose his own prede-fined objects, and have these passed between different ActiveX controls. This defined interface allows for the management of different user defined objects between different ActiveX controls.
The fourth characteristic of this interactive Grafcet/ActiveX environment is the mathematical compo-nent that allows the manipulation of data between ActiveX controls in a user friendly manner.
Glossary ActiveX Control. A name as defined by Micro-soft for programmable elements formerly known variously as OLE Controls, OCXs, or OLE Custom Controls. It uses component technologies built on the Component Object Model (COM). Generally, an ActiveX control is an object that, when instantiated, embodies both specific data and the functions that manipulate it. The ActiveX control is an entity that has state, behavior and identity. An object's state consists of its attributes also known as properties and the attributes' current values. An object's behavior consists of the operations (methods) that can be performed on it and the accompanying state changes. An object's identity is what one uses to distinguish it from other objects. Finally, an ActiveX
component has events that inform when something has happened. In contrast, COM objects' behavior is defined by the interfaces it supports. A COM object's state is not explicitly specified, but is implied by its inter-f aces .
COM. An open architecture for cross-platform development for applications based on object oriented technology as agreed upon by Digital Equipment and Microsoft Corporation. The Component Object Model defines an interface (similar to an abstract base class), Unknown, from which all COM-compatible classes are derived.
Ev~nt~. A notification message sent from an object in response to a state change or user action.
Grafcet. The Grafcet is a graphical program-ming language - originally developed by AFCET (Associa-tion Francais pour la Cybernetique Economique et Tech-nique) in France and has become an international stan-dard. The Grafcet language represents the functions of an automation system as a sequence of steps and transi-tions - with directed flow lines connecting the two. It was specifically designed for sequential control prob-lems where the steps are sequential or time dependent.
Each step in a Grafcet sequence represents a stable situation in an automation process and has one or more actions associated with it - while a transition indi-cater the possibility of evolution from one step to the next. Boolean events are associated with each transi-tion. Sequences may be linked together by a branch construct which may represent a conditional or a paral-lel flaw of execution. A single Flow Line is used to link steps and transitions together to indicate the flow of execution. Double flow lines are used when two or more steps within a program must be synchronized.
WD 99/14b46 PCT/US98/17066 IEC 1131. A standard set down by the Interna-tional Electrotechnical Commission that specifies the syntax and semantics of a unified suite of programming languages for programmable controllers. Part of the standard refers to Sequential Function Charts that is Grafcet-like in structure.
Methods. A procedure that provides access to an object's data and asks an object to behave as defined by the procedure.
Obiect-oriented Design. Or object-oriented programming. In traditional procedural languages (such as C, FORTRAN, and COBOL), code and data are separate.
In the object-oriented approach, code and data that belong together can be combined into objects. Object-oriented design is further characterized by the use of inheritance (derived classes), polymorphism, encapsula-tion and virtual functions (C++) in programming.
_Open. Open for our patent refers to existing standards that are non-proprietary and used by many companies. In the patent to Silver, open is that yes other people can use this system but it is a proprietary Cognex only standard, and not an open intercompany standard. Visual basic is an open programming language.
Visual C++ is an open programming language.
Properti~$. The attribute of an ActiveX
control. This attribute has a value that can either be read, written, or both.
Transition. A Grafcet transition defines the state of a transition and has an associated Boolean state. It indicates the possibility of evolution from one step to the next. Boolean events are associated with each transition.
RP_L. Rapid programming language. Languages like Visual Basic are rapid programming languages because the programmer can quickly set up user interface and insert objects to achieve his goal. Visual C++ is not regarded as a rapid programming language. Delphi and Visual Basic are regarded as rapid programming languages.
Step. Within the Grafcet/IEC 1131, each step or node in a Grafcet sequence represents a stable situa tion in an automation process and has one or more actions associated with it. It has been redefined herein to represent an ActiveX or similar standard component.
VisionBloxTM VisionBloxT'" is the tool-kit of machine vision components obeying the ActiveX standards.
The components include imaging processing, image acqui sition, calibration, geometric functions, and motion control.
The above objects and other objects, features, and advantages of the present invention are readily apparent from the following detailed description of the best mode for carrying out the invention when taken in connection with the accompanying drawings.
Brief Description Of The Drawings FIGURE 1 is a schematic diagram illustrating a preferred hardware configuration on which the method of the present invention can be implemented;
FIGURE 2 is a schematic diagram illustrating a machine vision system which can be supported by the method and system of the present invention;
FIGURE 3 is a portion of a programming envi ronment after standard controls of the present invention have been added;
FIGURE 4 is schematic view of a typical Grafcet sequence constructed in accordance with the present invention;
FIGURE 5 is a schematic view of a typical Grafcet sequence showing two alternative paths of execution;
FIGURE 6 is a schematic view of a section of a Grafcet sequence showing a parallel branch;
FIGURE 7 is a screen display illustrating math's control; and FIGURE 8 is a block diagram flow chart illus-trating a typical operation sequence in a normal running mode of the vision system which is waiting for a "part present" signal.
Best Modes For Cam 'i~ng Out The Invention Referring now to the drawings figures, there is illustrated in Figure 1 a workstation on which the method and system of the present invention can be implemented. The hardware illustrated in Figure 1 includes a monitor 10 such as a single SVGA display, a keyboard 12, a pointing device such a mouse 14, a magnetic storage device 16, and a chassis 18 including a CPU and random access memory. In a preferred embodi-ment, the chassis 18 is a Pentium-based IBM compatible PC or other PC having at least 8 megabytes of RAM and at least 12 megabytes of hard disk space.
The hardware configuration also includes the development environment of a high-level, graphic, flow-control language such as Grafcet modified by the IEC
1131 standard for pictorially specifying the structure and detailed interaction of a system of concurrent processes.
Referring now to Figure 2, there is illustrat-ed schematically a machine vision system generally indicated at 20 generally of the type which can be supported by the method and system of the present invention. The machine vision system 20 typically includes an image digitizer/frame grabber 22. The image digitizer/frame grabber 22 samples and digitizes the input images from one or more image sources such as cameras 24 and places each input image into a frame buffer having picture elements. Each of the picture elements may consist of an 8-bit number representing the brightness of that spot in the image.
The system 20 also includes a system bus 38 which receives information from the image digi-tizer/frame grabber 22 and passes the information on to the IBM compatible host computer.
The system 20 also includes input/output circuits 30 to allow the system 20 to communicate with external peripheral devices such as robots, programmable controllers, etc. having one or more stages.
One or more of the cameras 24 may be an image source such as an analog digital or line scan camera such as RS-170, CCIR, NTSC and PAL.
The system bus 26 may be either a PCI an EISA, ISA or VL system bus or any other standard bus.
The I/O circuits 30 may support a three axis stepper board (i.e. supports multiple axis control) or other motion boards.
The image digitizer/frame grabber 22 may be a conventional frame grabber board such as that manufac-tured by Matrox, Cognex, Data Translation or other frame grabbers. Alternatively, the image digitizer/frame grabber 22 may comprise a vision processor board such as made by Cognex.
The machine vision system 20 may be programmed at a mass storage unit 32 to include custom controls for image processing, image analysis, third party machine vision products, calibration, and interactive CAD/geome-try as described in greater detail hereinbelow. Exam-ples of image processing may include linear and non-linear enhancement, morphology, color and image arithme-tic. Also, image analysis may include search, edge, caliper, blob, template, color, 2-D and 3-D measure-ments.
Third party products may include digital I/O, various camera formats, motion, databases, SPC and others.
Calibration may include non-linear, 2-D, 3-D
and color calibration.
Also, interactive CAD/geometry custom control may be provided for both 2-D and 3-D space.
It is to be understood that other similar programming environments can be utilized by the method and system of the present invention.
Referring now to Figure 3, there are illus-trated icons of a toolbox which appear after various custom controls, which will be described in greater detail hereinbelow, are added thereto.
Each custom control is generally described as follows:
An editable shape custom control allows a user to define an image processing region by interactively editing a rectangular ellipsoidal or toroidal (donut) shape.
An editable image custom control works like the visual basic picture box control with enhancements that give the user added image viewing and processing options.
A camera custom control allows a user to capture and store live images from a video camera.
A search tool custom control looks for a specific feature in an image that matches a model that a user trained it to recognize; reports the characteris-tic of the feature found.
A Magic custom control allow the camera control to capture images when the user has a Magic vision board installed. (This tool is invisible at run time.) A blob tool custpm control finds groups of connected pixels or "blobs" in an image; reports on the characteristics of each blob it found.
An SE 100 custom control allows the camera control to capture images when the user has an SE 100 vision board installed. (This tool is invisible at run t ime . ) A VP 50 custom control allows the camera control to capture images - and optionally speeds up images searches - when the user has a VP vision board installed. (This tool is invisible at run time.) A Type I stage custom control allows the stage control to work with a specific manufacturer's stage known as "Type I". (This tool is invisible at run time. ) A Stage custom control allows a user to control a multi-axis stage through a graphical user interface. The developer can include stage control properties and methods in higher level code.
While not shown, a tool board custom control allows a user to navigate through windows and menus in the application by simply clicking buttons on a tool bar.
Also while not shown, a Type II stage custom control allows the stage control tv work with a specific manufacturer's stage known as "Type II". (This tool is also invisible at run time.) Finally, while also not shown, a Type III
stage custom control allows the stage control to work with a specific manufacturer's stage known as "Type III". (This tool is furthermore invisible at run time.) In general, it is possible to interactively and easily build machine vision applications or programs using the ActiveX custom controls illustrated in Figure 4 in the Grafcet/EIC 1131 programming system. In general, one creates the user interface or a vision application by placing the custom controls on a form.
Next, one sets the properties for both the Visual Basic standard controls and the custom controls. One need not write code to link together the different parts of the application to obtain the application program since this is done automatically.
A detailed description of many of the various machine vision controls represented by the icons of Figure 3 can be found in the above-noted patent applica-tion.
Two standards are being married together by the present invention to produce an open programmable/reconfigurable/teachable system that does not require the use of a software programming environ-s ment. These standards are Grafcet and its derivative IEC 1131 and the ActiveX standard. Optionally via the addition of some predefined COM interfaces, the exchange of data between ActiveX objects is enhanced.
Hence, the advantages of marrying these two standards allows any ActiveX components to be inserted within an environment that is familiar in the automation (i.e. PLC) context. Secondly, by interrogating the properties, methods and events of these ActiveX compo-nents operation control can be tailored by the user without doing any programming. Properties, i.e. data, can be exchanged between controls. Mathematical opera-tion can be performed on this data. By selecting which method to call in a control, its operation is con-trolled. Finally, events control the flow of the logic.
The Grafcet/IEC 1131 standard allows conditional and parallel control.
This interactive Grafcet/ActiveX environment is itself an ActiveX component and so can be used as a component within another interactive Grafcet/ActiveX
environment.
The user configures the method and system of the present invention either via a drag and drop envi-ronment or via a menu system. The menu system is designed to allow the operator to define a sequence either bottom-up or top-down. This is achieved as the user, who need not be a programmer, can "teach" a task.
The system queries his intentions, and guides the user's choices until the task is taught. In the process, the user does not write a single line of code and has available to him not only all the tools, vision or otherwise, but can set the variables that the tools require interactively. Certain tool's outputs will be another tool's input.
For example, to measure a distance a ruler tool requires two input geometries and, in turn, outputs a distance. Two other tools, e.g. an edge tool defines an edge and feature find tools match a model to define a point. Both these can be the inputs to the ruler tool.
The user may start top-down, i.e. he thinks measurement first and the system guides him to define the input's geometries. Alternatively, the user may think bottom-up and define the geometries first and then decide to do a measurement between them.
Features. Components aad Modules ~rafcet Control The Grafcet/IEC 1131 standard of the present invention includes transitions and steps.
A transition in the Grafcet/IEC 1131 standard is either:
True (the default case);
An event generated by some ActiveX Control, e.g. an IO ActiveX trigger control; or A Boolean expression defined by internal variable or properties of any Control.
A Step in the Grafcet standard is either:
Any 3rd party ActiveX control; or A mathematical step that allows the definition of new internal variables.
Clicking with the left button of the mouse 14 on a control gives the default control interface.
Clicking with the right button of the mouse 14 on the control allows the user to select which method to run and which properties need to be set.
The Grafcet control is an ActiveX custom control that allows vision applications to be programmed in the form of a Grafcet sequence or structure. Pro gramming takes place through a graphical user interface .
Existing ActiveX controls are integrated into the Grafcet sequence.
In one method of creating a Grafcet sequence, the user selects tools from a toolbar displayed on the Grafcet interface (such as illustrated in Figure 3) and drags and drops the tool onto a Sequence display area (a 2 dimensional grid). The steps are shown graphically with icons (representing the tool) which are displayed on the grid. Transitions and flow lines are automati-cally inserted to maintain a flow chart that obeys the rules of the Grafcet programming language (see Fig. 4).
Where inputs are required for some tools (steps) - the user is prompted for these through a message (displayed in a message panel) and the position for insertion of the input is highlighted. The user selects inputs in the same way as tools are selected -by dragging and dropping onto the sequence. Outputs from steps will be shown automatically. A Grafcet sequence may be inserted as a step in another sequence -allowing for sequence reuse and multi-level sequence definition. The user may also insert a "parallel" or "conditional" branch item into the sequence (Figures 5 and 6 ) .
Within a Grafcet Control, Grafcet sequences may be constructed, edited and executed. A pop-up menu is available to access Grafcet Control member functions. To display a pop-up menu click the right button of the mouse 14 when the cursor is within the Grafcet Control Environment (NOTE: if the right mouse button is clicked over a Vision control - then the pop up for that control will appear instead of the Grafcet control pop-up).
Math's and Event Step The math's step allows the definition of variables and expressions that set the values of these variables. These variables are either numeric or Boolean and can be set to some mathematical function of other variables or properties of Activex controls or some combination of these. (See Figure 7).
An event step allows an event to be fired into Visual Basic. Thus some Visual Basic can get control during a sequence to perform some predefined functions.
~iierarchical Menu Module As Related To Machine Vision Svstem Overview The Interactive GrafcetActiveX environment System of the present invention uses image analysis techniques to perform image analysis and metrology functions, i.e. calibrated measurement. Features such as Lines, circles and points etc. can be found and measured. Complex shapes can be identified and veri-fied.
Complete complex applications can be config-ured literally in minutes, and controlled remotely -using either a digital or a serial link.
The features of the system include:
~ Ease of Use ~ Calibrated Measurement Using Real World Units ~ Extensive Set of Inspection Commands ~ Efficient Use of Memory ~ Simple Digital and Serial Tnterfacing There are two phases in the operation of the system of the present invention:
~ Teach Phase: Here the user, using the computer keyboard and monitor, configures the system for his application e.g. part identification or measurement.
~ Execution Phase: Once configured, the teach system can run completely automati-cally and can be controlled via an RS232 serial link or it's digital inputs.
In the first phase the user "teaches" the system do a particular application. Using the computer CA 02304066 2000-03-16 ' keyboard 12 and monitor 10, the user "configures" the system for a particular application e.g. part identifi-cation, measurement etc. Use of the system is made easy by the menu-driven user interface which provides the user with all the commands in a structured and hierar-chical manner. Help is available at all points in the program.
The system stores the particular sequence of commands that make the application inta an application file. More than one application can be loaded at any one time.
In the second phase, applications can be executed at the press of a button on the keyboard, a remote input, or triggered through the RS232 or other communications line. Automatic- system diagnostics provide details of the performance of the application to the user, showing the nature and type of any problems.
All measurements are made in real world co-ordinates (m, mm, microns, i:n., thou., etc.).
Many commands are available to include metrol-ogy (measurement) functions and image analysis func-tions. Examples of metrology functions include the measurement of distances, angles, diameters, perimeter, areas, straightness, roundness and shape. Examples of vision functions include gray level edge finding (capa-ble of finding subtle grey level edges), projection, edge sniffing and feature correlation.
System functions can be divided into 3 main categories:
~ Image Analysis Functions: These func-tions determine features of an image ~ System Calibration: These functions allow the system to measure and work in a user defined co-ordinate frame ~ System I/O: These functions allow the system to respond to digital inputs, set digital outputs, communicate through a serial line, print information and save and retrieve information from files.
Image l4sialysis Furictions The image analysis functions can be sorted into six main categories:
~ Part locate Part locate identifies a point and orientation on the image. All search regions, e.g. lines, windows and ellipses that are defined after a part has been located are defined relative to this new part frame ~ Geometry definition These functions concern the defini-tion of the entities that the system uses during part inspection. Lines, ellipses, points and profiles can be defined using these functions.
~ Metrology These functions concern the measure-ment of various quantities between entities defined within the system.
Distances, areas, perimeters and angles can be measured. Some quan-tities can be measured between enti-ties of differing types, e.g. dis-tance from a point to a line. Geo-metric errors can also be measured.
These include the straightness of a line and roundness of a circle.
~ Morphology Measurement These functions concern the measure-ment of surface details, such as cracks, in the image.
~ Feature Correlation This function finds a pre-taught feature (model), to pixel accuracy within a grabbed image. The system calculates a "score" that shows the level of similarity between the model and the feature in the current image. The score is defined as a percentage, with 100% being a per-fect match.
~ Morphology enhancement These functions concern the enhance-ment of surface details, such as cracks, within the image.
Color Image Processing Calibration The purpose of system recalibration is to tell the vision system the units it is working in, i.e. mm, inches etc. Calibration defines the attitude of the camera 24 relative to the field of view. Camera cali-bration is accomplished by viewing a object of known shape and size, usually a square or round plate.
Calibration files are saved with the job file thus WO 99!14646 PCT/US98/17066 allowing different jobs to have different calibration data.
System calibration can be implemented using several methods. The first method/phase has been implemented using several methods. The full methods include:
Initial System Camera Calibration: This occurs whenever the camera 24 is moved, or every 6 months on system maintenance.
At calibration time an object of known size is placed in the field of view of each camera 24, its size and location being defined and identified by the system.
Monitoring Camera Movement and Automatic System Recalibration: This is the pro-cess whereby the system monitors camera movement, and compensates for such move-ment. The amount of movement observed is calculated, the calibration matrices adjusted and, if excessive, a warning is raised.
Continuous Automatic System Recalibra-tion: In certain instances system cali-bration can be accomplished completely automatically. The reference objects) is/are continuously in, or placed in, the field of view. The system, therefore, calibrates itself and returns information on camera movement. If the camera move-ment is so excessive that the reference object cannot be seen then a warning is WO 99/14646 PCTIUS98/1?066 raised. The system continues to operate using the last default calibration.
Menus This set of menus allows different geometric entities (points, lines, circles, ellipses and profiles) to be defined and selected. Such entities can either be defined using analysis of the video source or can be constructed using existing geometric entities.
Geometry Menu ZO Point Define Menu Point on Edge (Window Scan) Point Define via Feature Location Center of an Ellipse Line + Line Intersect Point on a Profile Inflection Point Deflection Point Line Def ine Menu Define by Scanning for Edge Define by Hand Define Via Two Points Profile Primary Vector Circle Define Menu Define by Scanning for an Edge Manual Profile Define Menu Metroloav Menus The metrology menus allow the user to perform measurements and implement tests based on the measure-ment results. The measurements and test results can be saved. Statistical Process Control (SPC) may be per-formed on the measured results. Metrology functions select geometric entries independent of where they are defined. When selecting an entity all geometry's are displayed, from which to select the required entity.
Linear Dimensions Distance Line to Line Distance Line to Point Distance Point to Point Menu Caliper:, Single Pair Caliper: Multiple Pairs Actual Distance Component Distance Circle/Ellipse Radius Angular Dimensions Angle Line to Line Angle Line Area Measurement Morphology Measurement Intensity Evaluation Connectivity Evaluation Geometric Errors Menu Parallelism Squareness Form Error Straightness Form Error Roundness Data Interface Several optional interfaces are suggested.
These interface are NOT compulsory as the interactive Grafcet/ActiveX environment can work without them.
However, if any of these interface are present this extends the functionality to the Grafcet/ActiveX envi-ronment.
These interface provides a set of defined methods and attributes obeying the COM standard.
Display Methods (iDiaplay) Draw Draws the tool ShowDisplayGraphicssplay result graphics HideDisplayGraphics Hide result graphics EnableAutoRefresh Enables the tool to refresh the result display upon use DisableAutoRefresh Disables the tool from refreshing the result display upon use IsAutoRefreshEnabled Returns if the tools auto refresh is enabled or dis-abled Information Methods (iInformation) GetMemorySize Returns the current quantity of heap, in bytes, which the tool is using SetMaximumTime Sets the maximum time allows for the use function to exe cute GetMinimumTime Returns the minimum time required for the use function to execute TimeTaken Returns the time taken by the Use function on the last use GetMaximumTime Returns the current setting for Use max-imum time allowed for the function to execute Data Access Methods (iData) GetlnputType Returns information regarding an input GetOutput Type Returns information regarding an output GetNumberOfInputs Returns the number of inputs GetNumberOfOutputs Returns the number of outputs GetOutput Returns a given out-put SetInput Set a given input Tool Methods(iTool}
AutoTrain Automatic Training method for the tool Train Initiate training of l0 the tool AddSample Add Sample to cur-rent training set for the tool PrepareToUse Prepares the tool for first use Use Uses the tool with the current trained information GetLastError Returns the last error generated by the tool Save Save tool configura-tion and training information to file Load Load tool configura-tion and training information to file Serialize Serializes the tool configuration to file While the best mode for carrying out the invention has been described in detail, those familiar with the art to which this invention relates will recognize various alternative designs and embodiments for practicing the invention as defined by the following claims.
The idea behind interactively generating a computer program is to ensure that the programmer cannot make syntax errors in producing his code. Hence, the environment guides the programmer via menus and appro-priate interface structure to produce code that is correct in syntax. The problem being addressed is that computer languages typically are tightly defined and intolerant to syntax errors.
_g_ The second idea is that the system operator does not need to be a skilled programmer. Rather, he can be an applications engineer who primarily under-stands the application and does not need to be a pro-s grammer. These days, automation engineers are used to using programming in ladder logic and Grafcet-type languages. Applications knowledgeable engineers tend to be familiar with these environments and the interactive environment being presented in this application is for these engineers.
The method and system of the present applica-tion is different from the examples mentioned above in that it takes a standard of Grafcet and modifies it to allow the integration of another standard, namely ActiveX controls which can be linked in.
Hence, a first unique feature of this inven-tion is that any third party tools in machine vision or automation or otherwise that are written as ActiveX
controls can be linked into this interactive environ-ment. To date, no interactive environment allows the interaction of third party components and definitely not the configuration of any ActiveX controls.
ActiveX controls are becoming widely accepted and many are being produced by many companies . These are able to be linked together and used without writing code that can produce syntax errors. A control sequence is set up triggered on events, timers and variable settings. This interactive modified Grafcet/IEC 1131 environment reads the properties of the ActiveX controls and allows these to be set or linked to other properties of other controls. The operator may select which methods) to run on the control and events fired by _g-controls are used to sequence the logic. Hence, the present invention sets how any third party ActiveX
control behaves by viewing the current standard inter-faces .
The second unique feature of the present invention is the menu structure that allows operators to think top-down or bottom-up in the way they program.
Hence, we are defining a methodology analogous to the human operator to define an inspection routine. This is addressing the type of person who needs to use the system. We want the operator to have applications knowledge and do not want to necessitate that he also be a programmer. Take the following example: An operator wishes to measure the distance between two points. He can think metrology first and start with a rule ActiveX
control. The interactive environment after interrogat-ing the ruler ActiveX control knows that two geometries are required. The interactive environment then queries all available ActiveX controls and determines which to present to the operator. Hence, the operator can choose between a line, point or other available geometry. The operator can select to define a new geometry or an existing geometry. Alternatively, the operator thinks points first and proceed to define the points. He then thinks metrology and selects the ruler ActiveX control to measure between already defined points. In this example, it was irrelevant whether the operator think geometry or metrology first. Both approaches work and the interactive environment create a program that runs and does not have any syntax errors. The whole program-ming sequence is done via menus and icons in a drag and drop environment.
WO 99114646 PCTlUS98/17066 The third feature of this invention is that predefined interfaces are supported to exchange data between tools. By default ActiveX controls expose a set of predefined variables such as floats, double string integers, etc. The invention exposes another interface (COMM) which allows the user to expose his own prede-fined objects, and have these passed between different ActiveX controls. This defined interface allows for the management of different user defined objects between different ActiveX controls.
The fourth characteristic of this interactive Grafcet/ActiveX environment is the mathematical compo-nent that allows the manipulation of data between ActiveX controls in a user friendly manner.
Glossary ActiveX Control. A name as defined by Micro-soft for programmable elements formerly known variously as OLE Controls, OCXs, or OLE Custom Controls. It uses component technologies built on the Component Object Model (COM). Generally, an ActiveX control is an object that, when instantiated, embodies both specific data and the functions that manipulate it. The ActiveX control is an entity that has state, behavior and identity. An object's state consists of its attributes also known as properties and the attributes' current values. An object's behavior consists of the operations (methods) that can be performed on it and the accompanying state changes. An object's identity is what one uses to distinguish it from other objects. Finally, an ActiveX
component has events that inform when something has happened. In contrast, COM objects' behavior is defined by the interfaces it supports. A COM object's state is not explicitly specified, but is implied by its inter-f aces .
COM. An open architecture for cross-platform development for applications based on object oriented technology as agreed upon by Digital Equipment and Microsoft Corporation. The Component Object Model defines an interface (similar to an abstract base class), Unknown, from which all COM-compatible classes are derived.
Ev~nt~. A notification message sent from an object in response to a state change or user action.
Grafcet. The Grafcet is a graphical program-ming language - originally developed by AFCET (Associa-tion Francais pour la Cybernetique Economique et Tech-nique) in France and has become an international stan-dard. The Grafcet language represents the functions of an automation system as a sequence of steps and transi-tions - with directed flow lines connecting the two. It was specifically designed for sequential control prob-lems where the steps are sequential or time dependent.
Each step in a Grafcet sequence represents a stable situation in an automation process and has one or more actions associated with it - while a transition indi-cater the possibility of evolution from one step to the next. Boolean events are associated with each transi-tion. Sequences may be linked together by a branch construct which may represent a conditional or a paral-lel flaw of execution. A single Flow Line is used to link steps and transitions together to indicate the flow of execution. Double flow lines are used when two or more steps within a program must be synchronized.
WD 99/14b46 PCT/US98/17066 IEC 1131. A standard set down by the Interna-tional Electrotechnical Commission that specifies the syntax and semantics of a unified suite of programming languages for programmable controllers. Part of the standard refers to Sequential Function Charts that is Grafcet-like in structure.
Methods. A procedure that provides access to an object's data and asks an object to behave as defined by the procedure.
Obiect-oriented Design. Or object-oriented programming. In traditional procedural languages (such as C, FORTRAN, and COBOL), code and data are separate.
In the object-oriented approach, code and data that belong together can be combined into objects. Object-oriented design is further characterized by the use of inheritance (derived classes), polymorphism, encapsula-tion and virtual functions (C++) in programming.
_Open. Open for our patent refers to existing standards that are non-proprietary and used by many companies. In the patent to Silver, open is that yes other people can use this system but it is a proprietary Cognex only standard, and not an open intercompany standard. Visual basic is an open programming language.
Visual C++ is an open programming language.
Properti~$. The attribute of an ActiveX
control. This attribute has a value that can either be read, written, or both.
Transition. A Grafcet transition defines the state of a transition and has an associated Boolean state. It indicates the possibility of evolution from one step to the next. Boolean events are associated with each transition.
RP_L. Rapid programming language. Languages like Visual Basic are rapid programming languages because the programmer can quickly set up user interface and insert objects to achieve his goal. Visual C++ is not regarded as a rapid programming language. Delphi and Visual Basic are regarded as rapid programming languages.
Step. Within the Grafcet/IEC 1131, each step or node in a Grafcet sequence represents a stable situa tion in an automation process and has one or more actions associated with it. It has been redefined herein to represent an ActiveX or similar standard component.
VisionBloxTM VisionBloxT'" is the tool-kit of machine vision components obeying the ActiveX standards.
The components include imaging processing, image acqui sition, calibration, geometric functions, and motion control.
The above objects and other objects, features, and advantages of the present invention are readily apparent from the following detailed description of the best mode for carrying out the invention when taken in connection with the accompanying drawings.
Brief Description Of The Drawings FIGURE 1 is a schematic diagram illustrating a preferred hardware configuration on which the method of the present invention can be implemented;
FIGURE 2 is a schematic diagram illustrating a machine vision system which can be supported by the method and system of the present invention;
FIGURE 3 is a portion of a programming envi ronment after standard controls of the present invention have been added;
FIGURE 4 is schematic view of a typical Grafcet sequence constructed in accordance with the present invention;
FIGURE 5 is a schematic view of a typical Grafcet sequence showing two alternative paths of execution;
FIGURE 6 is a schematic view of a section of a Grafcet sequence showing a parallel branch;
FIGURE 7 is a screen display illustrating math's control; and FIGURE 8 is a block diagram flow chart illus-trating a typical operation sequence in a normal running mode of the vision system which is waiting for a "part present" signal.
Best Modes For Cam 'i~ng Out The Invention Referring now to the drawings figures, there is illustrated in Figure 1 a workstation on which the method and system of the present invention can be implemented. The hardware illustrated in Figure 1 includes a monitor 10 such as a single SVGA display, a keyboard 12, a pointing device such a mouse 14, a magnetic storage device 16, and a chassis 18 including a CPU and random access memory. In a preferred embodi-ment, the chassis 18 is a Pentium-based IBM compatible PC or other PC having at least 8 megabytes of RAM and at least 12 megabytes of hard disk space.
The hardware configuration also includes the development environment of a high-level, graphic, flow-control language such as Grafcet modified by the IEC
1131 standard for pictorially specifying the structure and detailed interaction of a system of concurrent processes.
Referring now to Figure 2, there is illustrat-ed schematically a machine vision system generally indicated at 20 generally of the type which can be supported by the method and system of the present invention. The machine vision system 20 typically includes an image digitizer/frame grabber 22. The image digitizer/frame grabber 22 samples and digitizes the input images from one or more image sources such as cameras 24 and places each input image into a frame buffer having picture elements. Each of the picture elements may consist of an 8-bit number representing the brightness of that spot in the image.
The system 20 also includes a system bus 38 which receives information from the image digi-tizer/frame grabber 22 and passes the information on to the IBM compatible host computer.
The system 20 also includes input/output circuits 30 to allow the system 20 to communicate with external peripheral devices such as robots, programmable controllers, etc. having one or more stages.
One or more of the cameras 24 may be an image source such as an analog digital or line scan camera such as RS-170, CCIR, NTSC and PAL.
The system bus 26 may be either a PCI an EISA, ISA or VL system bus or any other standard bus.
The I/O circuits 30 may support a three axis stepper board (i.e. supports multiple axis control) or other motion boards.
The image digitizer/frame grabber 22 may be a conventional frame grabber board such as that manufac-tured by Matrox, Cognex, Data Translation or other frame grabbers. Alternatively, the image digitizer/frame grabber 22 may comprise a vision processor board such as made by Cognex.
The machine vision system 20 may be programmed at a mass storage unit 32 to include custom controls for image processing, image analysis, third party machine vision products, calibration, and interactive CAD/geome-try as described in greater detail hereinbelow. Exam-ples of image processing may include linear and non-linear enhancement, morphology, color and image arithme-tic. Also, image analysis may include search, edge, caliper, blob, template, color, 2-D and 3-D measure-ments.
Third party products may include digital I/O, various camera formats, motion, databases, SPC and others.
Calibration may include non-linear, 2-D, 3-D
and color calibration.
Also, interactive CAD/geometry custom control may be provided for both 2-D and 3-D space.
It is to be understood that other similar programming environments can be utilized by the method and system of the present invention.
Referring now to Figure 3, there are illus-trated icons of a toolbox which appear after various custom controls, which will be described in greater detail hereinbelow, are added thereto.
Each custom control is generally described as follows:
An editable shape custom control allows a user to define an image processing region by interactively editing a rectangular ellipsoidal or toroidal (donut) shape.
An editable image custom control works like the visual basic picture box control with enhancements that give the user added image viewing and processing options.
A camera custom control allows a user to capture and store live images from a video camera.
A search tool custom control looks for a specific feature in an image that matches a model that a user trained it to recognize; reports the characteris-tic of the feature found.
A Magic custom control allow the camera control to capture images when the user has a Magic vision board installed. (This tool is invisible at run time.) A blob tool custpm control finds groups of connected pixels or "blobs" in an image; reports on the characteristics of each blob it found.
An SE 100 custom control allows the camera control to capture images when the user has an SE 100 vision board installed. (This tool is invisible at run t ime . ) A VP 50 custom control allows the camera control to capture images - and optionally speeds up images searches - when the user has a VP vision board installed. (This tool is invisible at run time.) A Type I stage custom control allows the stage control to work with a specific manufacturer's stage known as "Type I". (This tool is invisible at run time. ) A Stage custom control allows a user to control a multi-axis stage through a graphical user interface. The developer can include stage control properties and methods in higher level code.
While not shown, a tool board custom control allows a user to navigate through windows and menus in the application by simply clicking buttons on a tool bar.
Also while not shown, a Type II stage custom control allows the stage control tv work with a specific manufacturer's stage known as "Type II". (This tool is also invisible at run time.) Finally, while also not shown, a Type III
stage custom control allows the stage control to work with a specific manufacturer's stage known as "Type III". (This tool is furthermore invisible at run time.) In general, it is possible to interactively and easily build machine vision applications or programs using the ActiveX custom controls illustrated in Figure 4 in the Grafcet/EIC 1131 programming system. In general, one creates the user interface or a vision application by placing the custom controls on a form.
Next, one sets the properties for both the Visual Basic standard controls and the custom controls. One need not write code to link together the different parts of the application to obtain the application program since this is done automatically.
A detailed description of many of the various machine vision controls represented by the icons of Figure 3 can be found in the above-noted patent applica-tion.
Two standards are being married together by the present invention to produce an open programmable/reconfigurable/teachable system that does not require the use of a software programming environ-s ment. These standards are Grafcet and its derivative IEC 1131 and the ActiveX standard. Optionally via the addition of some predefined COM interfaces, the exchange of data between ActiveX objects is enhanced.
Hence, the advantages of marrying these two standards allows any ActiveX components to be inserted within an environment that is familiar in the automation (i.e. PLC) context. Secondly, by interrogating the properties, methods and events of these ActiveX compo-nents operation control can be tailored by the user without doing any programming. Properties, i.e. data, can be exchanged between controls. Mathematical opera-tion can be performed on this data. By selecting which method to call in a control, its operation is con-trolled. Finally, events control the flow of the logic.
The Grafcet/IEC 1131 standard allows conditional and parallel control.
This interactive Grafcet/ActiveX environment is itself an ActiveX component and so can be used as a component within another interactive Grafcet/ActiveX
environment.
The user configures the method and system of the present invention either via a drag and drop envi-ronment or via a menu system. The menu system is designed to allow the operator to define a sequence either bottom-up or top-down. This is achieved as the user, who need not be a programmer, can "teach" a task.
The system queries his intentions, and guides the user's choices until the task is taught. In the process, the user does not write a single line of code and has available to him not only all the tools, vision or otherwise, but can set the variables that the tools require interactively. Certain tool's outputs will be another tool's input.
For example, to measure a distance a ruler tool requires two input geometries and, in turn, outputs a distance. Two other tools, e.g. an edge tool defines an edge and feature find tools match a model to define a point. Both these can be the inputs to the ruler tool.
The user may start top-down, i.e. he thinks measurement first and the system guides him to define the input's geometries. Alternatively, the user may think bottom-up and define the geometries first and then decide to do a measurement between them.
Features. Components aad Modules ~rafcet Control The Grafcet/IEC 1131 standard of the present invention includes transitions and steps.
A transition in the Grafcet/IEC 1131 standard is either:
True (the default case);
An event generated by some ActiveX Control, e.g. an IO ActiveX trigger control; or A Boolean expression defined by internal variable or properties of any Control.
A Step in the Grafcet standard is either:
Any 3rd party ActiveX control; or A mathematical step that allows the definition of new internal variables.
Clicking with the left button of the mouse 14 on a control gives the default control interface.
Clicking with the right button of the mouse 14 on the control allows the user to select which method to run and which properties need to be set.
The Grafcet control is an ActiveX custom control that allows vision applications to be programmed in the form of a Grafcet sequence or structure. Pro gramming takes place through a graphical user interface .
Existing ActiveX controls are integrated into the Grafcet sequence.
In one method of creating a Grafcet sequence, the user selects tools from a toolbar displayed on the Grafcet interface (such as illustrated in Figure 3) and drags and drops the tool onto a Sequence display area (a 2 dimensional grid). The steps are shown graphically with icons (representing the tool) which are displayed on the grid. Transitions and flow lines are automati-cally inserted to maintain a flow chart that obeys the rules of the Grafcet programming language (see Fig. 4).
Where inputs are required for some tools (steps) - the user is prompted for these through a message (displayed in a message panel) and the position for insertion of the input is highlighted. The user selects inputs in the same way as tools are selected -by dragging and dropping onto the sequence. Outputs from steps will be shown automatically. A Grafcet sequence may be inserted as a step in another sequence -allowing for sequence reuse and multi-level sequence definition. The user may also insert a "parallel" or "conditional" branch item into the sequence (Figures 5 and 6 ) .
Within a Grafcet Control, Grafcet sequences may be constructed, edited and executed. A pop-up menu is available to access Grafcet Control member functions. To display a pop-up menu click the right button of the mouse 14 when the cursor is within the Grafcet Control Environment (NOTE: if the right mouse button is clicked over a Vision control - then the pop up for that control will appear instead of the Grafcet control pop-up).
Math's and Event Step The math's step allows the definition of variables and expressions that set the values of these variables. These variables are either numeric or Boolean and can be set to some mathematical function of other variables or properties of Activex controls or some combination of these. (See Figure 7).
An event step allows an event to be fired into Visual Basic. Thus some Visual Basic can get control during a sequence to perform some predefined functions.
~iierarchical Menu Module As Related To Machine Vision Svstem Overview The Interactive GrafcetActiveX environment System of the present invention uses image analysis techniques to perform image analysis and metrology functions, i.e. calibrated measurement. Features such as Lines, circles and points etc. can be found and measured. Complex shapes can be identified and veri-fied.
Complete complex applications can be config-ured literally in minutes, and controlled remotely -using either a digital or a serial link.
The features of the system include:
~ Ease of Use ~ Calibrated Measurement Using Real World Units ~ Extensive Set of Inspection Commands ~ Efficient Use of Memory ~ Simple Digital and Serial Tnterfacing There are two phases in the operation of the system of the present invention:
~ Teach Phase: Here the user, using the computer keyboard and monitor, configures the system for his application e.g. part identification or measurement.
~ Execution Phase: Once configured, the teach system can run completely automati-cally and can be controlled via an RS232 serial link or it's digital inputs.
In the first phase the user "teaches" the system do a particular application. Using the computer CA 02304066 2000-03-16 ' keyboard 12 and monitor 10, the user "configures" the system for a particular application e.g. part identifi-cation, measurement etc. Use of the system is made easy by the menu-driven user interface which provides the user with all the commands in a structured and hierar-chical manner. Help is available at all points in the program.
The system stores the particular sequence of commands that make the application inta an application file. More than one application can be loaded at any one time.
In the second phase, applications can be executed at the press of a button on the keyboard, a remote input, or triggered through the RS232 or other communications line. Automatic- system diagnostics provide details of the performance of the application to the user, showing the nature and type of any problems.
All measurements are made in real world co-ordinates (m, mm, microns, i:n., thou., etc.).
Many commands are available to include metrol-ogy (measurement) functions and image analysis func-tions. Examples of metrology functions include the measurement of distances, angles, diameters, perimeter, areas, straightness, roundness and shape. Examples of vision functions include gray level edge finding (capa-ble of finding subtle grey level edges), projection, edge sniffing and feature correlation.
System functions can be divided into 3 main categories:
~ Image Analysis Functions: These func-tions determine features of an image ~ System Calibration: These functions allow the system to measure and work in a user defined co-ordinate frame ~ System I/O: These functions allow the system to respond to digital inputs, set digital outputs, communicate through a serial line, print information and save and retrieve information from files.
Image l4sialysis Furictions The image analysis functions can be sorted into six main categories:
~ Part locate Part locate identifies a point and orientation on the image. All search regions, e.g. lines, windows and ellipses that are defined after a part has been located are defined relative to this new part frame ~ Geometry definition These functions concern the defini-tion of the entities that the system uses during part inspection. Lines, ellipses, points and profiles can be defined using these functions.
~ Metrology These functions concern the measure-ment of various quantities between entities defined within the system.
Distances, areas, perimeters and angles can be measured. Some quan-tities can be measured between enti-ties of differing types, e.g. dis-tance from a point to a line. Geo-metric errors can also be measured.
These include the straightness of a line and roundness of a circle.
~ Morphology Measurement These functions concern the measure-ment of surface details, such as cracks, in the image.
~ Feature Correlation This function finds a pre-taught feature (model), to pixel accuracy within a grabbed image. The system calculates a "score" that shows the level of similarity between the model and the feature in the current image. The score is defined as a percentage, with 100% being a per-fect match.
~ Morphology enhancement These functions concern the enhance-ment of surface details, such as cracks, within the image.
Color Image Processing Calibration The purpose of system recalibration is to tell the vision system the units it is working in, i.e. mm, inches etc. Calibration defines the attitude of the camera 24 relative to the field of view. Camera cali-bration is accomplished by viewing a object of known shape and size, usually a square or round plate.
Calibration files are saved with the job file thus WO 99!14646 PCT/US98/17066 allowing different jobs to have different calibration data.
System calibration can be implemented using several methods. The first method/phase has been implemented using several methods. The full methods include:
Initial System Camera Calibration: This occurs whenever the camera 24 is moved, or every 6 months on system maintenance.
At calibration time an object of known size is placed in the field of view of each camera 24, its size and location being defined and identified by the system.
Monitoring Camera Movement and Automatic System Recalibration: This is the pro-cess whereby the system monitors camera movement, and compensates for such move-ment. The amount of movement observed is calculated, the calibration matrices adjusted and, if excessive, a warning is raised.
Continuous Automatic System Recalibra-tion: In certain instances system cali-bration can be accomplished completely automatically. The reference objects) is/are continuously in, or placed in, the field of view. The system, therefore, calibrates itself and returns information on camera movement. If the camera move-ment is so excessive that the reference object cannot be seen then a warning is WO 99/14646 PCTIUS98/1?066 raised. The system continues to operate using the last default calibration.
Menus This set of menus allows different geometric entities (points, lines, circles, ellipses and profiles) to be defined and selected. Such entities can either be defined using analysis of the video source or can be constructed using existing geometric entities.
Geometry Menu ZO Point Define Menu Point on Edge (Window Scan) Point Define via Feature Location Center of an Ellipse Line + Line Intersect Point on a Profile Inflection Point Deflection Point Line Def ine Menu Define by Scanning for Edge Define by Hand Define Via Two Points Profile Primary Vector Circle Define Menu Define by Scanning for an Edge Manual Profile Define Menu Metroloav Menus The metrology menus allow the user to perform measurements and implement tests based on the measure-ment results. The measurements and test results can be saved. Statistical Process Control (SPC) may be per-formed on the measured results. Metrology functions select geometric entries independent of where they are defined. When selecting an entity all geometry's are displayed, from which to select the required entity.
Linear Dimensions Distance Line to Line Distance Line to Point Distance Point to Point Menu Caliper:, Single Pair Caliper: Multiple Pairs Actual Distance Component Distance Circle/Ellipse Radius Angular Dimensions Angle Line to Line Angle Line Area Measurement Morphology Measurement Intensity Evaluation Connectivity Evaluation Geometric Errors Menu Parallelism Squareness Form Error Straightness Form Error Roundness Data Interface Several optional interfaces are suggested.
These interface are NOT compulsory as the interactive Grafcet/ActiveX environment can work without them.
However, if any of these interface are present this extends the functionality to the Grafcet/ActiveX envi-ronment.
These interface provides a set of defined methods and attributes obeying the COM standard.
Display Methods (iDiaplay) Draw Draws the tool ShowDisplayGraphicssplay result graphics HideDisplayGraphics Hide result graphics EnableAutoRefresh Enables the tool to refresh the result display upon use DisableAutoRefresh Disables the tool from refreshing the result display upon use IsAutoRefreshEnabled Returns if the tools auto refresh is enabled or dis-abled Information Methods (iInformation) GetMemorySize Returns the current quantity of heap, in bytes, which the tool is using SetMaximumTime Sets the maximum time allows for the use function to exe cute GetMinimumTime Returns the minimum time required for the use function to execute TimeTaken Returns the time taken by the Use function on the last use GetMaximumTime Returns the current setting for Use max-imum time allowed for the function to execute Data Access Methods (iData) GetlnputType Returns information regarding an input GetOutput Type Returns information regarding an output GetNumberOfInputs Returns the number of inputs GetNumberOfOutputs Returns the number of outputs GetOutput Returns a given out-put SetInput Set a given input Tool Methods(iTool}
AutoTrain Automatic Training method for the tool Train Initiate training of l0 the tool AddSample Add Sample to cur-rent training set for the tool PrepareToUse Prepares the tool for first use Use Uses the tool with the current trained information GetLastError Returns the last error generated by the tool Save Save tool configura-tion and training information to file Load Load tool configura-tion and training information to file Serialize Serializes the tool configuration to file While the best mode for carrying out the invention has been described in detail, those familiar with the art to which this invention relates will recognize various alternative designs and embodiments for practicing the invention as defined by the following claims.
Claims (32)
1. A method for interactively developing a graphical, control-flow structure and associated application software for use in a machine vision system utilizing a computer system, the structure including a control sequence having at least one node, and at least one transition, and at least one link for interconnecting the at least one node and the at least one transition, the method comprising the steps of:
storing an application development program, including a first set of control programs representing possible components of a user interface for the machine vision system, the first set of control programs defining a first set of standard controls;
staring a second set of control programs representing possible machine vision algorithms for the machine vision system, the second set of control programs defining a second set of standard controls;
storing hardware operating parameters corresponding to possible hardware for use in the machine vision system, the hardware operating parameters defining a third set of standard controls;
displaying graphical representations of the possible components, the possible hardware and the possible machine vision algorithms;
receiving commands from a user of the computer system to select a first control program corresponding to a desired component of the user interface, desired hardware operating parameters corresponding to desired hardware and a machine vision graphical representation and its associated second control program corresponding to a desired machine vision algorithm;
displaying the structure wherein the machine vision graphical representation is a node of the structure, the selected second control program being mapped into the structure at the node, the displayed structure also having at least one transition and at least one link for interconnecting the node and the at least one transition; and linking the first control program with the desired hardware operating parameters to the second control programs to form the application software in response to the commands without the user writing any of the application software.
storing an application development program, including a first set of control programs representing possible components of a user interface for the machine vision system, the first set of control programs defining a first set of standard controls;
staring a second set of control programs representing possible machine vision algorithms for the machine vision system, the second set of control programs defining a second set of standard controls;
storing hardware operating parameters corresponding to possible hardware for use in the machine vision system, the hardware operating parameters defining a third set of standard controls;
displaying graphical representations of the possible components, the possible hardware and the possible machine vision algorithms;
receiving commands from a user of the computer system to select a first control program corresponding to a desired component of the user interface, desired hardware operating parameters corresponding to desired hardware and a machine vision graphical representation and its associated second control program corresponding to a desired machine vision algorithm;
displaying the structure wherein the machine vision graphical representation is a node of the structure, the selected second control program being mapped into the structure at the node, the displayed structure also having at least one transition and at least one link for interconnecting the node and the at least one transition; and linking the first control program with the desired hardware operating parameters to the second control programs to form the application software in response to the commands without the user writing any of the application software.
2. The method as claimed in claim 1 wherein the standard controls include properties and wherein the step of linking includes the step of setting the properties of the standard controls.
3. The method as claimed in claim 2 wherein the step of setting includes the step of setting one property of one standard control to be equal to one property of another standard control.
4. The method as claimed in claim 1 wherein the structure has a hierarchy with a top and a bottom and wherein the hierarchy allows bottom-up or top-down development.
5. The method as claimed in claim 1 further comprising the steps of:
storing a third set of control programs representing possible software products for use in the machine vision system;
displaying machine vision graphical representations of the possible software products; and displaying the graphical representation of a selected software product in the structure as another node or an input/output to the node of the structure, wherein the step of linking links a third control program corresponding to a desired software product with the first and second control programs and the desired hardware operating parameters in response to the commands to form the application software.
storing a third set of control programs representing possible software products for use in the machine vision system;
displaying machine vision graphical representations of the possible software products; and displaying the graphical representation of a selected software product in the structure as another node or an input/output to the node of the structure, wherein the step of linking links a third control program corresponding to a desired software product with the first and second control programs and the desired hardware operating parameters in response to the commands to form the application software.
6. The method as claimed in claim 1 wherein the desired hardware operating parameters correspond to a desired image source of the machine vision system.
7. The method as claimed in claim 6 wherein the desired image source is a video camera.
8. The method as claimed in claim 6 wherein the desired operating parameters further correspond to a desired vision processor board of the machine vision system.
9. The method as claimed in claim 6 wherein the desired hardware operating parameters further correspond to a desired frame grabber board of the machine vision system.
10. The method as claimed in claim 6 wherein the desired hardware operating parameters further correspond to a desired motion board of the machine vision system.
11. The method as claimed in claim 6 wherein the desired hardware operating parameters further correspond to a desired bus of the machine vision system.
12. The method as claimed in claim 6 wherein the desired machine vision algorithm is an image processing algorithm.
13. The method as claimed in claim 6 wherein the desired machine vision algorithm is an image-analysis algorithm.
14. The method as claimed in claim 6 wherein the desired machine vision algorithm is a calibration space algorithm.
15. The method as claimed in claim 6 wherein the desired machine vision algorithm is an interactive CAD/geometry algorithm.
16. The method as claimed in claim 6 wherein the computer system includes a personal computer.
17. A computer system for interactively developing a graphical, control-flow structure and associated application software for use in a machine vision system, the structure including a control sequence having at least one node, and at least one transition, and at least one link for interconnecting the at least one node and the at least one transition, the computer system comprising:
means for storing an application development program, including a first set of control programs representing possible components of a user interface for the machine vision system, the first set of control programs defining a first set of standard controls;
means for storing a second set of control programs representing possible machine vision algorithms for the machine vision system, the second set of control programs defining a second set of standard controls;
means for staring hardware operating parameters corresponding to possible hardware for use in the machine vision system, the hardware operating parameters defining a third set of custom controls;
a display for displaying graphical representations of the possible components, the possible hardware and the possible machine vision algorithms;
means for receiving commands from a user of the computer system to select a first control program corresponding to a desired component of the user interface, desired hardware operating parameters corresponding to desired hardware and a machine vision graphical representation and its associated second control program corresponding to a desired machine vision algorithm, the display also displaying the structure wherein the machine vision graphical representation is a node of the structure, the selected second control program being mapped into the structure at the node, the displayed structure also having at least one transition and at least one link for interconnecting the node and the at least one transition; and means for linking the first control program with the desired hardware operating parameters to the second control program to form the application software in response to the commands without the user writing any of the application software.
means for storing an application development program, including a first set of control programs representing possible components of a user interface for the machine vision system, the first set of control programs defining a first set of standard controls;
means for storing a second set of control programs representing possible machine vision algorithms for the machine vision system, the second set of control programs defining a second set of standard controls;
means for staring hardware operating parameters corresponding to possible hardware for use in the machine vision system, the hardware operating parameters defining a third set of custom controls;
a display for displaying graphical representations of the possible components, the possible hardware and the possible machine vision algorithms;
means for receiving commands from a user of the computer system to select a first control program corresponding to a desired component of the user interface, desired hardware operating parameters corresponding to desired hardware and a machine vision graphical representation and its associated second control program corresponding to a desired machine vision algorithm, the display also displaying the structure wherein the machine vision graphical representation is a node of the structure, the selected second control program being mapped into the structure at the node, the displayed structure also having at least one transition and at least one link for interconnecting the node and the at least one transition; and means for linking the first control program with the desired hardware operating parameters to the second control program to form the application software in response to the commands without the user writing any of the application software.
18. The computer system as claimed in claim 17 wherein the standard controls include properties and wherein the means for linking includes means for setting the properties of the standard controls.
19. The computer system as claimed in claim 18 wherein the means for setting sets one property of one standard control to be equal to one property of another standard control.
20. The computer system as claimed in claim 17 wherein the structure has a hierarchy with a top and a bottom and wherein the hierarchy allows bottom-up or top-down development.
21. The computer system as claimed in claim 17 further comprising:
means for storing a third set of control programs representing possible software products for use in the machine vision system, the display displaying graphical representations of the possible software products and a graphical representation of a selected software product in the structure as another node or an input/output to the node of the structure, wherein the step of linking links a third control program corresponding to a desired software product with the first and second control programs and the desired hardware operating parameters in response to the commands to form the application software.
means for storing a third set of control programs representing possible software products for use in the machine vision system, the display displaying graphical representations of the possible software products and a graphical representation of a selected software product in the structure as another node or an input/output to the node of the structure, wherein the step of linking links a third control program corresponding to a desired software product with the first and second control programs and the desired hardware operating parameters in response to the commands to form the application software.
22. The computer system as claimed in claim 17, wherein the desired hardware operating parameters correspond to a desired image source of the machine vision system.
23. The computer system as claimed in claim 22 wherein the desired image source is a video camera.
24. The computer system as claimed in claim 22 wherein the desired operating parameters further correspond to a desired vision processor board of the machine vision system.
25. The computer system as claimed in claim 22 wherein the desired hardware operating parameters further correspond to a desired frame grabber board of the machine vision system.
26. The computer system as claimed in claim 22 wherein the desired hardware operating parameters further correspond to a desired motion board of the machine vision system.
27. The computer system as claimed in claim 22 wherein the desired hardware operating parameters further correspond to a desired bus of the machine vision system.
28. The computer system as claimed in claim 22 wherein the desired machine vision algorithm is an image processing algorithm.
29. The computer system as claimed in claim 22 wherein the desired machine vision algorithm is an image-analysis algorithm.
30. The computer system as claimed in claim 22 wherein the desired machine vision algorithm is a calibration space algorithm.
31. The computer system as claimed in claim 22 wherein the desired machine vision algorithm is an interactive CAD/geometry algorithm.
32. The computer system as claimed in claim 22 wherein the computer system includes a personal computer.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US08/931,227 US5940296A (en) | 1995-11-06 | 1997-09-16 | Method and system for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system |
| US08/931,227 | 1997-09-16 | ||
| PCT/US1998/017066 WO1999014646A1 (en) | 1997-09-16 | 1998-08-18 | Method and system for interactively developing a graphical control-flow and associated application software for machine vision system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2304066A1 true CA2304066A1 (en) | 1999-03-25 |
Family
ID=25460433
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002304066A Abandoned CA2304066A1 (en) | 1997-09-16 | 1998-08-18 | Method and system for interactively developing a graphical control-flow and associated application software for machine vision system |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US5940296A (en) |
| EP (1) | EP1023651A1 (en) |
| JP (1) | JP3773789B2 (en) |
| CA (1) | CA2304066A1 (en) |
| WO (1) | WO1999014646A1 (en) |
Families Citing this family (71)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5847953A (en) * | 1996-09-23 | 1998-12-08 | National Instruments Corporation | System and method for performing class checking of objects in a graphical data flow program |
| US5877957A (en) | 1996-11-06 | 1999-03-02 | Ameritech Services, Inc. | Method and system of programming at least one appliance to change state upon the occurrence of a trigger event |
| US6408429B1 (en) * | 1997-01-17 | 2002-06-18 | Cognex Corporation | Machine vision system for identifying and assessing features of an article |
| US6990652B1 (en) * | 1997-03-04 | 2006-01-24 | National Instruments Corporation | System and method for determining methods and properties to be invoked on objects in a graphical program |
| US6233703B1 (en) * | 1997-12-31 | 2001-05-15 | Triconex Corporation | Automatic generation of evaluation order for a function block diagram and detection of any associated errors |
| US7743362B2 (en) * | 1998-02-17 | 2010-06-22 | National Instruments Corporation | Automatic generation of application domain specific graphical programs |
| US6226783B1 (en) * | 1998-03-16 | 2001-05-01 | Acuity Imaging, Llc | Object oriented method of structuring a software step program |
| US6061602A (en) | 1998-06-23 | 2000-05-09 | Creative Lifestyles, Inc. | Method and apparatus for developing application software for home automation system |
| US7092860B1 (en) * | 1999-02-03 | 2006-08-15 | Mitutoyo Corporation | Hardware simulation systems and methods for vision inspection systems |
| US6298474B1 (en) * | 1999-04-30 | 2001-10-02 | Intergral Vision, Inc. | Method and system for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system and computer-readable storage medium having a program for executing the method |
| US7882426B1 (en) | 1999-08-09 | 2011-02-01 | Cognex Corporation | Conditional cell execution in electronic spreadsheets |
| US7937665B1 (en) | 2000-06-13 | 2011-05-03 | National Instruments Corporation | System and method for automatically generating a graphical program to implement a prototype |
| US7210117B2 (en) * | 1999-08-19 | 2007-04-24 | National Instruments Corporation | System and method for programmatically generating a graphical program in response to program information |
| US7505817B2 (en) * | 2000-03-06 | 2009-03-17 | Siemens Technology-To-Business Center, Llc | Programming automation by demonstration |
| EP1132863A1 (en) * | 2000-03-08 | 2001-09-12 | MVTec Software GmbH | Method and system for developing image processing applications |
| US8302072B2 (en) * | 2000-06-05 | 2012-10-30 | National Instruments Corporation | System and method for programmatically generating a graphical program based on a sequence of motion control, machine vision, and data acquisition (DAQ) operations |
| US6763515B1 (en) * | 2000-06-05 | 2004-07-13 | National Instruments Corporation | System and method for automatically generating a graphical program to perform an image processing algorithm |
| US7376904B2 (en) | 2000-06-13 | 2008-05-20 | National Instruments Corporation | Automatic generation of programs with GUI controls for interactively setting or viewing values |
| US8640027B2 (en) * | 2000-06-13 | 2014-01-28 | National Instruments Corporation | System and method for configuring a hardware device to execute a prototype |
| US7849416B2 (en) * | 2000-06-13 | 2010-12-07 | National Instruments Corporation | System and method for graphically creating a sequence of motion control, machine vision, and data acquisition (DAQ) operations |
| US7917863B2 (en) * | 2000-06-13 | 2011-03-29 | National Instruments Corporation | System and method for graphically creating a sequence of motion control operations |
| US7146347B1 (en) | 2000-08-03 | 2006-12-05 | National Instruments Corporation | System and method for automatically creating a prototype to perform a process |
| US7200838B2 (en) * | 2000-12-20 | 2007-04-03 | National Instruments Corporation | System and method for automatically generating a graphical program in response to a state diagram |
| US6931602B1 (en) * | 2000-12-22 | 2005-08-16 | Cognex Corporation | Approach facilitating the selection of various machine vision functionality from among different platforms |
| US8056009B2 (en) | 2001-06-02 | 2011-11-08 | Cognex Technology And Investment Corporation | System for initiating communication between a user interface and a vision processor |
| US7051317B2 (en) * | 2001-06-29 | 2006-05-23 | National Instruments Corporation | System and method for specifying a computer-implemented process using different programming methodologies |
| US7010779B2 (en) * | 2001-08-16 | 2006-03-07 | Knowledge Dynamics, Inc. | Parser, code generator, and data calculation and transformation engine for spreadsheet calculations |
| US7079141B2 (en) * | 2001-08-27 | 2006-07-18 | National Instruments Corporation | System and method for specifying a machine vision process using different programming methodologies |
| US7076332B2 (en) * | 2002-01-18 | 2006-07-11 | National Instruments Corporation | System and method for invoking execution of a sequence of operations that includes motion control, machine vision, and data acquisition (DAQ) functionality |
| US20030195762A1 (en) * | 2002-04-12 | 2003-10-16 | David Gleason | Automated workflow |
| US7139979B2 (en) * | 2002-06-10 | 2006-11-21 | National Instruments Corporation | Displaying operations in an application using a graphical programming representation |
| US7293112B2 (en) * | 2002-11-12 | 2007-11-06 | National Instruments Corporation | Graphical program node for displaying acquired images |
| CA2425719A1 (en) * | 2003-04-25 | 2004-10-25 | Dean Mills | Modular system and method for programming machine automation controls |
| US7496480B2 (en) * | 2003-08-15 | 2009-02-24 | National Instruments Corporation | Sweep manager for signal analysis |
| US7836426B2 (en) | 2004-05-06 | 2010-11-16 | National Instruments Corporation | Automatic generation of application domain specific graphical programs |
| US8453111B2 (en) * | 2004-05-14 | 2013-05-28 | National Instruments Corporation | Generating a hardware description for a programmable hardware element based on a graphical program including multiple models of computation |
| US8397214B2 (en) * | 2004-05-14 | 2013-03-12 | National Instruments Corporation | Generating a hardware description for a programmable hardware element based on a graphical program including multiple physical domains |
| CN104392292B (en) * | 2004-05-21 | 2019-07-26 | 派拉斯科技术公司 | Graphical re-inspection user setup interface |
| EP1603035A3 (en) * | 2004-06-03 | 2008-03-05 | MDL Information Systems, Inc. | Methods and apparatus for visual application design |
| WO2006087728A1 (en) * | 2005-02-15 | 2006-08-24 | Codito Technologies | System for creating parallel applications |
| US7761846B2 (en) * | 2005-08-16 | 2010-07-20 | National Instruments Corporation | Graphical programming methods for generation, control and routing of digital pulses |
| US7864178B2 (en) * | 2005-11-09 | 2011-01-04 | National Instruments Corporation | Creating machine vision inspections using a state diagram representation |
| US7975233B2 (en) * | 2006-07-24 | 2011-07-05 | National Instruments Corporation | Automatic conversion of a textual language into a graphical program representation |
| US7954059B2 (en) * | 2006-07-24 | 2011-05-31 | National Instruments Corporation | Automatic conversion of text-based code having function overloading and dynamic types into a graphical program for compiled execution |
| US20080126956A1 (en) * | 2006-08-04 | 2008-05-29 | Kodosky Jeffrey L | Asynchronous Wires for Graphical Programming |
| US8108784B2 (en) * | 2006-08-04 | 2012-01-31 | National Instruments Corporation | Configuring icons to represent data transfer functionality |
| US8028242B2 (en) * | 2006-08-04 | 2011-09-27 | National Instruments Corporation | Diagram with configurable wires |
| US8028241B2 (en) * | 2006-08-04 | 2011-09-27 | National Instruments Corporation | Graphical diagram wires whose appearance represents configured semantics |
| US8271943B2 (en) * | 2007-04-24 | 2012-09-18 | National Instruments Corporation | Automatically generating a graphical program with a plurality of models of computation |
| US7996782B2 (en) * | 2007-06-08 | 2011-08-09 | National Instruments Corporation | Data transfer indicator icon in a diagram |
| US7949422B1 (en) * | 2007-06-22 | 2011-05-24 | Vermont Machine Tool Corporation | Machine tool control system |
| KR101425668B1 (en) * | 2007-07-26 | 2014-08-04 | 페어차일드코리아반도체 주식회사 | Frequency modulation device and switch mode power supply using the same |
| US9492886B2 (en) * | 2008-11-21 | 2016-11-15 | Precitec Kg | Method and device for monitoring a laser processing operation to be performed on a workpiece and laser processing head having such a device |
| JP4767309B2 (en) * | 2008-11-28 | 2011-09-07 | キヤノンＩｔソリューションズ株式会社 | Information processing apparatus, information processing method, and computer program |
| US8423981B2 (en) * | 2009-06-18 | 2013-04-16 | National Instruments Corporation | Compiling a graphical program having a textual language program portion for a real time target |
| US8458653B2 (en) * | 2009-09-29 | 2013-06-04 | National Instruments Corporation | Debugging a graphical program deployed on a programmable hardware element |
| KR101853694B1 (en) | 2011-08-29 | 2018-05-02 | 한화테크윈 주식회사 | One modulation apparatus using carrier board |
| US20130104101A1 (en) * | 2011-09-16 | 2013-04-25 | Mitsubishi Electric Corporation | Sequence program creating apparatus |
| US8612637B2 (en) | 2011-09-25 | 2013-12-17 | National Instruments Corportion | Configuring buffers with timing information |
| US9200890B2 (en) | 2012-05-22 | 2015-12-01 | Cognex Corporation | Machine vision systems and methods with predictive motion control |
| US10083496B2 (en) | 2012-05-22 | 2018-09-25 | Cognex Corporation | Machine vision systems and methods with predictive motion control |
| FR3001553B1 (en) * | 2013-01-31 | 2018-11-02 | Wesby Sarl | CONTROL DEVICE FOR AN AUTOMATION SYSTEM |
| JP2014203309A (en) * | 2013-04-05 | 2014-10-27 | オムロン株式会社 | Image processing apparatus, control method, and program |
| US10436977B2 (en) | 2013-12-11 | 2019-10-08 | Ademco Inc. | Building automation system setup using a remote control device |
| JP6350037B2 (en) * | 2014-06-30 | 2018-07-04 | 株式会社安川電機 | Robot simulator and robot simulator file generation method |
| CN107040709A (en) | 2015-11-12 | 2017-08-11 | 精工爱普生株式会社 | image processing apparatus, robot system, robot and image processing method |
| US10488062B2 (en) | 2016-07-22 | 2019-11-26 | Ademco Inc. | Geofence plus schedule for a building controller |
| CN107293995B (en) * | 2017-06-20 | 2018-09-25 | 南京理工大学 | Transmission line of electricity mechanical deicing order optimization method based on differential evolution algorithm |
| CN110673844A (en) * | 2019-09-26 | 2020-01-10 | 苏州中科全象智能科技有限公司 | Image processing software development method and system |
| CN110706147B (en) * | 2019-09-29 | 2023-08-11 | 阿波罗智联(北京)科技有限公司 | Image processing environment determination method, device, electronic equipment and storage medium |
| BE1029306B1 (en) * | 2021-04-30 | 2023-07-14 | Zebra Technologies | INDUSTRIAL ETHERNET CONFIGURATION TOOL WITH PREVIEW FUNCTIONS |
Family Cites Families (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4462046A (en) * | 1982-07-02 | 1984-07-24 | Amaf Industries Incorporated | Machine vision system utilizing programmable optical parallel processing |
| US4679137A (en) * | 1985-04-30 | 1987-07-07 | Prometrix Corporation | Process control interface system for designer and operator |
| JPS62232006A (en) * | 1986-04-02 | 1987-10-12 | Yokogawa Electric Corp | Robot system |
| US4759074A (en) * | 1986-10-28 | 1988-07-19 | General Motors Corporation | Method for automatically inspecting parts utilizing machine vision and system utilizing same |
| US5005119A (en) * | 1987-03-02 | 1991-04-02 | General Electric Company | User interactive control of computer programs and corresponding versions of input/output data flow |
| US4852047A (en) * | 1987-04-14 | 1989-07-25 | Universal Automation Inc. | Continuous flow chart, improved data format and debugging system for programming and operation of machines |
| US4914567A (en) * | 1987-11-02 | 1990-04-03 | Savoir | Design system using visual language |
| JP2624753B2 (en) * | 1988-03-18 | 1997-06-25 | 株式会社日立製作所 | How to create higher-level specifications |
| JP2538069B2 (en) * | 1989-09-13 | 1996-09-25 | 株式会社日立製作所 | Plant control system, control circuit display method and maintenance device |
| JP2978189B2 (en) * | 1989-11-16 | 1999-11-15 | シャープ株式会社 | Automatic flowchart creation device |
| JP2526688B2 (en) * | 1990-02-22 | 1996-08-21 | 三菱電機株式会社 | Programmable controller and partial execution method of sequence program |
| JPH03265906A (en) * | 1990-03-15 | 1991-11-27 | Fanuc Ltd | Programming system for interactive numerical controller |
| DE69122762T2 (en) * | 1990-08-25 | 1997-05-15 | Intelligent Automation Systems | PROGRAMMABLE, REGROUPABLE FEEDING DEVICE FOR WORKPIECES |
| JPH04181403A (en) * | 1990-11-16 | 1992-06-29 | Fanuc Ltd | Interactive numerical controller |
| FI921220A (en) * | 1992-03-20 | 1993-09-21 | Rautaruukki Oy | ANORDNING FOER ATT GENERERA STRAOLNING |
| US5517663A (en) * | 1993-03-22 | 1996-05-14 | Kahn; Kenneth M. | Animated user interface for computer program creation, control and execution |
| US5481712A (en) * | 1993-04-06 | 1996-01-02 | Cognex Corporation | Method and apparatus for interactively generating a computer program for machine vision analysis of an object |
| JP3446256B2 (en) * | 1993-09-03 | 2003-09-16 | 株式会社日立製作所 | Control method and apparatus for FA system |
| US5453933A (en) * | 1993-09-08 | 1995-09-26 | Hurco Companies, Inc. | CNC control system |
| US5576946A (en) * | 1993-09-30 | 1996-11-19 | Fluid Air, Inc. | Icon based process design and control system |
| US5511147A (en) * | 1994-01-12 | 1996-04-23 | Uti Corporation | Graphical interface for robot |
| US5574828A (en) * | 1994-04-28 | 1996-11-12 | Tmrc | Expert system for generating guideline-based information tools |
-
1997
- 1997-09-16 US US08/931,227 patent/US5940296A/en not_active Expired - Lifetime
-
1998
- 1998-08-18 WO PCT/US1998/017066 patent/WO1999014646A1/en not_active Application Discontinuation
- 1998-08-18 EP EP98940961A patent/EP1023651A1/en not_active Withdrawn
- 1998-08-18 CA CA002304066A patent/CA2304066A1/en not_active Abandoned
- 1998-08-18 JP JP2000512118A patent/JP3773789B2/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| US5940296A (en) | 1999-08-17 |
| JP2001516910A (en) | 2001-10-02 |
| WO1999014646A1 (en) | 1999-03-25 |
| EP1023651A1 (en) | 2000-08-02 |
| JP3773789B2 (en) | 2006-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5940296A (en) | Method and system for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system | |
| US6298474B1 (en) | Method and system for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system and computer-readable storage medium having a program for executing the method | |
| US8640027B2 (en) | System and method for configuring a hardware device to execute a prototype | |
| US7543281B2 (en) | Disabling and conditionally compiling graphical code in a graphical program | |
| US5940294A (en) | System for assisting configuring a process control environment | |
| US7302675B2 (en) | System and method for analyzing a graphical program using debugging graphical programs | |
| JP5762716B2 (en) | How to select a shape in a graphic display | |
| US7139979B2 (en) | Displaying operations in an application using a graphical programming representation | |
| US7444595B2 (en) | Graphical programming system and method for creating and managing a scene graph | |
| EP0597316A2 (en) | Computer simulation system and method for specifying the behavior of graphical operator interfaces | |
| JP2012164320A (en) | Associated graphic display in process environment | |
| US7051317B2 (en) | System and method for specifying a computer-implemented process using different programming methodologies | |
| Ichikawa et al. | Visual programming—Toward realization of user-friendly programming environments | |
| US7079141B2 (en) | System and method for specifying a machine vision process using different programming methodologies | |
| Gordon et al. | An extensible visual environment for construction and analysis of hierarchically-structured models of resource contention systems | |
| Naylor et al. | PROGRESS--A graphical robot programming system | |
| US8671394B1 (en) | Monitoring test steps in test environments | |
| US20090013308A1 (en) | Programming interface for computer programming | |
| Eckstein et al. | Interactive data inspection and program development for computer vision | |
| Årén | A survey of commercial real-time expert system environments | |
| Lee et al. | User-interface development tools | |
| Chang et al. | The Research Queuing Package Modeling Environment (RESQME) | |
| Piegl | Rational B-spline curves and surfaces for CAD and graphics | |
| Eckstein et al. | Architecture for computer vision application development within the HORUS system | |
| Radig et al. | Automatization in the design of image understanding systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Discontinued |