DE10164944B4 - Apparatus and method for correcting the movement of gripping and machining tools - Google Patents

Abstract

Method for three-dimensional correction of the relative movement with several degrees of freedom between workpieces on the one hand, and grippers or tools on the other hand, with an image pickup device of one or more cameras, wherein the image pickup device and / or the workpiece is reproducibly movable - with a Einrichtbetrieb with the following steps a) b) arranging a workpiece in a systematically selected or random standard position, c) arranging the image recording device in one or more regions of the workpiece, via at least one imaging beam path associated with the regions Starting position, d) recording at least one image via each imaging beam path, e) storing the images from step d) and / or data derived therefrom, and f) arranging the image recording device or the workpiece in relative positions to the starting position e or standard position, wherein each degree of freedom is varied at least once compared to the starting position or standard position, wherein for each relative position: - the relative position ...

Description

The invention relates to a device and a method for correcting the movement of gripping or machining tools relative to articles in an unknown or inaccurate position, in particular by means of robots.

When the items are presented in fixed position and orientation, robotic gripping or manipulation can be done without modification of a once programmed movement. Is the location of the items uncertain, z. B. due to uncertainties of parts recording, it must be corrected for gripping or editing the movement of the gripping or machining tool and thus the robot.

Examples of gripping are removal from molded films, shelves or wire mesh boxes. Examples of the processing are grinding or adhesive application; for machining are also to count the mounting and joining, such as the mounting of attachments such as washers or wheels on automobile bodies or the insertion or welding of bolts.

For a better understanding of the invention, terms used herein will be explained in more detail below.

Since the objects are mostly workpieces, we shall speak of workpieces in the following, and in the general sense of the word, of any precision concerning the application of sufficiently rigid objects.

Depending on the task, the term "tool" generally refers to either the gripper or the machining tool.

As a location in the following, the three-dimensional position and orientation in space (English "position and attitude") referred to. The position of a workpiece is described by at least 6 parameters, i. d. R. by three translational and three rotational, with which a workpiece-fixed coordinate system in relation to fixed spatial coordinates.

The multidimensional finite space of the position deviations possible in the present task (i.al., a 6-dimensional hyper-cube) is referred to below as a working area. The more accurate the mounting of the workpieces, the smaller the working area may be.

The term robot is to be understood here generally in the sense of a mechanical or virtual system with programmable or algorithmically controllable or controllable movement. The movement alters the relative position of the tool and workpiece, either tool against workpiece or vice versa or both.

The number of degrees of freedom means the number of axes of the robot used. Normally there are 6 degrees of freedom. However, depending on the situation, more degrees of freedom are possible or less. There may be robot axes whose motion / position does not have to be changed to solve the problem set here; they do not count here to the degrees of freedom.

In setup mode (offline operation) of a system - i. d. R. under human supervision or leadership - the system parameterized for the application, including z. B. Teaching a robot track, teaching an image analysis system and calibration, if necessary.

In automatic mode (online operation), the system operates automatically or semi-automatically according to the task at hand.

In the following, we understand the standard position of a workpiece to be a random or specially selected position for which the robot is to be taught. The task is to correctly grasp or process the workpiece even in a position deviating from this standard position. The default situation does not need to be in any known relationship to the space coordinates. In a nutshell, you could throw the piece on a table, and use the resulting coincidental and geometrically unspecified position as a standard for teaching.

The starting point is a position of system components which is generally specified during teaching, in which the first image recordings are carried out in automatic mode.

As fixed features (here specially introduced term) are those features of a workpiece that are stationary on the workpieces are available, such as "naturally" existing forms such as edges, corners, holes, beads, but also artificial brands (factory printed stationary attached imprints, Deposits, adhesive dots or workpiece-related stationary projections such as laser spots or laser patterns).

Fix features are also those that produce a pattern with the help of structured light, which is fixed to the workpiece stationary. This can be useful for increasing the collision safety: for example, you can to workpiece edges To render more reliably, to project a thin line pattern onto the surface, and to emphasize the regions with striped patterns by means of a customized software filter, to obtain a secure figure-background separation (there is no striped pattern with matching line spacing on the background).

Excellent points are those fixed features which are represented by dots, e.g. B. the center of a circle or a vertex of a contour, given as the intersection of two straight pieces.

In addition to the workpiece-related stationary features can not stationary, lighting technology generated features are used. These are referred to in the following as Flex features (here specially introduced term).

Examples of flex features:

• a) Woelbungen: As the camera moves against the work piece and directed illumination, the reflection edges on the illumination wander relative to the work piece and also deform. They are not fixed to the workpiece but are reproducible and suitable for obtaining the position information.
• b) projection of light patterns (usually realized by laser): the resulting shapes can be used for three-dimensional analysis; Since normally the lighting is not moved parallel to the work piece, the shapes are not fixed to the work piece.

Characteristics are fixed features or flex features.

One feature is i. a. in three dimensions; the mapping of a feature area comprising a feature into a two-dimensional coordinate system via a ray beam, also referred to below as an imaging ray path, is referred to as a feature image.

Such a feature image can be further transformed into a more compact description in the form of a t-dimensional image description vector of values, w1, w2, ... wt, with t> = 1. As a rule, t> = 2.

The components of the image description vector are measures that capture characteristics of mapping features in feature images. The simplest example, with t = 2, is the measurement of the center of gravity of the image of a hole, which is described by the two center of gravity image coordinates. Another example is the calculation of displacement and rotation of an arbitrarily structured image pattern via correlation methods (t = 3). As will be explained below on the basis of the exemplary embodiments of the invention, masses are also suitable for the image description vector, such as size change, brightness and color change, spatial frequency changes - ie values which do not describe any shift or rotation in image coordinates.

The trivial way of determining a feature vector from a gray value feature image of size nx by ny pixels is simply to translate all pixel gray values directly into an image description vector with nx by ny components.

By combining several image description vectors which are to be assigned to the same (in the first instance unknown) mechanical workpiece recording situation (eg the starting position), a vector is created, which is referred to below as a positional description vector.

Mind you, the position description vector does not describe the position deviation to be determined directly, but implicitly!

A correction of the robot movement can be derived from the deviation of the current position from the standard position.

• a) modellbasierte Methoden,
• b) dreidimensional messende Systeme.

This deviation is determined according to the conventional procedure by defining a workpiece-own coordinate system and by determining the absolute position of the workpiece for both the standard position and the current position. For the determination of an absolute position by means of optical sensors, especially image processing systems, two different methods are indicated in the literature:

• a) model-based methods,
• b) three-dimensional systems.

Model-based systems use the familiar geometry of fix features in the workpiece's coordinate system.

Model-based methods include, for example, [GMR], [Fan], [Isr]. In all three examples, the reference to the characteristics of a body coordinate system must be known. [GMR] uses points or points and lines, one or more cameras. [Isr] uses only lines, three or more cameras. [Fan] uses points, three or more cameras, plus a calibrator to be positioned in front of each camera for calibration.

• – die Lage der Merkmale im werkstueckseigenen Koordinatensystem muss bekannt sein,
• – die Kamera muss aufwendig kalibriert werden,
• – sie koennen grundsaetzlich keine Flexmerkmale handhaben.

Disadvantages of the model-based methods are:

• - the location of the features in the workpiece coordinate system must be known,
• - The camera must be calibrated consuming,
• - they basically can not handle flex features.

In three-dimensional measuring systems, the part geometry need not be known; with them, for example, handmade models are scanned to generate CAD data for series production.

Another example is the web control by means of light section method.

• b1) punktweise Abstandsmessung ueber gepulstes oder moduliertes Licht, zusammen mit Messung der Licht-Laufzeit bzw. Phasenverschiebung,
• b2) Triangulation ueber mindestens eine Kamera und strukturiertes Licht,
• b3) Triangulation ueber mindestens 2 Kameras (Stereo).

The measuring methods for three-dimensional measuring systems are:

• b1) Pointwise distance measurement via pulsed or modulated light, together with measurement of the light transit time or phase shift,
• b2) triangulation via at least one camera and structured light,
• b3) Triangulation over at least 2 cameras (stereo).

Disadvantageously, the method b1) is associated with a very high expenditure on equipment and requires a complex, stable measurement technique.

Disadvantageously, the method b2) is associated with high expenditure on equipment and, moreover, requires a very high outlay in the calibration of the systems; Camera and lighting geometry must be known and very stable.

In measuring systems for scanning surfaces according to method b2), the sensor can be moved in a defined manner in order to expand the measuring range of such systems in connection with a known mechanical movement for large-area workpieces:
After [Dif], the sensor is either moved according to a previously known ideal shape and the (b2) sensor measures deviations therefrom, or the sensor serves as a zero indicator to produce a controlled movement at a constant distance from the surface, this movement serves as a measurement result.

According to [Per], a robot is used to guide a (b2) sensor, whereby the generally insufficient accuracy of the robot is determined by an additional device (eg additional "photogrammetry" system) which is independent of the robot and the sensor. Kinematic correction module ") increases.

For stereo methods (b3), the same characteristics are recorded by at least 2 cameras each; From the disparity of the representation of the features in the images and from the known data of the camera geometry, the three-dimensional position of the features is calculated. The workpiece geometry and the location of the features in the workpiece coordinate system generally need not be known.

• – entweder die Bildfelder sehr gross werden, was nun doch wiederum zu Genauigkeitsproblemen wegen geringer Bildpunktaufloesung fuehrt,
• – oder dass fuer weit auseinanderliegende Merkmale je zwei Kameras erforderlich sind, fuer drei weit auseinander liegende Merkmale also insgesamt mindestens 6 Kameras (s. z. B. [Lee]).

A disadvantage of the stereo method is that different cameras must capture the same characteristics. This results in a problem with large workpieces: In order to achieve the required rotational accuracy, several possible far apart features must be recorded. That leads to that

• Either the image fields become very large, which in turn leads to accuracy problems due to low pixel resolution,
• - or that two cameras are required for widely separated features, for a total of at least 6 cameras for three widely separated features (see eg [Lee]).

In addition, the features should not change with changing lighting and viewing geometry. Flex features are therefore often prohibited.

Stereo methods are described in detail in [Kle].

A serious disadvantage of the triangulating measuring methods (b2 and b3) is the need for accurate calibration, camera (s) and possibly structured lighting regarding; In addition, the structure must be very stable in order to maintain the required measurement accuracy even in harsh industrial operation.

Note: The calibration of a camera describes the imaging geometry; one usually distinguishes external and internal parameters. The external parameters describe the position of the projection center and the optical axis in space, as well as the (scalar) distance between the projection center and the imaging plane. The internal parameters describe the screw position of the imaging plane to the optical axis and the position of the piercing point of the optical axis through the image plane relative to the image frame.

In recent years, the field of work "Visual Servoing", in the following called servo method [Hut]. Here an attempt is made to realize a control of robots with uncalibrated or unequally calibrated systems via optical feedback. During the robot movement an attempt is made to bring the position of the features in the feature images as accurately as possible to desired. This implies the possibility of calculating the position of features in the image area.

Note: in contrast to a control, the feedback of a measured variable (here the position of the features in the feature images) in a closed-loop control affects a manipulated variable (in this case the robot position). In servo processes, continuous image acquisitions are therefore required during the movement in order to minimize the deviation of the measured variables from the desired values during the movement.

In servo method the following case distinction can be made:

Case 1: Hand-eye coordination.

Hand-eye coordination moves the tool relative to the camera (s).

The camera captures the tool or parts thereof or marks or mechanical parts that are geometrically related to the tool.

Distinction after movement:

• Case 1a: The tool is moved and the camera is locked.
• Case 1b: The tool is locked and the camera is moved.
• Case 1c: The tool and the camera are both moved, for example, different degrees of freedom can be distributed to the tool and camera.

Distinction after image acquisition:

• Case 1x: The camera records the work piece and the tool (standard case).
• Case 1y: The camera detects the work piece and not the tool.
• Case 1z: The camera does not pick up the workpiece and detects the tool.

In the case of 1x, the robot position can be controlled or regulated with respect to the current workpiece position.

In case 1y, the robot position can be controlled with respect to the current workpiece position.

In case 1z only a regulation of the robot movement per se can be realized, but not in relation to the current workpiece position.

Case 2: Eye-in-Hand:

With eye-in-hand, the camera (s) are moved parallel to the tool relative to the workpiece.

Distinction after movement:

• Case 2a: The tool and cameras are moved and the workpiece stands still.
• Case 2b: The tool and cameras are fixed and the workpiece is moved.
• Case 2c: The tool and cameras are moved together, and the workpiece is moved, for example, various degrees of freedom can be distributed to tool and camera on the one hand and workpiece on the other hand.

Distinction after image acquisition:

• Case 2x: The camera captures the workpiece and the tool.
• Case 2y: The camera detects the work and does not detect the tool.

In the case of 2x, the robot position can be controlled or regulated with respect to the current workpiece position, the situation corresponds approximately to the case 1x.

In case 2y, the robot position can be controlled or regulated with respect to the current workpiece position.

In order to optimize the control in servo processes, an attempt is made to model the relationship between movement of the robot axes and the local displacement of the image of fix features.

• – Stereo,
• – Fall 1x,
• – Tracking ohne exakte Tiefenschaetzung,

mit den dementsprechenden Einschraenkungen.The precursors of the servo processes are simple, controlling systems, each of which considers one of the following trout:

• - stereo,
• - case 1x,
• - tracking without exact depth estimation,

with the corresponding restrictions.

The linear approximation of this relationship around an operating point is described by the so-called visual Jakobi matrix. For hand-eye coordination, it has been proposed to obtain the Jakobi matrix by performing a set of orthogonal robot motions in setup mode while observing the motion of the position of the mapping of the features [Ja1].

Various known methods of function approximation have been proposed: piecewise linear approximation, piecewise with splines [Ja0], "quasi-newton" method [pie].

In all these approaches, during the control process, a hand-to-eye Coordinating task tries to deduce a guideline for the optimal movement by the approximation, in order to be able to carry out with the next image acquisition a further, hoped for, more exact, estimation.

Servo methods have the following disadvantages:
Servo processes can work in case 1x or case 2x without calibration or with very inaccurate calibration. However, in case 2y included herein, we require accurate calibration for purposes of workpiece handling or workpiece processing.

• a) analytisch [Mar]: das wiederum setzt Kenntnis der Kamerageometrie voraus (d. h. modellbasiert),
• b) durch online Schaetzung; Problem ist die Schaetzung der Entfernungsabhaengigkeit der Parameter in der Jacobimatrix ([Hut] S. 26); fuer das Tracking kommt man mit einer sehr groben Schaetzung dieser Parameter aus, indem man die Zielpositionen in die Naehe der optischen Achse verlegt. Ohne zuverlaessige Abstandsinformation sind jedoch Aufgaben der Werkstueckhandhabung oder Werkstueckbearbeitung nicht loesbar.

For purposes of tracking by eye-in-hand systems with servo method, the movement of the position of the mapping of features is determined depending on the camera movement

• a) analytic [Mar]: this in turn requires knowledge of camera geometry (ie model-based),
• b) by online appraisal; Problem is the estimation of the distance dependence of the parameters in the Jacobian matrix ([Hut] p. 26); For the tracking one comes with a very rough estimation of these parameters, by moving the target positions near the optical axis. Without reliable distance information, however, tasks of workpiece handling or workpiece processing are not solvable.

Servo methods rely on the explicit derivation of position information from the images, as can be realized, for example, when using excellent points. Without such explicit position information no visual servoing can be realized.

In addition, in the servo method due to the regulatory procedure in automatic mode, the current recording and evaluation of images is required, and thus in particular a very fast evaluation. This limits the approach to relatively simple or hardware-supported image evaluation methods.

The DE 199 30 087 A1 relates to a method and a device for controlling the advance position of a manipulator of a handling device. In detail, the advance position of the manipulator, which can be moved in several degrees of freedom, of a handling device is regulated relative to an object along a travel path. In this case, a plurality of points of at least part of the actual trajectory are stored and used for a predictive control of the advance position of the manipulator along this part of the trajectory. During a training phase, the manipulator is moved by a desired advance position by one degree of freedom in different Vorhaltepositionen relative to the object. Sensors, which can be attached to the manipulator, provide measurement data on the basis of which a holding position of the manipulator is characterized. These measurement data can also be image measurement data obtained by measuring images. A later, in the context of an operation mode carried out control of the hold position is carried out in dependence on the stored measurement data.

The DE 198 14 779 A1 relates to a method and a device for controlling the movement of a movable first object relative to a second object, by means of an optical detection of the second object or a part thereof by an optical detection system, with recognition of a given on the second object pattern or a structure thereon, wherein comparison patterns are stored in a memory device, wherein the first and second objects are approached and engaged with each other from a remote position.

The object of the invention is to provide a method and a device for correcting the movement of gripping or machining tools relative to workpieces.

The object is achieved according to the invention according to claim 1 or 17.

• – ohne eine Kamerakalibrierung durchfuehren zu muessen,
• – ohne die Geometrie des Werkstueckes oder seiner Merkmale kennen zu muessen,
• – ohne ueberhaupt ueber positionsmaessig erfassbare Merkmale in den Bildern verfuegen zu muessen (bei Servoverfahren nach bisherigen Ansaetzen erforderlich); es sind keine Ausgezeichneten Punkte erforderlich.
• – unter Vermeidung des oben angegebenen Nachteils von Stereosystemen, dass grosse Werkstuecke nur ungenau oder mit sehr vielen Kameras gehandhabt werden koennen.

The invention enables such a correction

• - without having to perform a camera calibration,
• - without knowing the geometry of the work or its characteristics,
• - without having to have any positionally detectable features in the images (in servo processes according to previous approaches required); no points are required.
• - Avoiding the above-mentioned disadvantage of stereo systems that large workpieces can be handled only inaccurate or with very many cameras.

• a) zum Greifen: die Modifikation einer vorgeteachten Bewegung von einer meist festen Ausgangslage bis zur Greifposition fuer das aktuelle Werkstueck,
• a) zum Bearbeiten: die Modifikation einer vorgeteachten Bearbeitungsbahn.

The correction of the robot movement is a geometric transformation of a generally three-dimensional movement

• a) for gripping: the modification of a pre-noted movement from a mostly fixed starting position to the gripping position for the current workpiece,
• a) for editing: the modification of a pre-taught machining path.

The modification is typically implemented as an additive correction (displacement, rotation) on the pre-gripped gripping position or the pre-scanned processing path. In special cases, it may be useful to derive a non-linear distortion of the movement.

The method described here can also be used for the pre-correction of the movement in order to enable a more accurate fine correction in conjunction with other methods, in particular additional sensors, z. B. for the sweat path correction.

In the following preferred embodiments of the invention will be described with reference to the drawings, in which:

1 shows a schematic diagram of a first embodiment according to the invention,

2 shows a schematic diagram of a second embodiment according to the invention,

3 FIG. 2 shows a schematic representation of a third embodiment according to the invention, FIG.

4 shows a schematic diagram of a fourth embodiment according to the invention,

5 a schematic diagram of a fifth embodiment according to the invention shows.

1 shows an arrangement with three cameras 10 . 11 . 12 and a beam beam or imaging beam path 20 . 21 . 22 per camera. The features are fixed features (holes, contour) 30 . 31 . 32 on the work piece 2 , The cameras 10 . 11 . 12 are at the tool 1 attached. Either the tool becomes 1 moved with the cameras or the workpiece 2 is moved.

The fixed features 30 . 31 . 32 Although here are representable as points, but no point coordinates must be calculated, it suffice simple image comparison methods to determine the Werkstuecklage. Without departing from the characteristics of the invention, of course, coordinate values for position description vectors (definition see above) can also be used.

2 shows the principle of the same arrangement, but only two cameras 10 . 11 intended. There are two ray tubes 20a . 20b with a single camera 10 realized.

3 shows the use of a reflex from a light source 40 on a smooth, convex surface, as an example of a flex feature 34 , The reflex is on the surface as a spot, the movement of the workpiece 2 but it is suitable for deriving the work piece position from it.

It is not necessary to know the position or the size nor the shape of the spot; it suffices the fact that the spot is moving from the perspective of the camera when moving the workpiece 10 reproducibly changed.

4 shows the use of structured light from light sources 50 . 51 . 52 , for the generation of flex features. The flex features 60 . 61 . 62 are projected light patterns with some structure that does not need to be further specified; neither the geometric shape nor any dimensions need to be known. For the example, a "wild" dot pattern, a circle and a rectangle were used arbitrarily. Such flex characteristics are for the location of the workpiece 2 significant.

5 shows the same arrangement as 4 However, the projecting device is 50 . 51 . 52 not freaky, but tangentially aligned. The projecting device 50 . 51 . 52 is wide-angle, so when moving the workpiece 2 A distortion of the projection patterns results within the viewing beam, which, while fundamentally sufficient for compliance with the significance condition, is less recommendable in terms of stability and accuracy than the strict observance of the ban on tangents.

The calculation of the current position can be derived directly from the stored images, if one scans the working area in all degrees of freedom with sufficiently small steps in setup mode and takes pictures, stores the feature images and searches in automatic mode for the situation with the most similar set of images. This brute-force method only requires that different k-tuples (k = number of imaging beam lengths or ray bundles) of feature images result reproducibly in different positions, which can be uniquely assigned to the position information within the working range. This condition is simply called the significance condition in the following.

Compliance with the significance condition is a generally necessary requirement of the invention and will be discussed below.

The method preferably results in searching a more or less large database of location description vectors. A large, high-dimensional database may be required, especially when working without interpolation and long vectors. For the efficient search of such databases, methods from the field of database systems are available.

Regarding operation in automatic mode, one can distinguish:

- After scan:

• - Search in a database of location description vectors, which results from fine scanning of the work area.
• - Interpolating work with a database, which results from rough scanning of the work area.

In the latter case, the database is much smaller because the work area was only roughly scanned. Subsequently, the position is determined from several adjacent position candidates, by interpolating the similarity values.

- By database content:

• - Working with large position description vectors in the form of feature images
• - Working with more compact position description vectors (in principle, only 6 vector components are required to realize a position correction with only one image acquisition per bundle or imaging beam path with 6 degrees of freedom).

These methods can also be combined with each other.

The interpolation with pure image data can be realized under certain conditions: the image contents must be structured as far as possible in different directions, coarse and, if possible, also fine structures should be present).

The two-dimensional coordinate systems associated with the ray bundles (also called the imaging beam path) are normally (but not necessarily) plane; they can also be coplanar. Different raybands are either realized by different cameras ( 1 ) or realized by the same camera ( 2 there includes camera 11 the bundle 31 and camera 10 the bundles 20a and 20b ).

The areas may basically overlap, but are advantageously implemented disjointly or with only slight overlap.

The delineation of the beam can be done by mechanically limiting image fields, but also - especially in the realization of multiple beams by the same camera - procedurally by splitting the entire camera image in areas. The image sections (feature images) corresponding to the areas or data derived therefrom are stored in set-up mode or further processed in automatic mode.

If all the beams are realized by the same camera, to comply with the significance condition, it is necessary that not all the beams be parallel; a telecentric beam path is excluded in this case.

In the case of central projection, the significance of the significance condition is more certain the better the tangency prohibition is fulfilled; this, in turn, is more satisfiable when using a single camera; the more wide-angle the lens is. In practice, in this case, an object field angle of more than about 20 degrees is sufficient; preferably at least 45 degrees; At approx. 90 degrees of the field angle, the tangent ban is best fulfilled: then workpiece areas lying in one plane can all be cut at approximately 45 degrees to the tangent.

The same applies to the use of a total of two cameras, wherein two beams are realized by one of these two cameras: the tangent ban is the more satisfiable, the more wide-angle the lens of this camera is.

If several beams are realized by the same camera, it is also sufficient not explicitly, but implicitly perform the picking out of the workpiece areas of this camera: The image comparison of two workpiece areas with the corresponding areas from the setup mode can summarize in a single step with one image from Einrichtbetrieb and automatic operation can be realized.

In this comparison harmless other parts of the picture may be included outside the realms; You can then z. B. also work with overall images, because if the image areas that do not meet the tangent condition, do not change when moving the image pickup device against the workpiece, these image parts for image analysis, although no contribution, but it is not harmful. In this case, the combined areas are addressed as a single area, which is imaged via an associated beam path.

In a special case, an area may also comprise the entire part of the workpiece surface visible from the point of view of the relevant camera.

The raybands do not necessarily correspond to a central projection, as in 1 shown; for example, they can also be parallel, as with the use of telecentric lenses (parallel projection).

The limitation of the ray bundles is not necessarily circular, as in 1 shown; the evaluated areas can be delimited arbitrarily.

Different beam bundles can also be realized with the same camera in different positions and possibly in different settings (eg zoom), one after the other.

In the context of the present invention, the image recordings can take place both at standstill and during the movement at times when the system components are in the relevant positions.

The process according to the invention can be used without restriction in both cases. The practice of industrial image processing offers many proven possibilities of image acquisition in motion, by the z. B. the problem of motion blur is prevented (eg flash lighting, Shutter technology, CMOS sensors).

The method can also work with alternative "switchable" workpiece areas. Example: to adjust the position of a screwdriver for the assembly of a car wheel with 6 screws, two alternative sets of three work piece areas are to be defined in a single (short-focal length, to comply with the below-mentioned tangent prohibition) camera. The ranges are chosen so that regardless of the rotational position of the wheel hub in at least one of the sets of image areas in each of the three areas always a screw hole is visible. The sets of ranges assigned to the sets basically do not need to be disjoint.

The relationship between the starting position and the further positions may be measured in arbitrary units describing this relationship, preferably in the number of steps of travel axes or in space coordinate increments.

The storage of relative positions (step f) can be done explicitly by storing the values, but also implicitly, for example, by specifying only corresponding step sizes and end values in a program loop.

• a) um die Eindeutigkeit der Zuordnung mit weniger Kameras zu erreichen und/oder
• b) zur Gewinnung von Redundanzen und dadurch Erhoehung der Genauigkeit und Stoersicherheit.

In automatic mode, a defined movement of the robot can occur between image recordings

• a) to achieve the uniqueness of the assignment with fewer cameras and / or
• b) to gain redundancy, thereby increasing accuracy and security.

Irrespective of this, several image recordings in the same robot position can be useful for increasing the accuracy and interference safety.

With the method according to the invention, both a control and a regulation (similar to visual servoing, with several image recordings per beam cluster) can be realized.

With the method according to the invention, preferably also a multi-level, hierarchical determination of the position can be realized: In a first step, a coarse position determination is realized on the basis of a data base with coarse scanning, whereupon the robot initially roughly co-corrects the situation. In the following steps, databases with increasingly smaller capture ranges and increasingly smaller scan sizes are used.

The significance condition can be achieved as described below:
At first we start from 3 points.

In [Fis] it is shown that the external camera parameters can be calculated from the two-dimensional mapping of 3 marked points with known distance, via a single camera with known internal parameters. Conversely, this corresponds to the determination of the position of the 3-D point triplet in known camera parameters. Transformation of the geometric equations [Fis] describing the task leads to a fourth-degree equation with up to 4 discrete, real positive solutions. However, these solutions can basically be close together.

If it is possible to guarantee, by choosing the geometry of the beam bundle, that there can be a maximum of one solution in the working range, the significance condition can be maintained with 3 excellent points and one camera. The solutions should therefore be far enough apart that only one solution can be found in the work area.

• – wenn Strahlenbuendel parallel zu den translatorischen Freiheitsgraden des Roboters sind,
• – aufgrund der Rotationsfreiheitsgrade.

Ambiguities arise

• If beamlines are parallel to the translatory degrees of freedom of the robot,
• - due to rotational degrees of freedom.

If, during rotations, the marked points tangentially intersect the rays of the ray bundles, 2 discrete solutions coincide. Corresponding illustrative considerations are made in [Wil] for the 1-camera case; they are topologically equivalent for the case in question here with more than one camera in general: if the rotating excellent points cut the ray bundles almost tangentially, the intersection points are close together. If you cut them at a larger angle (eg> 20 degrees), they are far apart.

If the work area is not too large, it can be ensured that there is only one solution in the work area.

The condition for excellent points is that during rotation, the excellent points do not intersect the rays of the ray tangents tangentially, but rather at a larger angle (eg> 20 degrees). This condition, together with the requirement that ray bundles are not parallel to the translatory degrees of freedom of the robot, is called tangent ban in the following. It is easy and many ways to comply with and, for example, satisfied when the beam columns represent a tripod and the fulcrum is in the middle of the base of the tripod and the axes of rotation do not go through the features.

The tangent ban on flex features is also met when the LIGHT beam bumps where they meet the flex features are not tangent to the circle around the rotation axes.

The ban on tangents is an example of the achievement of the significance condition. The ban on tangents is thus a sufficient but not necessary condition for the attainment of the (necessary) significance condition. See example 5 : here the significance condition is fulfilled even without adhering to the tangent ban.

On the other hand, adherence to the ban on tangents is advantageous, as can be seen from the comparison of 4 and 5 can take: in 5 In the case of body rotation, the patterns shown are distorted only slightly, while they are in 4 change significantly.

Since this invention is the CORRECTION of a movement, the work areas are usually sufficiently small to z. B. to ensure that a rotation axis does not pass through a feature.

For geometrical considerations, Fis and Wil presuppose the knowledge of the points in space; for the present invention, this knowledge is not essential. The considerations of [Fis] and [Wil] are helpful to recognize under which topological geometric conditions the significance condition can be achieved.

The solubility and uniqueness and stability considerations are abstract-geometric considerations and independent of the concrete distance measures. The additional information required to determine the position deviation comes from the data stored in set-up mode. Location information is only determined implicitly.

What remains is the transition from the measurement of the image of excellent points to the storage and comparison of general position description vectors, which in extreme cases can be given directly as feature images:
As is known, correlation methods can be used to calculate the displacement and rotation of an image relative to a reference image without excellent points, provided that the images have certain approximately reproducible structures that are expressed in at least two directions. Such methods allow the displacement calculation even in the presence of disturbances.

In the more general case, one does not have any positionally detectable features, ie either only directly via images or image description vectors with masses that do not indicate positions in images. Such mass are z. B. size change, rotation, distortion, brightness and color change, spatial frequency changes. These changes can also be brought about deliberately by suitable lighting measures, for example by structured light or Moire patterns on the workpiece.

• – Wertevergleich: Zur eindeutigen Zuordnung kann zum Beispiel ein auf dem Werkstueck festes Gittermuster verwendet werden, dessen Gitterweite und Orientierung sich bei der Roboterbewegung in der Abbildung veraendert; es genuegt die Berechnung der Gitterweite und der Orientierung, ohne Positionsdaten zu berechnen.
• – Bildvergleich: Ein begrenztes Schachbrettmuster zum Beispiel, dessen Rastergroesse und Orientierung sich bei der Roboterbewegung in der Abbildung veraendert, kann durch Bildvergleich einem von mehreren vorher gespeicherten Schachbrettmustern zugeordnet werden, ohne die jeweilige Rastergroesse und die Orientierung direkt zu berechnen (ein Schachbrett ist fuer direkten Bildvergleich guenstiger als beispielsweise ein Gitter mit feinen Linien).

This general case can be attributed to the case with excellent points by imagining that as the workpiece moves in space, image structures and associated value tuples tend to change in a monotone related manner, as in measuring the image coordinates of excellent points of the image Case would be.

• - Comparison of values: For unambiguous assignment it is possible, for example, to use a grid pattern fixed on the work piece, its grid width and Orientation changed in the robot movement in the picture; it is sufficient to calculate the grid width and the orientation without calculating position data.
• - Image comparison: A limited checkerboard pattern, for example, whose grid size and orientation changes in the robot motion in the image, can be assigned to one of several previously stored checkerboard patterns by image comparison without directly calculating the respective grid size and orientation (a chessboard is for direct Image comparison is cheaper than, for example, a grid with fine lines).

This makes it plausible that the significance condition can be met with the given arrangement, even with direct image comparison, without having any position values derived from the images.

There remains the transition from fix features to flex features. Here again, a similar consideration applies: Arrangements are also possible here with which image structures arise, as a result of which scalar value tuples tend to change in a monotone manner, as is the case when the image coordinates of excellent points are measured.

Example of a Flex Feature: The creation of a grid pattern or a checkerboard pattern by means of structured illumination leads, with suitable lighting geometry, to the same effect described above without such patterns having to be present on the work piece. See examples 4 and 5 ,

For flex features see also the example 3 ,

In the following, advantages of the invention will be explained.

Not necessarily conventional geometric features such as edges, corners, holes must be recorded. Basically, illustrations are sufficient from which any reproducible image structures result, which change in any way when the recording geometry changes and in their entirety make possible an unambiguous assignment to the situation.

There are many possibilities.

Features in the sense indicated above meet this requirement if the tangent ban described above is adhered to. Such features can be used, but their geometry need not be known.

• – Oberflaechenformen wie Sicken, Woelbungen,
• – kuenstlich erzeugte Flexmerkmale (s. o.),
• – inhomogene Strukturen wie reproduzierbare Uebergaenge von Oberflaechenstrukturen; beliebige metrisch undefinierte, aber in der optischen Abbildung reproduzierbare Merkmale.

Also suitable for selecting the areas to be detected are, for example

• - surface shapes such as beads, veins,
• - artificially created flex features (see above),
• - inhomogeneous structures such as reproducible transitions of surface structures; any metrically undefined but reproducible features in the optical image.

An analytic or experimental determination of the visual Jacobian matrix or a functional (eg, piecewise linearly approximating) relationship between robot movement and the movement of conventional (i.e., linkable with positional data) image features is not required. In the general case of working with a large image database, no data is derived from the images at all. A representation of any functional context is lost.

The method presented here does not rely on the use of pulsed, modulated or structured light. The method presented here can basically work with any light, z. B. the daylight. If a special lighting is used, it may well be additionally pulsed or modulated to increase the susceptibility to external light or limited or structured to a specific spectral range. However, this is not fundamentally necessary; the method is therefore neither under b1) nor under b2) and is - even with additional use of structured or pulsed or modulated light - not with the metrological problems of the methods b1) and b2) connected.

In the invention presented here, in contrast to stereo methods, the cameras generally consider different features, u. a. with the advantage that large workpieces are easier and more accurate to grasp. The further apart the features, the more accurate the movement correction is possible. On the other hand, on the other hand, in contrast to stereo systems, a single camera is sufficient.

In contrast to model-based systems, the workpiece geometry or the position of the features in a workpiece coordinate system need not be known, and even a workpiece coordinate system need not be defined. The task has been solved by implicitly determining the deviation of the workpiece position from the standard position, without having to know the geometrical relationships in the standard situation. In principle, the actual position of the work piece in the room does not need to be known in the standard position or in the position encountered in the automatic mode and also not to be calculated.

Calibration of the robot system to world coordinates, as is realized, for example, in [Ben] by means of movement of cameras with a view to a calibration plate, is omitted.

In contrast to conventional stereo systems, the camera geometry does not need to be known. The position of the plane coordinate systems in space need not be known. A calculation of the inner or outer camera coordinates is thus eliminated.

Without mathematical ambiguity problems systems with more than 6 robot degrees of freedom can be realized. Should z. B. an articulated robot produce a certain position of the gripper or tool in space, it may happen that certain positions can be achieved with several different robot positions. The associated mathematical problems do not exist here, since only these (and possibly adjacent) settings for the automatic mode come into question due to the presentation in different robot settings.

It is not necessary to attach artificial features such. B. adhesive dots.

The method allows a controlling or regulatory procedure.

The method can be used without it being necessary to be able to obtain positionally detectable features in the images.

In a concrete system, the transition from fix features to flex features is basically feasible solely by changing the lighting fixture without changing anything about the rest of the components such as robots, control, especially image exploitation techniques.

The method allows self-monitoring by defining different starting positions of the robot and, for these positions, each time after a corrective robot movement, with additional image recordings, the achievement of the correction target is checked. The Werkstuecklage need only be constant for this and does not need to conform to the standard situation.

The deviation occurring during the self-monitoring can be used for an automatic adaptation to slowly or slightly varying conditions.

Sources:

• [Ben] Luis Manuel Conde Bento Duarte Miguel Horta Mendonca: "Computer Vision and Kinematic Sensing in Robotics", Master Thesis, Department of Automatic Control, Lund Institute of Technology, June 2001
• [Dif] US Pat. No. 6,211,506 B1
• [Fan] US 4 639 878 A
• [Fis] FISCHLER, M.A .; BOLLES, R.C .: "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography", In: Communications of the A.C.M., June 1981, Vol. 6, pp. 381-395.
• [GMR] US 4,942,539 A
• [Hat] HUTCHINSON, S .; HAGER, G .; CORKE, P .: "A Tutorial on Visual Servo Control", In: IEEE Transactions on Robotics and Automation, Vol. 5, pp. 651-670, Oct. 1996th
• [Isr] EP 0 911 603 B1
• [Ja0] www.cs.rochester.edu/u/jag/PercAct/dvfb.html Stand 23.09.2001
• [Ja1] JAEGERSAND, M .; FUENTES, O .; NELSON, R .: "Experimental Evaluation of Uncalibrated Visual Servoing for Precision Manipulation", In: Proc. Int. Conf. on Robotics and Automation, April 1997, Vol. 4, pp 2874-2880
• [Kle] R. Klette, A. Koschan, K. Schluens: Computer Vision; spatial information from digital images. Vieweg Verlag Braunschweig, 1996.
• [Lee] LEES, D.E. B .; TREPAGNIER, P .: "Guiding Robots with Stereo Vision", In: Robotics Today, April 1984, pp. 79-81.
• [Mar] MARCHAND, E .: "VISP: a Software Environment for Eye-in-Hand Visual Servoing", In: IEEE Int. Conf. on Robotics and Automation, ICRA, Vol. 4, pp3224-3230, Detroit, Michigan, May 1999.
• [By] EP 1 076 221 A2
• [Pie] ARMSTRONG PIEPMEIER, J .; McMURRAY, G. V .; LIPKIN, H .: "Dynamic Quasi-Newton Method for Uncalibrated Visual Servoing", In: 1999 IEEE Int. Conf. on Robotics & Automation, ICRA, Vol. 2, pp. 1565-1600, Detroit, Michigan, May 1999.
• [Wil] WOLFE, W.J .; MATHIS, D .; WEBER SKLAIR, C .; MAGEE, M .: The Perspective View of Three Points, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 1, Jan. 1991, pp 66-73.

Claims (32)

Method for three-dimensional correction of relative movement with multiple degrees of freedom between workpieces on the one hand, and grippers or tools on the other hand, with an image pickup device of one or more cameras, wherein the image pickup device and / or the workpiece is reproducibly movable - with a Einrichtbetrieb with the following steps a) imaging one or more areas of the workpiece, at least one imaging beam path assigned to the areas, the location of the features in the respective area of the workpiece not being known, b) arranging a workpiece in a systematically selected or random standard position, c) arranging the image recording device in a starting position, d) recording at least one image via each imaging beam path, e) storing the images from step d) and / or data derived therefrom, and f) arranging the image recording device or the workpiece in relative positions to the starting position or standard position, wherein each degree of freedom is varied at least once compared to the starting position or standard position, wherein for each relative position: the relative position itself is stored explicitly or implicitly, steps d) and e) are repeated, with an automatic mode in which, due to the Setup mode stored images and / or davo n is derived from the relative movement, wherein one of the following two conditions is satisfied for taking the images: - two imaging beam paths S1 and S2 have mutually parallel beams and a third imaging beam path S3 has no beams parallel to a midplane between any two parallel beams S1 and S2 are located or - at least three imaging beam paths with each other have no parallel beams. Method according to claim 1, - with an automatic mode with the following steps g) picking up at least one image from each of the regions of the workpiece via at least one of the imaging beam paths associated therewith, in the initial position of the image recording device or in a position that is reproducibly deviating from the initial position, h) comparison of the images recorded in step g) and / or of the data derived therefrom with images or data of the corresponding imaging beam paths from a set-up operation, i) determining the current relative position of the workpiece to the standard position on the basis of the comparison in step h), j) Correcting the relative movement as a function of the relative position calculated in step i). Method according to claim 1 or 2, In which one of the following conditions is fulfilled in the image recording device: A single camera with central projection is used and an object field angle of at least 20 degrees, Two cameras are used, of which at least one works with central projection and an object field angle of at least 20 degrees, - Three or more cameras are used. Method according to one of claims 1 to 3, wherein of at least three workpiece areas, no two overlap each other. Method according to one of claims 1 to 4, wherein for at least two imaging beam paths separate cameras are used. Method according to one of claims 1 to 5, wherein the same camera is used for at least two imaging beam paths. Method according to one of claims 1 to 6, wherein at least one region has a feature which is fixedly formed on the workpiece. Method according to one of claims 1 to 7, wherein in at least one region, a pattern is present, which is generated by means of an illumination beam path on the workpiece. Method according to one of claims 1 to 8, wherein the workpiece or the gripper or the tool is held by a robot and the robot has at least one degree of freedom in a translatory coordinate direction and / or by one or more axes of rotation, the robot preferably a total of six Has degrees of freedom. Method according to claim 9, wherein for at least three workpiece areas either for at least one imaging beam path and / or for the illumination beam path according to claim 8: the beam path in question is oriented obliquely or at right angles to the translatory coordinates of the robot. Method according to claim 9, wherein for at least three workpiece areas either for at least one imaging beam path and / or for the illumination beam path according to claim 8: the beam path in question is obliquely or at right angles to the tangent to the circles of the points of the workpiece area about the axes of rotation. Method according to one of claims 1 to 11, with multi-stage operation, with execution of steps g) to j) in each stage, wherein, in the first stage, a coarse position correction is realized and in the further stages, a increasingly accurate position correction is realized, each with a corresponding starting position in each stage. Method according to one of claims 1 to 12, with several sets of image areas, wherein, depending on the current image content, a set of image regions is selected for the position correction. Method according to one of claims 1 to 13, with a self-monitoring operation with the steps: - Performing the steps g) to j) and correcting the position of the image pickup device, and - Performing again steps g) to j) for this situation and checking whether the newly calculated relative movement is sufficiently small. Method according to one of claims 1 to 13, with a self-monitoring operation with the steps: - Performing steps g) to i) and storing the correction values K-zero  - Repeated placement of the image pickup device in different positions l, with known positional deviations Delta-l from the starting position - for each of these positions l: Performing steps g) to i), calculating the corrections K-new-l, Compare the corrections K-new-l with the superposition of K-zero and delta-1. Method according to claim 14 or 15, wherein the self-monitoring delivers a test result about a deviation of the achieved correction, with which an automatic adaptation to slowly and / or slightly variable geometrical conditions is realized, preferably a change of the fastening and / or adjustment of an image recording device and / or a temperature drift of the kinematics of the robot according to claim 9. Device for three-dimensional correction of the relative movement with several degrees of freedom between workpieces on the one hand, and grippers or tools on the other hand, with an image pickup device from one or more cameras, wherein the image pickup device and / or the workpiece is reproducibly movable In which the following steps can be carried out in a set-up operation: a) imaging of one or more regions of the workpiece, via at least one imaging beam path associated with the regions, wherein the position of the features in the respective region of the workpiece is not known, b) arranging a workpiece in a systematically selected or random standard position, c) arranging the image recording device in a starting position, d) taking at least one image over each imaging beam path, e) storing the images from step d) and / or data derived therefrom, and f) arranging the image recording device or the workpiece in relative positions to the starting position or standard position, wherein each degree of freedom is varied at least once compared to the starting position or standard position, wherein for each relative position: The relative position itself is stored explicitly or implicitly, - the steps d) and e) are repeated, With an automatic mode in which the relative movement is corrected on the basis of the images stored in the set-up mode and / or data derived therefrom, wherein one of the following two conditions is met to capture the images: Two imaging beam paths S1 and S2 have mutually parallel beams and a third imaging beam path S3 has no beams parallel to a midplane between any two parallel beams of S1 and S2 or - Have at least three imaging beam paths with each other no parallel rays. Apparatus according to claim 17, wherein in an automatic mode the following steps are feasible: g) picking up at least one image from each of the regions of the workpiece via at least one of the imaging beam paths associated therewith, in the initial position of the image recording device or in a position that is reproducibly deviating from the initial position h) comparison of the images recorded in step g) and / or of the data derived therefrom with images or data of the corresponding imaging beam paths from a set-up operation, i) determining the current relative position of the workpiece to the standard position on the basis of the comparison in step h), j) Correcting the relative movement as a function of the relative position calculated in step i). Device according to claim 17 or 18, In which one of the following conditions is fulfilled in the image recording device: A single camera with central projection is used and an object field angle of at least 20 degrees, Two cameras are used, of which at least one works with central projection and an object field angle of at least 20 degrees, - Three or more cameras are used. Device according to one of claims 17 to 19, wherein of at least three workpiece areas, no two overlap each other. Device according to one of claims 17 to 20, wherein separate cameras are provided for at least two imaging beam paths. Device according to one of claims 17 to 21, wherein the same camera is provided for at least two imaging beam paths. Apparatus according to any one of claims 17 to 22, wherein at least one region has a feature which is fixedly formed on the workpiece. Device according to one of claims 17 to 23, wherein in at least one area a pattern is present, which is generated by means of an illumination beam path on the workpiece. Device according to one of claims 17 to 24, wherein the workpiece or the gripper or the tool is held by a robot and the robot has at least one degree of freedom in a translatory coordinate direction and / or by one or more axes of rotation, the robot preferably a total of six Has degrees of freedom. Device according to claim 25, wherein for at least three workpiece areas either for at least one imaging beam path and / or for the illumination beam path according to claim 24: the beam path in question is oriented obliquely or at right angles to the translatory coordinates of the robot. Device according to claim 25, wherein for at least three workpiece areas either for at least one imaging beam path and / or for the illumination beam path according to claim 24: the beam path in question is obliquely or at right angles to the tangent to the circles of the points of the workpiece area about the axes of rotation. Device according to one of claims 17 to 27, in which a multi-step way of working is feasible, with the execution of steps g) to i) in each stage, wherein, in the first stage, a coarse position correction is realized and in the other stages, an increasingly accurate position correction is realized, each with a respective starting position in each stage. Device according to one of claims 17 to 28, with several sets of image areas, with the possibility of selecting a set of image areas for position correction, depending on the current image content. Device according to one of claims 17 to 29, wherein in a self-monitoring operation the following steps are feasible: - performing steps g) to i) and correcting the position of the image pickup device, and - Performing again steps g) to i) for this situation and checking whether the newly calculated relative movement is sufficiently small. Device according to one of claims 17 to 29, wherein in a self-monitoring operation the following steps are feasible: Performing steps g) to i) and storing the correction values K zero, Repeatedly arranging the image recording device in different positions l, with known position deviations Delta-l from the starting position, - for each of these positions l: - Performing steps g) to i), with calculating the corrections K-New-l. Compare the corrections K-new-l with the superposition of K-zero and delta-1. Apparatus according to claim 30 or 31, wherein the self-monitoring provides a test result about a deviation of the achieved correction, with which an automatic adaptation to slowly and / or slightly varying geometric conditions is realized, preferably a change of attachment and / or adjustment of an image pickup device and / or a temperature drift of the kinematics of the robot according to claim 25.

Images