United States Patent [w]
Franke
US005937079A [ii] Patent Number: 5,937,079 [45] Date of Patent: Aug. 10,1999
[54] METHOD FOR STEREO IMAGE OBJECT DETECTION
[75] Inventor: Uwe Franke, Uhingen, Germany
[73] Assignee: Daimler-Benz AG, Stuttgart, Germany
[21] Appl. No.: 08/923,937
[22] Filed: Sep. 5, 1997
[30] Foreign Application Priority Data
Sep. 5, 1996 [DE] Germany 196 36 028
[51] Int. CI. G06K 9/00
[52] U.S. CI 382/103; 382/154
[58] Field of Search 382/103, 106,
382/107, 154; 348/42, 43, 47; 356/12
[56] References Cited
U.S. PATENT DOCUMENTS
5,684,890 11/1997 Miyashita et al 382/154
5,719,954 2/1998 Onda 382/154
5,818,959 10/1998 Webb et al 382/154
FOREIGN PATENT DOCUMENTS
0 626 655 A2 5/1994 European Pat. Off. .
43 08 776 Al 9/1993 Germany .
44 31 479 Al 3/1996 Germany .
Primary Examiner—-Jon Chang
Assistant Examiner—Jingge Wu
Attorney, Agent, or Firm—Evenson McKeown Edwards &
Lenahan, PLLC
[57] ABSTRACT
In a method for detecting and tracking objects by stereo image evaluation a part of structure class images is initially generated from a recorded stereo image pair. Differences in brightness of selected pixels in the environment are determined for each pixel as digital values, which are combined to form a digital value group, with identical groups defining their own structure classes. Structure classes which lack a brightness change along the epipolar line are discarded. Corresponding disparity values are then determined for the pixels in the other structure classes and are collected in a disparity histogram with a given frequency increment. The pixel group that belongs to a given grouping point area of the histogram is then interpreted as an object to be detected.
8 Claims, 2 Drawing Sheets
1
METHOD FOR STEREO IMAGE OBJECT
DETECTION
BACKGROUND AND SUMMARY OF THE
INVENTION 5
This application claims the priority of German Application No. 19636028.5, filed Sep. 5, 1996, the disclosure of which is expressly incorporated by reference herein.
The invention relates to a method for detecting and possibly tracking objects by recording and evaluating stereo 10 images. A method of this type is useful for example as an aid for automated guidance of highway vehicles, and also in the field of robotics, where it may be necessary to detect relevant objects and determine their position and size.
In autonomous vehicle guidance, for example in urban stop-and-go traffic, in addition to fixed objects located in the travel area of the vehicle, all moving objects in the immediate vicinity must be detected. On the other hand, no complete, dense depth chart of the scene is necessary for such applications. Thus in most cases a relatively flat base surface and clearly elevated objects can be used as a point of departure. In addition, no detailed information is required to be derived, such as the exact shape of a vehicle ahead. Such simplifying boundary conditions likewise apply to a plurality of problems in the fields of robotics and monitoring technology.
Methods of stereo image object detection can be divided into area-based methods and feature-based methods. Areabased methods are described, for example, in the conference 3Q papers by K. Sanejoschi, "3-D Image Recognition System by Means of Stereoscopy Combined with Ordinary Image Processing," Intelligent Vehicles '94, Oct. 24, 1994 to Oct. 26, 1994, Paris, pages 13 to 18 and L. Matthies et al, "Obstacle Detection for Unmanned Ground Vehicles: A 35 Progress Report," Intelligent Vehicles '95, Sep. 25-26, 1995, Detroit, pages 66 to 71. They require a higher computing capacity than feature-based methods. For an overview of current stereo image object detection methods, see O. Faugeras, "Three-Dimensional Computer Vision," MIT 4Q Press, 1993.
A method for detection of objects, especially vehicles, is known from German patent document DE 44 31 479 Al, in which two images are taken of a given area from different viewing angles. From a comparison of the two images, 45 especially their gray values, an object is detected for at least a partial area if the difference between the two images for the partial area in question is greater than a predetermined threshold.
In a system disclosed in German patent document DE 43 50 08 776 Al for monitoring a state external to the vehicle, a stereo image object detection method is used by which a given object is imaged within a fixed region outside a vehicle. The images recorded are subjected to an image processing device which calculates distance distribution 55 over the entire image. In order to discover a given object in the left and right images, the respective image is divided into small regions and color or brightness samples are compared within these regions for the two images in order to discover regions with corresponding object details and from this to 60 determine the distance distribution over the entire stereo image.
German patent document EP 0 626 655 A2 describes a device for detecting vehicles ahead and for determining their distance, which uses a stereo image object detection method. 65 For image evaluation, special techniques are employed, tailored to detection of vehicle contours, which use vehicle
2
contour models. The corresponding disparity and hence the vehicle distance are determined for a recognized vehicle.
One object of the present invention is to provide a method for stereo image object detection that is especially suitable for the above applications, with simplified boundary conditions, and is comparatively efficient and reliable.
In the method according to the invention, a structure class image pair (that is, a pair of images, with pixel data processed as discussed below) is initially generated from the recorded stereo image pair, with the differences in brightness of predetermined pixels in the environment being determined for each pixel, in digital form. The digital values are then combined in a predetermined sequence to form a digital value group, each of the various possible groups thus defining a separate structure class (that is, a different pattern of brightness variation).
Next, an advantageous and simply-designed correspondence analysis is conducted, in which all structure classes are omitted from consideration that show no structure gradients in the direction of the epipolar line; that is, along the line of corresponding pixel pairs of a common original pixel. (These are the structure classes whose pixels do not differ in brightness by a predeterminable amount from the brightness of the ambient pixels located in the direction of the epipolar line.) This results in considerable savings in image processing, since structures that extend in this direction, by virtue of the system, are not useful in any case for determining distance in feature-based stereo image evaluation.
For all the other structure classes, disparity values of corresponding pixels in the same structure class are then determined and collected in a disparity histogram to form a frequency value. Optionally, for each corresponding pixel pair, the corresponding disparity value can be included in the histogram with a weighted frequency increment. Then the histogram is studied for grouping point areas. At fixed grouping point areas of interest, the corresponding pixel group of a corresponding structure class image is then conversely represented and interpreted as an object that is located at a certain distance. It turns out that this object detection method operates very efficiently for many applications, and offers reliable results which are less prone to error.
In one embodiment of the method according to the invention, a ternary logic is used to digitize the brightness differences. This arrangement permits a structural classification that is very advantageous for the applications under consideration here, for two reasons: first it permits sufficiently differentiated structure classifications; and second it offers structure classifications that can be performed rapidly.
In another embodiment, the four pixels are selected as ambient pixels that directly adjoin the respective reference pixel on both sides, parallel to and then perpendicular to the epipolar line, respectively, or are separated therefrom by a predeterminable sampling width. A sampling width with the size of one or possibly several pixels allows the incorporation of a larger neighborhood area, which is advantageous in cases in which the brightness gradients typically extend over a range of several pixels.
In a further embodiment, the necessary calculation is further decreased by the fact that no disparities are favored. For each pixel of one structure class image, only the minimal disparity (in other words the distance from the closest pixel with the same structure class in another structure class image) is determined and taken into account to plot the disparity histogram. The basic assumption of favoring small disparities and hence larger object distances is especially
|
