US20110211071A1

US20110211071A1 - Method and system for evaluating brightness values in sensor images of image-evaluating adaptive cruise control systems, especially with respect to day/night distinction

Info

Publication number: US20110211071A1
Application number: US12/674,918
Authority: US
Inventors: Andreas Kuehnle; Cathy Boon
Original assignee: Valeo Schalter und Sensoren GmbH
Current assignee: Bendix Commercial Vehicle Systems LLC
Priority date: 2007-08-28
Filing date: 2008-08-28
Publication date: 2011-09-01
Also published as: US20130251208A1; WO2009030419A2; WO2009030419A3; US8823799B2; EP2181415A2

Abstract

The invention proposes a method and an arrangement for evaluating sensor images of an image-evaluating environment recognition system on a carrier, in which, in order to distinguish the light conditions in the area of the image-evaluating environment recognition system with regard to day or night, at least the gain and/or the exposure time of the at least one image sensor detecting the environment is/are monitored, a profile of the gain and/or the exposure time against time with relatively high gain or relatively long exposure times characterizing night-time light conditions, and a profile of the gain and/or the exposure time with relatively low gain and/or relatively short exposure times characterizing daytime light conditions. The environment recognition system according to the invention can also be used to search the detected environment for bright objects, the headlights of another carrier being used as additional information, for example.

Description

PRIOR ART

The invention relates to a method and an arrangement for evaluating brightness values in sensor images of an image-evaluating environment recognition system, in particular with regard to distinguishing between day and night, such as is preferably used in driving assistance systems in motor vehicles, according to the precharacterizing clause of Method Claim 1 et seq. and the arrangement according to Claim 8 et seq., as well as to a computer program product according to Claim 11.
Such driving assistance systems are used to assist a driver of a motor vehicle in carrying out particular operations in road traffic and are already used for a wide variety of tasks. For example, DE 10 2004 017 890 A1 discloses that a so-called LIDAR radar sensor is used to detect a predefined area in the direction of travel ahead of the vehicle in terms of particular objects, and particular safety functions can be triggered in good time by means of corresponding evaluation of the sensor signals. Examples of such vision-based or image-based driving assistance systems which attempt to understand or interpret a roadway situation or a scene are sufficiently well-known lane-keeping assistants, lane departure warning assistants, collision warning assistants or the like.
These systems which are known per se can be used, for example, as part of adaptive cruise control and/or inter-vehicle distance control of a vehicle, wherein such a control process can then be carried out without intervention by the driver, a previously set driving speed and/or a previously set distance from a vehicle travelling ahead or from items and/or objects located in the direction of travel. This is generally done by correspondingly taking into account the environment of the vehicle and, if appropriate, further parameters, for example the weather conditions and visibility conditions. Such a control system is also often referred to as an adaptive cruise control system (ACC system). The ACC system must be flexible enough, in particular in respect of the increasing traffic density of the present time, in order to react suitably to all driving situations. This in turn requires a corresponding object-detection sensor system in order to supply the measurement data which are necessary for the control process in each driving situation.
For this purpose, camera images, video images or sensor images are also used in image-evaluating or else so-called vision-based driving assistance systems for environment detection and recognition, wherein, in the known systems, objects, obstacles, boundaries of roadways and lanes and distances therefrom are determined from the camera images. For the actual capturing of images, image sensors which are referred to as imagers are used in so-called vision-based environment detection systems, which image sensors then supply a camera image (also referred to as a sensor image) of the detected environment, the pixels or image areas of which can be evaluated in terms of intensity, contrast, colour or other parameters using corresponding data-processing means.
In this case, the sensor images of an identical scene or an identical roadway situation may have a different appearance depending on the external lighting. Environment recognition systems in video-based driving assistance systems which attempt to interpret the same scene or roadway situation with different lighting may therefore have problems dealing with these different conditions during evaluation. The different appearance on account of changed lighting often requires readjustment or adjustment of the control algorithms, parameters or presettings needed or used to understand or interpret the scene or roadway situation.
A particularly frequent example of a scene which has a different appearance depending on the lighting is a traffic situation with vehicles which travel with the vehicle lighting switched on at night and with the vehicle lighting switched off during the day. Recognition of other vehicles by a vehicle with an image-based driving assistance system may thus be made more difficult or impaired depending on the light conditions. Recognition of lane and/or roadway markings by an abovementioned driving assistance system or by the environment recognition system which interprets the scene or roadway situation may also be made more difficult or impaired during a change from day to night.

DISCLOSURE OF THE INVENTION

The invention is based on a method for evaluating sensor images of an image-evaluating environment recognition system on a carrier, for example a vehicle in road traffic, possibly taking into account the vehicle's own movement, in which, in order to distinguish the light conditions in the area of the image-evaluating environment recognition system with regard to day or night, at least the gain and/or the exposure time of the at least one image sensor detecting the environment is/are advantageously monitored according to the invention. A profile of the gain and/or the exposure time against time with relatively high gain or relatively long exposure times will characterize night-time light conditions, and a profile of the gain and/or the exposure time with relatively low gain and/or relatively short exposure times will characterize daytime light conditions. However, further additional information may also be used in this case to make a statement on a daytime or night-time state.
In order to determine a daytime or night-time state, it is thus possible to evaluate at least the profile of the gain and/or the exposure time of the image sensor in a predefined period of time by determining, for example, whether a significant part or a sufficiently large part of this profile corresponds to a daytime or night-time state.
The inventive method for determining a daytime or night-time state of a scene or roadway situation detected in a video-based or image-based manner using at least one image sensor therefore uses a different light intensity of the scene or roadway situation by day or by night to the effect that a statement on a daytime or night-time state can therefore be made in a simple manner.
The invention makes it possible to carry out an instantaneous daytime or night-time recognition process in regular cycles using the instantaneous gain and/or the instantaneous exposure time of the image sensor, a number of previous stored daytime or night-time recognition processes which may represent a profile that can be evaluated also being able to be evaluated in addition to the current or last daytime or night-time recognition process, said number corresponding to a predefinable period of time and the period of time on which the profile is based also being able to be variable. In this case, a sufficient number of previous daytime or night-time recognition processes which represent the profile or the development of the situation should be present in order to make a statement on a daytime or night-time state, which number can be gathered from the profile.
If there is no difference from a previously or recently made statement on a daytime or night-time state, for example because there is not a sufficient number of daytime or night-time recognition processes for a new decision, the previously or recently made statement preferably remains unchanged.
The profile or the temporal development may be determined by carrying out an instantaneous daytime or night-time recognition process in regular cycles using the instantaneous gain and/or the instantaneous exposure time of the image sensor, a number of previous daytime or night-time recognition processes being stored in addition to the current or last daytime or night-time recognition process, said number corresponding to a predefinable period of time.
If the period of time over which the profile is tracked or the number of stored previous daytime or night-time recognition processes is variable, for example in order to avoid having to imperatively resort to a long chain of previous night-time recognition processes after turning off a vehicle at night and when reusing the vehicle the next morning, an even better statement on a daytime or night-time state can be made.
The method according to the invention can also be extended by the fact that, in the event of an inconclusive assignment to a daytime or night-time state, a third setting which does not contribute to a conclusive statement on a daytime or night-time state is provided for the image sensor in the environment recognition system. An average value of the state recognition profile can be used to determine the third setting in order to determine a state lying between a daytime state and a night-time state, the average value being formed, in particular, in a fashion proportional to the profile or with the aid of a lookup table. The average value is preferably formed in a fashion proportional to the profile, for example proportional to the profile of the daytime or night-time recognition processes carried out.
This particularly advantageous refinement of the invention thus provides for an intermediate zone to correspond to a “neither daytime nor night-time state” of the image sensor. Currently, such a “neither daytime nor night-time state” initially results neither in a daytime recognition process nor in a night-time recognition process, that is to say the image sensor has in this case a gain and/or an exposure time which, if permanently maintained, would not result in any conclusive statement on a daytime or night-time state. As a result of the fact that a “neither daytime nor night-time state” is thus provided as a third setting, it is possible to avoid frequently changing over between determination of a daytime state and determination of a night-time state.
For example, for a profile with 40% daytime recognition, 50% night-time recognition and 10% “neither daytime nor night-time states”, the parameters or presettings may be set or readjusted with a 50/90 share of the setting range between day and night. A proportionality which is not necessarily linear in this case may thus be formed between the brightness state of the environment and the parameter settings.
It is particularly advantageous if the environment detected using the environment recognition system according to the invention is searched for bright objects which indicate the headlights of another carrier and whose occurrence is then used as additional information. Additional environment-specific information may also be used to distinguish the daytime or night-time state, which environment-specific information includes, in particular, further indication objects which are recognized in a detected environment and whose different appearance by day or night is known. For example, reflectors fitted to the surface of the roadway shine brightly at night as a result of their highly reflective properties; they are almost invisible or dark during the day.
For this purpose, bright objects in the form of headlights, in particular on the front of a vehicle, may be determined in particular, the occurrence of which is then used as additional information. Such objects have, for example, a generally round or elliptical shape and occur in pairs. They are thus generally easy to identify during image evaluation.
In this case, it is likewise advantageous if the frequency with which the bright objects and/or further environment-specific information are/is found in the detected environment is determined, a connection being established between a night-time state and a predefined frequency of the occurrence of such objects in an environment. One example is vehicle lights which are seen more and more often at night, with the result that the quotient of the number of lights found and the number of vehicles found increases.
The area-specific knowledge thus preferably also includes particular indication objects which can be identified in an environment, for example objects which are used to delimit roadways and lanes and the appearance of which is known, or other patterns in the form of roadway or lane markings in the detected situations. These also differ by day and by night, for example some appear to be larger by night than by day.
The frequency of occurrence may in turn be included as additional information in a statement or decision on a daytime or night-time state. For example, a connection may be established between an increasing frequency of the occurrence or observation of indication objects with a known appearance at night in a scene or situation and a decision or statement tending towards a night-time state.
In the case of an advantageous arrangement for carrying out the method described above, the image-evaluating environment recognition system includes, as image sensor, an electronic camera which is fitted to a vehicle as a carrier and continuously records the front area (in the direction of travel) of the vehicle on a roadway in such a way that a sensor image constructed from pixels whose brightness values and, if appropriate, colour values image the environment is respectively present. There is also an evaluation unit which can be used to distinguish the light conditions in the area of the image-evaluating environment recognition system with regard to day or night on the basis of the gain and/or the exposure time of the at least one camera which records the environment.
In the case of such an arrangement, a signal from the determination of the daytime or night-time state, that is to say in particular parameters or presettings which can be used to set a video-based or image-based driving assistance system with regard to the daytime or night-time state, can then be applied to the evaluation unit in a simple manner. In this case, it is conceivable, for example, to set the parameters or presettings on a sliding scale in such a manner that the best performance of an environment recognition system which recognizes a situation in the environment is achieved in conjunction with a video-based or image-based driving assistance system.
In summary, the invention thus provides a method which makes it possible to classify a lighting situation as a daytime or night-time situation. This classification thus makes it possible to recognize a roadway situation without errors independently of the lighting state thereof.
The invention also proposes a computer program product which, stored on a computer-usable medium, comprises computer-readable program means which, in the event of the computer program product being executed on a microprocessor with associated storage means or on a computer, cause said microprocessor or computer to carry out the method according to the invention or to operate the arrangement.

BRIEF DESCRIPTION OF THE DRAWING

One exemplary embodiment of the invention is illustrated in the figures of the drawing and is explained below. In the drawing:

FIG. 1 shows a diagrammatic illustration of a vehicle having a camera as part of an environment recognition system for evaluating daytime and night-time states, additionally also using the headlights of another vehicle, and

FIG. 2 shows a flowchart of the method features according to the invention in an evaluation unit of the environment recognition system.

DESCRIPTION OF THE EXEMPLARY EMBODIMENT

FIG. 1 diagrammatically reveals a situation of a vehicle 1 as a carrier of an environment recognition system which can move in this case on a roadway 2 in the direction of an arrow 3. The environment recognition system of the vehicle 1 has, as an image sensor, a camera 4, in particular a digital video camera, which records an area between dashed lines 5 and 6.
The roadway 2 is separated by a marking 7, and another vehicle 8 which is fitted with front headlights 9 approaches on the other side of the roadway. At an input 11, the digital data of the pixel-comprising sensor image from the camera 4 and additionally, for example at an input 12, also the current speed data of the vehicle 1 are evaluated in an evaluation device 10.
In order to distinguish the light conditions in the area of the image-evaluating environment recognition system on the vehicle 1 with regard to day or night, the gain and/or the exposure time of the camera 4 is/are monitored in the evaluation device 10 according to FIG. 1. In this case, a profile of the gain and/or the exposure time against time with relatively high gain or relatively long exposure times characterizes the night-time light conditions and a profile of the gain and/or the exposure time with relatively low gain and/or relatively short exposure times characterizes the daytime light conditions.
In this case, however, it is also possible to use further additional information to make a statement on a daytime or night-time state. The environment recorded with the camera 4 is searched for bright objects, the headlights 9 of the other vehicle 8 being used as additional information in the exemplary embodiment shown here. Furthermore, it is also possible to use additional environment-specific information (not explained in any more detail here) to distinguish the daytime or night-time state, which environment-specific information includes, in particular, further indication objects which are recognized in a detected environment and whose different appearance by day or night is known.
In order to detect a different light intensity which allows a conclusion to be drawn on a daytime or night-time state, it is generally possible to proceed using the flowchart according to FIG. 2, essentially the following method steps which are then explained in detail using the flowchart according to FIG. 2 being carried out in the evaluation unit 10 in this case:

- At least the gain and/or the exposure time of the camera 4 is/are monitored.
- A profile or a development with consistently high gain or consistently long exposure times is used as an indication that it is currently night-time.
- A profile or a development with consistently low gain or consistently short exposure times is used as an indication that it is currently daytime.
- The profile or the development of the gain and/or the exposure times of the image sensor or similar settings is/are preferably evaluated and it is determined whether a significant part or a sufficiently large part of this profile or this development corresponds to a daytime or night-time state.
- A profile or a development of the daytime and night-time decisions can subsequently then be stored, which profile or development can then be used to make a subsequent statement on a daytime or night-time state.
- The profile (which can also be referred to as the history) or the temporal development can be determined, for example, by carrying out an instantaneous daytime or night-time recognition process in regular cycles using the instantaneous gain and/or the instantaneous exposure time of the image sensor. A number of previous daytime or night-time recognition processes which represent the profile or the development of the daytime or night-time recognition processes can be stored in addition to the current or last daytime or night-time recognition process, said number corresponding to a predefinable period of time.
- The period of time may be variable, for example in order to avoid having to imperatively resort to a long chain of previous night-time recognition processes after turning off a vehicle at night and when reusing the vehicle the next morning, in order to make a statement on a daytime or night-time state.
- In order to avoid frequently changing over between determination of a daytime state and determination of a night-time state, provision is preferably made of a “neither daytime nor night-time state” which corresponds to an intermediate zone and preferably provides neither daytime settings nor night-time settings for the image sensor. Currently, such a state initially results neither in a daytime recognition process nor in a night-time recognition process, that is to say the image sensor has a state, that is to say a gain and/or an exposure time, which state, if permanently maintained, would not result in any conclusive statement on a daytime or night-time state.
- This “neither daytime nor night-time” state preferably also does not contribute to a conclusive statement on a daytime or night-time state.
- A sufficient number, which can be gathered from the profile or the development for example, or a sufficient proportion, which corresponds to a percentage for example, of previous daytime or night-time recognition processes is preferably needed to make a statement on a daytime or night-time state.
- If there is no difference from a previously or recently made statement on a daytime or night-time state, for example because there is not a sufficient number of daytime or night-time recognition processes for a new decision, the previous state is preferably retained or the previously or recently made statement preferably remains unchanged.
- According to the invention, additional information may be used to make a statement or decision on a daytime or night-time state. For example, it is conceivable to search detected situations or scenes for bright objects in the form of headlights, as illustrated in FIG. 1. Objects in the form of headlights on the front of another vehicle 8 have, for example, a generally round or elliptical shape and occur in pairs and at approximately the same image height.
- The frequency with which such objects are found in the detected situations or scenes which are searched can also be monitored. A connection is preferably established between an increasing frequency of the occurrence or observation of such objects in a scene or a situation and a decision or statement tending towards a night-time state.
- Area-specific knowledge relating to how a scene or a situation has a different appearance by day and by night can also be used to make a statement or decision on a daytime or night-time state.
- An example of such area-specific knowledge is the fact that particular objects, for example reflectors which are used, for example, to delimit roadways and lanes, appear larger by night than by day.
- In this case, the frequency with which objects having such properties or such an appearance occur, for example, in a pattern in the form of roadway or lane markings can then be monitored. In this case too, a connection is preferably established between an increasing frequency of the occurrence or determination of such objects in a scene or in a situation and a decision or statement tending towards a night-time state.
- The decision itself or the statement made need not necessarily be binary, that is to say does not necessarily need to have only two states, for example it is day or it is night. For example, an average value of the daytime or night-time recognition processes can be used, for example in a proportional manner, to set or readjust the rules or presettings of a video-based or image-based driving assistance system, which understands or interprets a situation or a scene, or the image recognition system thereof. For example, for a profile with 40% daytime recognition, 50% night-time recognition and 10% “neither daytime nor night-time states”, the rules or presettings may be set or readjusted with a 50/90 share of the setting range between day and night.
- Such setting or readjustment can also be carried out in a non-linear manner, for example with the aid of a lookup table.

A flowchart of the method according to the invention is now explained by way of example using FIG. 2.
The method starts in a first method step 21. A daytime state is set as the starting or initial value in a second method step 22. A so-called frame is recorded in a third method step 23, which frame contains at least the instantaneous gain and the instantaneous exposure time of at least the camera 4 according to FIG. 1, which records an environment as a roadway situation or scene, at least at the time at which the environment is detected in a video-based or image-based manner.
The gain and the exposure time of the at least one camera 4 are read from the frame in a fourth method step 24. In a fifth method step 25 (inserted under A after the method step 24), an enquiry may take place in order to determine whether a sufficiently fast movement has taken place since the last frame was recorded, for example in order to determine whether a sufficient distance on the roadway 2 has been travelled since then. If this is the case, the method proceeds with a sixth method step 26. If this is not the case, a new frame is recorded again in the third method step 23. However, the method according to the invention does not necessarily presuppose any movement of the carrier and thus of the camera 4; however, the vehicle's own movement is taken into account in the exemplary embodiment described here.
The sixth method step 26 clarifies whether the instantaneous exposure time and instantaneous gain stored in the recorded frame correspond to an instantaneous daytime state or a night-time state or a “neither daytime nor night-time state”. The result is an instantaneous daytime or night-time recognition process or an instantaneous “neither daytime nor night-time state”.
In a seventh method step 27, the knowledge obtained in the sixth method step 26 is added to a profile. In an eighth method step 28, an enquiry takes place in order to determine whether the profile has a sufficient length. If this is the case, the method continues with a ninth method step 29. If this is not the case, the method continues with the third method step 23.
In the ninth method step 29, an enquiry takes place in order to determine whether items of knowledge added to the profile in a sufficient number are night-time decisions. If this is the case, the method continues with a tenth method step 30. If this is not the case, the method continues with an eleventh method step 31.
In the tenth method step 30, the statement that a night-time state prevails is made. In the eleventh method step 31, an enquiry takes place in order to determine whether items of knowledge added to the profile in a sufficient number are daytime decisions. If this is the case, the method continues with a twelfth method step 32. If this is not the case, the method preferably continues with the third method step 23. In the twelfth method step 32, the statement that a daytime state prevails is made.
In a thirteenth method step 33 which follows the tenth and twelfth method steps 30, 32, the profile is erased and the method continues in the third method step 23 again.
It is important to emphasize that it is not the aim or objective of the invention to control lighting. Rather, the aim of the invention is to describe or classify a lighting situation of an environment, a scene or a roadway situation by monitoring the camera and incident or process statistics. A result obtained thereby in the form of the statement made is preferably used to set the rules, parameters or presettings of a video-based or image-based driving assistance system, which understands or interprets the environment, or the environment recognition system thereof, for example on a sliding scale, in such a manner that the best performance is achieved.
The invention is industrially applicable, in particular, in the field of the production and operation of video-based or image-based driving assistance systems or video-based or image-based video systems which can be used in road traffic. In this case, it is particularly advantageous and also provided for the driver assistance system to have a lane-keeping assistance system.

Claims

1. A method for evaluating sensor images of an image-evaluating environment recognition system on a carrier having a driver assistance system, wherein the driver assistance system has a lane-keeping assistance system, the method comprising:

monitoring at least one of a gain and an exposure time of the sensor images detecting the environment to distinguish light conditions in an area of the image-evaluating environment recognition system with regard to day or night,

wherein a state recognition profile of the gain and/or the exposure time against time with relatively high gain or relatively long exposure times characterizes night-time light conditions, and the state recognition profile of the gain and/or the exposure time with relatively low gain and/or relatively short exposure times characterizes daytime light conditions.

2. The method according to claim 1, wherein the profile is determined by carrying out an instantaneous daytime or night-time recognition process in regular cycles using one of the instantaneous gain and the instantaneous exposure time of the image sensor, and wherein the method further comprises storing a number of previous daytime or night-time recognition processes which represent the profile in addition to the current daytime or night-time recognition process, said number corresponding to a predefined period of time.

3. The method according to claim 1, wherein, in the event of an inconclusive assignment to a daytime or a night-time state, a third setting is provided for the image sensor, wherein the third setting does not contribute to a conclusive statement of the daytime or the night-time state.

4. The method according to claim 3, an average value of the state recognition profile is used to determine the third setting in order to determine a state lying between a daytime state and a night-time state, wherein the average value is formed proportional to the state recognition profile or with the aid of a lookup table.

5. The method according to claim 1, wherein the environment detected using the environment recognition system is searched for bright objects which indicate switched-on headlights of another carrier, wherein the occurrence of bright objects is then used as additional information.

6. The method according to claim 5, wherein additional information used for distinguishing the daytime or night-time state further comprises indication objects which are recognized in a detected environment, wherein indication objects appear different by day and night.

7. The method according to claim 6, further comprising:

determining wherein a frequency with which the bright objects and indication objects are found in the detected environment; and

establishing a connection between a night-time state and a predefined frequency of the occurrence of the bright and indication objects in an environment.

8. An arrangement for evaluating sensor images of an image-evaluating environment recognition system on a carrier having a driver assistance system, wherein the image-evaluating environment recognition system comprises: an image sensor,

an electronic camera which is fitted to a vehicle, wherein the vehicle is a carrier provided with a lane-keeping assistance system, wherein the camera is configured to continuously record the front area in a direction of travel of the vehicle on a roadway

wherein the camera and the image sensor capture a sensor image constructed from pixels comprising brightness values and colour values

an evaluation unit configured to distinguish light conditions in an area of the image-evaluating environment recognition system with regard to day or night using a gain and/or an exposure time of the camera which records the environment.

9. The arrangement according to claim 8, further comprising:

signaling the evaluation unit, based on the determination of a daytime or a night-time state, to apply parameters and presettings used to set a video-based or image-based driving assistance system the daytime or night-time state.

10. The arrangement according to claim 9, wherein a speed of the vehicle, is applied to the evaluation unit.

11. A computer program product stored on a computer-usable medium, comprising computer-readable program instructions which, when the computer program product is executed on a microprocessor with associated storage means, cause said microprocessor or computer to carry out a method according to claim 1.