DE102015013943A1 - Testing a navigation system of a motor vehicle - Google Patents

Abstract

The invention relates to a method for testing a navigation system (1), sensor data (5) being fed into the navigation system (1) by a test device (2) in a test run (9) and individual pictures (6) of a picture sequence (4) of the navigation system (FIG. 1), wherein each individual image (6) represents a combination (29) of key graphical objects (11) and at least one of the key graphical objects (11) is selected and in the individual images (6) each selected key object (11) is extracted and from a respective feature value (12) is determined for each extracted key object (11) for at least one predetermined object feature (30) and for each object feature (30) the determined feature values (12) of all captured individual images (6) are combined to form a time signal (14); it is detected whether at least one time signal (14) satisfies a predetermined error criterion (7) and, if the error criterion (7) is met, a flag signal l (8) is generated.

Description

The invention relates to a method for testing a navigation system. The method is carried out by a test device, which is also part of the invention.

The complexity of navigation systems in motor vehicles is constantly increasing. This is due not only to the number of available functions and the required graphical processing of the navigation, but also to the variety of variants of the installed navigation systems and their interaction with other devices and infotainment software components (infotainment system information entertainment system). As the complexity increases, so does the likelihood that the navigation software will contain errors. These errors must be found before the software reaches the customer as series production. The experience of recent years shows that rare, complex errors can only be made with hundreds or even thousands of hours of automated testing. In order to achieve the necessary test coverage and test time for each of the relevant navigation variants in an economically sensible manner, automated testing with little evaluation effort is indispensable for the tester (human).

From the DE 10 2012 009 876 A1 For example, a test method for navigation systems is known in which an image sequence that generates a navigation system for outputting navigation instructions is examined to see which areas in the image sequence have been animated and which have remained unchanged as static or stationary areas. Subsequently, a motion model simulating the animation is configured based solely on the moving image areas, resulting in a more accurate motion description compared to using the entire image sequence.

In the DE 10 2008 012 697 A1 A method for operating a navigation system is described to check whether current map information and current driving information match. From this, a degree of the quality of the map information can be determined.

In the DE 10 2013 015 145 A1 A method for computer-aided testing and rating of the quality of digital road map data is described. The quality is assessed on the basis of quality criteria, which may include several subcriteria and which may be chosen depending on the requirements of the applications that are to use the roadmap.

The invention has for its object to automatically test a navigation system.

The object is solved by the subject matters of the independent claims. Advantageous developments of the invention are given by the features of the dependent claims.

The invention provides a method for testing a navigation system. The method is performed by a test device. In a test run of the navigation system sensor data are fed into the navigation system, which describe a drive of a motor vehicle. The navigation system is thus operated exactly as if it had been installed in a motor vehicle and would receive sensor data from the motor vehicle. Sensor data may be, for example, GPS data (GPS - Global Positioning System) and / or yaw rate data of a yaw rate sensor and / or a speed signal. The sensor data can be generated by means of a simulation or during a test drive of the navigation system. As an alternative to feeding the sensor data, these can be simulated within the navigation system. For this purpose, a so-called demo mode with test scenarios can be used. "Demo mode" means that the navigation software of the navigation system generates the sensor data itself, based solely on an externally set starting point and destination point. This saves a supply of sensor data from the outside. The vehicle drives "virtually" at a constant speed.

The navigation system then generates in response to the sensor data and a predetermined navigation target in a known manner a video sequence or animation or image sequence that can be displayed on a screen of the navigation system. The image sequence can represent, for example, a symbol for the own vehicle as well as the road ahead. The method now provides for individual images (frames) of this image sequence to be detected by the test device. The individual images can be detected, for example, by reading a video signal or a so-called frame grabbing or by filming the screen. The important thing is that each frame represents a combination of key graphical objects. Such a key object may be, for example, the described symbol for the own vehicle or a road or a warning for a construction site. At least one of these key graphical objects is selected based on configuration data. In other words, in the test device based on the configuration data, for example, set by a user, which key object or which key objects to be observed by the test device.

In the frames, each selected key object is extracted. In other words, a segmentation of the selected key object is made in each frame. It is thus determined which pixels (picture elements) of the single picture belong to the key object. From each extracted key object, a respective feature value is determined for at least one predetermined object feature. An object feature or an object property is therefore quantified in each individual image. For example, an object feature may be the area or size of the key object or a geometric orientation. This is expressed or described for each frame as a respective feature value. The determined feature values of all acquired individual images are then combined to form a time signal or characteristic curve for each object feature. Thus, for example, a time signal is generated which describes the area or size of the key object over time. It is then detected whether the time signal or at least one of a plurality of time signals satisfies a predetermined error criterion. If the error criterion is met, an alert signal is generated.

The advantage of the invention is that the response of the navigation system to the sensor data and the predefined navigation destination is checked automatically by the test device and a user of the test device, i. H. a human needs to check the image sequence only when the alert signal is generated, that is, when an unusual event in the image sequence has been detected by the test device. Which events in the image sequence are classified as unusual can be adjusted by the error criterion. The test apparatus thus makes it possible to operate a navigation system for many hours or even days, without the image sequence generated by the navigation system actually having to be observed by a human observer. On the basis of the generated notification signal can then be specifically checked that portion of the image sequence actually by the user, to which the error criterion has been met. It is also possible to generate a further in-order signal if the error criterion is not met.

The invention also includes optional developments, by the characteristics of which additional benefits.

One aspect concerns the question of which object features are suitable for reliably detecting an error in the image sequence. A development for this purpose provides that an object feature is a scalar value, ie a "one-number feature". For example, a pixel count of the key graphic object can be determined. This corresponds to the display area of the key graphic object. A scalar value has the advantage that the said time signal is determined as a time series of individual values. It can then be processed particularly efficiently by the test device. An object feature may, according to a further development, describe a dynamic or temporal property of the key object, for example a movement speed or a motion vector. This has the advantage that a relationship between successive individual images is taken into account. An object feature can be a feature tuple of several individual features according to a development. As a result, the time signal expresses a multidimensional trajectory, so that the temporal evolution of several individual features is described by a single time signal.

In a further development, an object feature is selected that depends on a navigation context. In this way, a respective suitable object feature can then be determined in advance for different sensor data and / or navigation targets, so that a suitable object feature is selected specifically for different navigation situations. Additionally or alternatively, a correlation of two predetermined individual features may also be described as object feature. In other words, a single feature is not observed directly from the individual image, but a relationship between two such individual features. These individual features may originate from the key object or even describe a reference of a single feature of the key object to a single feature of another object. For example, by way of example, a size ratio of two key objects can be described as an object feature.

A further development provides that the extraction of the key object takes place on the basis of a color and / or a graphic pattern. Extraction based on color is particularly efficient with low computational resources. A graphic pattern can be detected with a low-effort pattern recognition.

The error criterion described may, for example, state that the course of the time signal must be within a predetermined value interval or tolerance range. Once the time signal is outside the interval, the error criterion is met. A development for this purpose provides that no absolute value interval is used, but a comparison of the time signal with a reference signal is performed. In this development, the test run is repeated several times. One of the test passes, a reference sequence, then generates a time signal or multiple time signals, depending on how many key objects and how many object features per key object to be watched. Each time signal of this test run represents a reference signal in each case. For another test run, a test sequence, a corresponding time signal is generated in each case, which is then used in each case as a test signal. The error criterion then includes that a difference between at least one test signal and the corresponding reference signal is outside a predetermined tolerance range.

The test runs can be performed with the same or different sensor data. Even if the same sensor data is used per test run, different reactions of the navigation system occur, depending on the utilization of other vehicle systems that are realized with the same hardware. For example, in a multithreaded processor system, the processing of the sensor data in different test runs in the navigation system can be different. Accordingly, there are also different image sequences. For example, there may be an interruption of the presentation of the image sequence at an unfavorable utilization of the processor system.

To be able to compare the test signal with the reference signal, a further development provides that the reference signal and / or the test signal is interpolated to a sampling rate predetermined for scanning the image sequence. This results in a synchronization of the test signal with the reference signal. The synchronization is therefore not necessary from the beginning. Thus, individual images within the respective image sequence can be acquired at different times for the different test runs. Thus, a hardware-in-the-loop test can be made possible in which the navigation system runs independently or asynchronously to the test device. If the described scalar value is used as object feature, the interpolation is particularly robust. There is no need for multidimensional, for example graphic interpolation of an image.

In order to make the comparison between reference signals and test signals even more meaningful, a development provides that the test signal is registered with the reference signal by a window-based comparison. By registration, it is meant that a time offset or time offset between the test signal and the reference signal is set based on the window-based comparison. The window-based comparison may, for example, be a correlation or a sum of absolute difference values (SAD). The minimum correlation value or SAD value then indicates that the test signal and the reference signal are most similar, that is, the time offset is optimal in that a detected or detected difference between the reference signal and the test signal is caused solely by a difference in the image sequences and not due to a faulty time offset.

A further development even makes it possible to automatically generate the error criterion for the comparison of test signal and reference signal. For further development, a further reference signal is generated, for example, from another section of the reference sequence or by running through the test run again. From the further reference signal (or the plurality of further reference signals for a plurality of object features), the tolerance range for the difference of the test signal to the corresponding reference signal is determined by means of a method of machine learning (ML). As a method of machine learning, for example, the so-called Novelty Detetection method (novelty detection) can be used. By means of the machine learning it is recognized on the basis of the further reference signal which curves or values are typical for the reference signal, ie. H. the expected behavior. This can be used to detect statistical outliers that indicate an error.

As a particularly reliable automatic tolerance setting, the following has been found. It has the additional advantage that it can be checked by a user by means of a graphical display whether the tolerance range has been suitably set. In the development, the feature values of at least second reference signals are used or interpreted as coordinates of a data point. In other words, the feature values of the reference signals to a given frame (or interpolation at a given time) are interpreted as coordinates. Taking, for example, two reference signals, the result is a data point per point in time in a two-dimensional coordinate system. If this is carried out for all times or sampling times of the reference signals, the result is a point cloud. From this point cloud, d. H. from the data points of the reference signals z. B. determined by means of radial basis functions (RBF - Radial Basis Function) of the tolerance range as a so-called decision boundary (Decision Boundary). The decision boundary is the line that includes the data points or at least most of the data points. The course or the shape of the decision boundary can be adapted particularly flexibly by means of the described radial basis functions, whereby false alarms can be reduced in error detection.

The invention also includes the described test device for a navigation system of a Motor vehicle. The test device is set up to carry out an embodiment of the method according to the invention. For this purpose, the test device can have a processor device, which can be realized, for example, on the basis of one or more microprocessors. The described components of the test device can thus be realized in each case as program modules.

In the following an embodiment of the invention is described. This shows:

1 a sketch of a test run for a navigation system,

2 a flow chart diagram illustrating various embodiments of the method according to the invention,

3 a sketch to illustrate an extraction of several key objects from a single image,

4 a schematic representation of a time signal of an object feature of one of the key objects,

5 a schematic representation for illustrating a registration of the time signal with a reference signal,

6 a schematic representation for illustrating a review of an error criterion and

7 a schematic representation for illustrating an automatic determination of the error criterion by means of a method of machine learning.

The exemplary embodiment explained below is a preferred embodiment of the invention. In the exemplary embodiment, the described components of the embodiment each represent individual features of the invention that are to be considered independently of one another, which also each independently further develop the invention and thus also individually or in a different combination than the one shown as part of the invention. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

In the figures, functionally identical elements are each provided with the same reference numerals.

In 1 are a navigation system 1 and a test device 2 shown. The navigation system 1 can be part of an infotainment system. The navigation system 1 may be installed in a motor vehicle (not shown) or located in, for example, an experimental laboratory outside a motor vehicle. The test device 2 may for example be realized on the basis of a processor device, for example a personal computer. By means of the test device 2 becomes the navigation system 1 then checked to see if on a display device 3 of the navigation system 1 a picture sequence 4 is generated error-free with navigation instructions. For this purpose, a trip with the motor vehicle is readjusted. From the ride gets the navigation system 1 only with what sensor signals from sensors of the motor vehicle to the navigation system 1 send or transmit. Accordingly, by the test device 2 sensor data 5 in the navigation system 1 fed. This is called a so-called sensor data playback (re-import of sensor data). Alternatively, the described demo mode can be used.

The test device 2 recorded meanwhile from the navigation system 1 Single images 6 the picture sequence 4 and checks based on the individual images 6 whether the picture sequence 4 an error criterion 7 Fulfills. If the filter criterion 7 is met, is a warning signal 8th which generates a user (not shown) of the test device 2 signals that the image sequence 4 contains an error. The user can then target this image sequence 4 look again to see the error. If the image sequence does not contain an error here, another alert signal ("image sequence okay") may signal that the user is viewing the image sequence 4 do not have to check. As a result, the user is spared to look through each image sequence. He can focus on those image sequences that are in accordance with the error criterion 7 contain an error.

Overall, this results in a test run 9 for the navigation system 10 , This test lead 9 can with the same sensor data 5 or repeated with other sensor data. Even when using the same sensor data 5 can other frames 6 Result, which is related, that each time a different load, ie other operating conditions in the navigation system 1 due to other processes in the navigation system 1 or in the infotainment system. These effects are known in the art.

2 illustrates how by means of the test device 2 different ways of checking the navigation system 1 can be realized. 2 shows how multiple test runs 9 can be performed, with a Test run represents a reference sequence REF and the remaining test runs 9 one test sequence each TEST. To every test run 9 are each the described frames 6 the picture sequence 4 determined.

In the single pictures 6 then becomes an extraction 10 of key graphical objects 11 performed, which is described in more detail below 3 is described. To the key graphical objects 11 are then each for each frame 6 characteristic values 12 determined for different object features (see below the description in connection with 3 ). From the characteristic values 12 are then through an object analysis 13 time signals 14 generated, wherein for the reference sequence REF is a reference signal 15 and for a respective test sequence TEST, a corresponding test signal 16 every key object 11 and for each object feature results (see below 4 ).

If one renounces the reference sequence REF, then each test signal 16 in a store 17 stored and tested on the basis of a first test T1 by means of a threshold-based error detection without reference, which in connection with 4 is explained in more detail.

Do you save the reference signal 15 and the test signal 16 in a respective memory 18 In a test T2, a threshold-based error detection with reference can be performed. For a temporal equalization of the signals 15 . 16 can be an interpolation and registration 19 be provided what follows in the context of 5 and 6 is explained in more detail.

The described threshold-based comparison in the test T2 can be further automated by adding, for each reference signal 15 a feature distribution 20 determined and based on the feature distribution 20 decision limits 21 be determined as a tolerance range. Decision boundaries can be determined by a machine learning method ML. Do you save these decision limits 21 in a store 22 off and the test signals 16 in a store 23 , so can be performed in a test T3 machine-learned error detection with reference. This is related to 7 described.

3 shows from the image sequence 4 captured frames 6 , Every single picture 6 Represents several key objects 11 represents, for example, a vehicle symbol 24 for the own vehicle, a planned driving route 25 , a traffic flow indicator 26 , a road 27 , In general, these are key objects 11 shown in different colors. By extraction 10 can the key objects 11 segmented or separated or individually and independently recognized. For example, the representation color can be used for this purpose as an extraction criterion or recognition criterion. 3 shows how through the extraction 10 the route 25 , the traffic flow indicator 26 and the road 27 can be separated. In a synopsis 28 is once again illustrated how the combination 29 the key objects 11 essentially the single frame again 6 results. To every key object 11 can now for each frame 6 one characteristic value each 12 be determined. For example, in the illustrated example, the number N of pixels with which the respective key object 11 will be counted.

4 illustrates how this object feature 30 (for example, number N of pixels) by combining all the feature values 12 of the object characteristic 30 at a time signal 14 be combined over time t the change or development of the object feature 30 describes. 4 illustrates the passage of time 14 for the test signal 16 ,

The error criterion 7 For example, only for the test T1 can a threshold interval be used 31 specify in which the test signal 16 must lie. Will the error criterion 7 injured, so in this case lies the time signal 16 outside the threshold interval 31 , so for this aberration 32 the warning signal 8th is produced. The warning signal 8th can also specify at what time in the image sequence 4 the picture error 32 was detected.

5 illustrates how two time signals 14 namely, a reference signal 15 and a test signal 16 , can be registered with each other. This is done first by re-sampling or interpolation 19 ( 2 ) every time signal 14 adapted to a given sampling rate, so that feature values 12 , which were recorded at different times, each with a corresponding feature value 12 ' can be compared at the same time.

5 further illustrates how by means of a window-based comparison 32 by moving 33 the test signal 16 opposite to the reference signal 15 an offset value 34 can be determined by which the test signal 16 opposite to the reference signal 15 must be shifted over the time t, so that apart from image errors, an optimal coverage of the two time signals 14 results. As a comparison measure, for example, the described sum of the absolute differences SAD can be calculated for each time shift value.

6 then illustrates how by subtracting the reference signal 15 and the test signal 16 a difference signal 35 that is, the difference dN of the number of pixels in the illustrated example. In the test T2 can now be used as an error criterion 7 again a value interval 36 be defined, which indicates that difference signal 35 within the value interval 36 must lie, so there is no error. In addition to the picture error 32 , which was also detected in the absolute value analysis in the T1 test, can now be compared with the reference signal 15 a second mistake 37 be detected in the illustrated example.

7 illustrates, as for the test T3 by means of machine learning first an error criterion 7 in the form of a decision boundary 38 can be determined. For this purpose, two reference signals 15 . 15 ' for two key objects 11 determined. For example, it is assumed below that the two key objects 11 around the traffic flow 26 ("Speed &flow") and the road 27 is. For this reason, reference is made to this example below as "speed-and-flow example". As object feature 30 In this case, the number N of pixels displayed is again evaluated in the example in each case. Hereby every reference signal becomes 15 . 15 ' considered individually for each time, so that so the individual signal values of the two reference signal 15 . 15 ' be applied. By combining a respective signal value of the reference signal 15 and the reference signal 15 ' at any given time, one data point will result 39 of which in 7 for clarity, only a few are provided with a reference numeral. By means of machine learning ML, for example, by applying radial basis functions, the decision boundary 38 to determine which most or all data points 39 includes. By setting a per se known machine learning parameter ML, alternative borderlines can be created 40 for the decision limit 38 can be set, so by means of the parameter so the error rate can be set.

In test T3, corresponding data points can now be used 41 a test sequence TEST be determined. Are these data points 41 outside the decision boundary 38 , so this is also an aberration 42 detected in the test sequence TEST.

The embodiments of the method according to the invention which are described concretely with reference to the exemplary embodiment can be modified in order to test the navigation system 1 adapt to different test conditions. For this purpose, further notes for the modification are given below.

The concept presented is based on the repetitive recording of the navigation screen content at the same distance. In addition to the well-known demo mode of navigation, this is made possible in particular by the recently available technology of sensor data replays: the sensor data of a several-hour, real drive are re-recorded in the lab in the navigation. The recording of the first run is referred to below as the reference sequence, the recordings of all subsequent recordings as test sequences.

• 1. Berechnung von Schlüsselobjekt-Merkmalsverläufen
• 2. Synchronisierung der Sequenzen, sowie
• 3. Fehlererkennung mit/ohne maschinellem Lernen

The process basically comprises three components:

• 1. Calculation of key object feature histories
• 2. Synchronization of the sequences, as well
• 3. Error detection with / without machine learning

In combination with each other, these three components allow a reference-based, automated testing of the navigation, which eliminates or significantly reduces the four disadvantages described. In addition, rare, as yet unknown navigation errors can be detected automatically without explicit modeling of the error. 2 shows the basic process chain of the presented concept. Depending on the decision as to whether an error detection with / without reference sequence as well as with / without machine learning should take place, different procedures apply, which are indicated with the appropriate arrows.

The basis is the definition of elementary objects of navigation, here called key objects. These key objects allow a simple characterization of the behavior of important navigation elements as well as a renunciation of traditional pattern recognition (= "Pattern Matching", comparison of texture content). 3 clarifies this with an example. Here, the objects Route, Speed & Flow and Roads have been selected as key objects. Each of the selected key objects is recognized and extracted by the associated color in the frame. Then, for each key object, a characterizing key object feature is calculated (= number), for example the number N of pixels of this object. The sequence results in a key object characteristic curve (= curve or time signal) for each key object, which reliably maps the behavior of the key object and makes it comparable to a defect-free reference sequence.

• 1. Definition zeitlicher Schlüsselobjekt-Merkmale, beispielsweise von affinen Bewegungsparametersätzes zwischen Einzelbildern oder Frames,
• 2. Extrahierung der Schlüsselobjekte nicht aufgrund der Farbe, sondern aufgrund (einfacher) Mustererkennungsalgorithmen,
• 3. Berechnung eines Schlüsselobjekt-Merkmalstupels, also eines mehrdimensionalen Merkmals (siehe Bewegungsparametersatz),
• 4. Einbeziehung von Kontexten und Korrelationen bei der Berechnung und Verarbeitung von Schlüsselobjekt-Merkmalen und Schlüsselobjekt-Merkmalsverläufen (z. B. Einbeziehung von Bewegungsparametersätzen bei der Threshold- oder ML-basierten Fehlererkennung, Threshold-Schwellwert).

It should be noted that this concept is arbitrarily adaptable / expandable. Examples of a possible adaptation / extension would be:

• 1. Definition of temporal key object features, for example of affine motion parameter etching between individual frames or frames,
• 2. extraction of the key objects not by color but by (simple) pattern recognition algorithms,
• 3. calculation of a key object feature tuple, ie of a multi-dimensional feature (see motion parameter set),
• 4. Inclusion of contexts and correlations in the calculation and processing of key object characteristics and key object feature histories (eg inclusion of motion parameter sets in threshold or ML-based error detection, Threshold Threshold).

The availability of a reference sequence ( 2 , left) also allows the use of machine learning (ML) methods. An example of this is Novelty Detection, a one-class classification method that is closely related to statistical outlier detection. As part of testing, Novelty Detection has been experimentally combined with key object feature calculation and adapted to allow learning of the expected behavior of certain key object features based on the reference sequence.

This allows any number of test iterations (= test sequences) to be automatically entered into the classifier thus learned and to identify rare bugs based on possible "unexpected" behavior. In comparison with threshold-based or threshold-based error detection mechanisms, the evaluation by humans is simplified in addition, since the computer has learned the expected behavior of the key objects, this knowledge using classifier maps and thus from hundreds of hours of test iterations only unusual events to the tester ( Human). This basically corresponds to a reference-based preparation and filtering of existing key object information.

• 1. Berechnung von Schlüsselobjekt-Merkmalsverläufen
• 2. Synchronisierung der Sequenzen, sowie
• 3. Fehlererkennung mit/ohne maschinellem Lernen

eine Erprobung der Navigation in automatisierter Form. Zusammengefasst weist die vorgestellte Idee folgende Vorteile auf:In combination with each other allow the three components

• 1. Calculation of key object feature histories
• 2. Synchronization of the sequences, as well
• 3. Error detection with / without machine learning

a trial of navigation in automated form. In summary, the idea presented has the following advantages:

1. Low modeling effort

The need to model error behavior (explicit or implicit) is largely eliminated because a reference sequence is available for a computational comparison and the comparison can be performed reliably using the presented synchronization.

2. Reduced computational complexity

The computational effort for complex pattern recognition is eliminated, the calculation can be performed in approximately real time. Most of the effort involved is logical indexing, curve interpolation and, if desired, learning the decision boundary together (once).

3. Reduced evaluation effort

The mapping of elementary key objects of the navigation as simple curves and their difference value formation after synchronization allow for the first time a temporal processing of the image content of entire test runs in plot form. Common evaluation software tools such as EXAM do not support this: the sequence must always be checked by eye in a full-length video player or full-length picture browser. Through the representation of the navigation image content in the form of a curve, a quick, uncomplicated evaluation of the respective test run by the tester (human) is possible.

4. Use of reference information

Due to the described concept of features and feature histories of key objects, combined with the presented synchronization, it is now possible for the first time to carry out a computer-aided comparison against an error-free reference sequence with little effort. The tester thus has the opportunity to check the first run (= reference sequence REF) with the eye for freedom from errors, and then to repeat the test as often as desired with minimal evaluation effort (automated test of the test sequences TEST).

5. Detection of rare and unknown errors

The usability of a reference sequence can be interpreted as an implicit modeling of unknown errors (or as an explicit modeling of the expected behavior). This in turn makes it possible to identify unknown errors based on their unexpected key-object behavior without creating and maintaining implicit models with massive effort. The identification of rare, unknown errors is a major challenge in the field of automated navigation testing.

6. Machine Learning ML

Due to the described concept of features or characteristic curves of key objects, expected properties or expected behavior can be interpreted as feature tuples. In machine learning, these feature tuples (also called "features") form the basis for the calculation of a descriptive model (= training of a model). The model (often a parameter set of a "kernel" such as the RBF kernel) can then be used to classify features of unknown classes. The presented concept thus opens up the field of machine learning for future approaches.

7. Temporal. Characteristics, calculation of a context

The presented concept of features or characteristic curves of key objects currently uses "local features" (→ number of pixels of extracted key objects). "Temporal features" such as the movement between frames can also be considered in the future. Likewise, the introduction of a "context" is possible, ie an error detection on the basis of the nearer past of the considered characteristic course as well as in correlation with other characteristic courses. It is likely that an automatic "learning" of these networks (Bayesian networks) based on the reference sequence is a simple solution for an uncomplicated, low-maintenance testing of the navigation.

In the following, the technical implementation of the presented idea will be described in principle.

Calculation of Key Object Characteristics

Instead of searching for texture similarities (pattern matching, pattern recognition), elementary navigation elements are identified by means of color-based extraction. These elementary objects are referred to below as key objects. Other important terms in the following are also the key object characteristic and the key object characteristic course.

Depending on the purpose of the automated tests, the key objects can be chosen differently. Examples of key objects are roads, route, speed & flow, vehicle symbol or others (see, for example) 3 ). It is taken advantage of the fact that these key objects are typically monochrome (used for the purpose of clear graphical presentation of the navigation). With the help of the color feature, the key object can be uniquely extracted as a collection of pixels. The number of pixels gives the key object feature. Other key object features than the pure number of pixels are conceivable. However, it is important for subsequent synchronization and error detection that the key object feature per frame corresponds to a single number. This ensures that the behavior of a key object can be easily characterized using a simple but representative waveform for the entire sequence under consideration. A key object feature history is in 4 exemplified. In addition, this convention supports graphical processing and thus the evaluation by the tester (human). It has been shown that simplification to a one-number feature represents an optimal tradeoff between unambiguous characterization and manageability of the feature histories.

From these processes it is possible to deduce the correct function of the individual key objects of the navigation (more on this under Error detection with / without machine learning). They are also essential for resolving the disadvantages of modeling effort and computational complexity: they implicitly model the expected behavior; the complexity is limited to logical indexing and a feature calculation (here: the sum of the key object pixels). The key object feature trajectories also create the basic requirement for the usability of a reference sequence, which considerably simplifies the search for rare errors.

Synchronization of the sequences

• 1. Interpolationsschritt: Die Schlüsselobjekt-Merkmale aller Frames einer Sequenz stellen eine Serie an Einzelwerten dar (→ Merkmalsverlauf). Diese Einzelwerte sind als Abtastwerte zu gewissen Zeitpunkten anzusehen. Die Abtastung besitzt eine variierende Abtastrate. Es ist daher nötig, diese unregelmäßige Abtastung so zu interpolieren, dass zu jedem regulären Zeitschritt (z. B. jede Sekunde) ein Abtastwert (= Schlüsselobjekt-Merkmal) existiert.
• 2. Registrierungsschritt: Der Registrierungsschritt beinhaltet einen fensterbasierten Vergleich der interpolierten Merkmalsverläufe von Referenz-Sequenz und Test-Sequenz. Dabei wird für jede Position der SAD (Summe der Absoluten Differenzen) der Schlüsselobjekt-Merkmale berechnet und jene Position für die Registrierung verwendet, die den minimalen SAD aufweist.

The Synchronization component is essential in order to be able to automatically compare the key object characteristic curves of a test sequence with an already tested, error-free reference sequence. This reference comparison is highly interesting because it is an effective tool for detecting previously unknown errors (explicit modeling of the error is not possible, implicit modeling requires a lot of effort). This reference comparison can only be performed if the synchronizing of the two time axes is ensured for the time period to be compared. Given the varying frame rates, the asynchronous, internal clocks of the frame grabber and the load of the test bench PCs, this is not trivial. Synchronization consists of two simple steps:

• 1. Interpolation step: The key object characteristics of all frames of a sequence represent a series of individual values (→ characteristic curve). These individual values are to be regarded as samples at certain points in time. The sample has a varying sampling rate. It is therefore necessary to interpolate this irregular sampling such that at each regular time step (e.g. Second) a sample (= key object feature) exists.
• 2. Registration Step: The registration step includes a window-based comparison of the interpolated feature trajectories of reference sequence and test sequence. In this case, for each position, the SAD (sum of absolute differences) of the key object features is calculated and the position for the registration that has the minimum SAD is used.

By means of this synchronization, a section in the test sequence can be uniquely assigned to the corresponding section in the reference sequence. The synchronization makes a reliable, meaningful comparison of two sequences possible, even with long recording times.

Error detection with / without machine learning

• 1. Threshold-basierte Fehlererkennung, ohne Referenz (Test T1)
• 2. Threshold-basierte Fehlererkennung, mit Referenz (Test T2)
• 3. Machine-Learning-basierte Fehlererkennung, mit Referenz (Test T3)

The purpose of error detection is to recognize one or more faulty frames (in series or distributed over the sequence) as faulty and to pass it on to the tester (human) for the final assessment. In this case, a high number of incorrectly reported errors (false positives, errors does not exist in reality) to avoid, otherwise the final evaluation by the tester is considerably more difficult. Basically, three different approaches to automated error detection by means of the method described in this invention message possible (see 2 ):

• 1. Threshold-based error detection, without reference (test T1)
• 2. Threshold-based error detection, with reference (test T2)
• 3. Machine-learning-based error detection, with reference (test T3)

T1: Threshold-based error detection, without reference

This is the simplest approach, but with a significant number of false positives, and only suitable for a few key objects. An algorithm checks the maximum / minimum values of a characteristic curve and decides whether they exceed or fall below a defined threshold. The jumping behavior of the characteristic curve can also be checked (via differences, eg interesting with blue route or road network). 4 shows an example. It has also been shown that a weighting of the jump behavior with the respective sampling interval can significantly reduce the number of false positives.

T2: Threshold-based error detection, with reference

Through synchronization and reference sequence comparison, there is considerably more information about the expected key object feature history than with a referenceless comparison. For each key object, a precise calculation of the difference value between the reference and the test characteristic course is thus possible. For some key objects, smoothing the difference curve (eg convolution with Gauss curve) before a subsequent threshold calculation makes sense.

T3: Machine-learning-based error detection, with reference

It has been found that certain key object characteristics are suitable for evaluation by statistical classification methods. It is thus possible to apply Machine Learning (ML) approaches to the calculated key object characteristics and key object feature histories. For the purpose of investigating this possibility, the so-called Novelty Detection method was applied to the characteristics Speed & Flow and Street Network.

It should be noted that also more than two key object features can be learned in their correlation. However, the higher-dimensional feature space makes it difficult to easily visualize the results.

The basic procedure can be deduced from the above-mentioned speed-and-flow example ( 7 ) describe in three steps:

1. Training by reference sequence

Based on the reference sequence z. For example, the correlation of "common" Speed & Flow features to Street Network features (SF-vs-SN) is learned. Each processed frame yields one data point 39 ( 7 ). Based on these data points 39 becomes a decision boundary or decision boundary by means of an RBF kernel 38 calculated. There is thus a mathematical description of the expected behavior. Outside the decision boundary 38 Data points (= frames) are classified as faulty.

2. Processing a test sequence

This is illustrated by two examples. Example 1: Processing of an error-free test sequence. As expected, SF-vs-SN data points embedded in error-free SF-vs-SN data points, ie within the decision boundary 38 , Example 2: Processing of a faulty test Sequence (disturbed speed & flow behavior). The SF vs SN data points 42 the disturbed frames are off the expected distribution, ie outside the decision boundary 38 ,

3. Error detection using decision boundary 38

The error detection 42 can thus by means of the learned Decision Boundary 38 respectively. Since the behavior of the selected key objects has been learned based on the reference sequence and is mapped using the decision boundary, automated error detection is thus possible for each incoming navigation frame.

In the Decision Boundary learned in step 1 38 It is important to select the parameters of the Novelty Detection that are necessary for the calculation in such a way that the most comprehensive possible inclusion of all data points of the error-free reference sequence is guaranteed. This can be done via the known parameter "v"("Nu"). This parameter in the range 0 to 1 corresponds to the ratio of the training errors to the total number of training data points and acts as an upper limit for the maximum permitted training errors. A v <= 0.01 has proven to be completely sufficient for the tested navigation sequences ( 7 ).

Overall, the example shows how navigation errors by means of key object analysis and reference sequence comparison can be detected by the invention.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102012009876 A1 [0003]
• DE 102008012697 A1 [0004]
• DE 102013015145 A1 [0005]

Claims (10)

Method for testing a navigation system ( 1 ), whereby by a test device ( 2 ): - in a test run ( 9 ) Sensor data ( 5 ) describing a journey of a motor vehicle into the navigation system ( 1 ) or within the navigation system ( 1 ), and - frames ( 6 ) of a picture sequence ( 4 ), the navigation system ( 1 ) in dependence on a predetermined navigation target and the sensor data ( 5 ), each frame ( 6 ) a combination ( 29 ) from graphic key objects ( 11 ), and - at least one of the key graphical objects ( 11 ) is selected based on configuration data and - in the individual images ( 6 ) each selected key object ( 11 ) and - of each extracted key object ( 11 ) to at least one predetermined object feature ( 30 ) a respective characteristic value ( 12 ), and - for each object feature ( 30 ) the determined characteristic values ( 12 ) of all recorded individual images ( 6 ) to a time signal ( 14 ), and - it is detected whether at least one time signal ( 14 ) a predetermined error criterion ( 7 ), and - if the error criterion ( 7 ), a warning signal ( 8th ) is produced. The method of claim 1, wherein the at least one object feature ( 30 ) comprises: a scalar value, in particular a pixel number (N), and / or a dynamic property and / or a feature tuple of a plurality of individual features. Method according to one of the preceding claims, wherein the at least one object feature ( 11 ) is dependent on a navigation context and / or describes a correlation of second predetermined individual features. Method according to one of the preceding claims, wherein the extraction ( 10 ) of the at least one key object ( 11 ) takes place on the basis of a color and / or a graphic pattern. Method according to one of the preceding claims, wherein the test run ( 9 ) is repeated several times, and each time signal ( 14 ) one of the test runs ( 9 ) each as a reference signal ( 15 ) and each time signal ( 14 ) of another of the test runs ( 9 ) each as a test signal ( 16 ) and the error criterion ( 7 ) that a difference ( 35 ) between at least one test signal ( 16 ) and the corresponding reference signal ( 15 ) outside a predetermined tolerance range ( 36 . 38 ). Method according to claim 5, wherein the reference signal ( 15 ) and / or the test signal ( 16 ) to one for the scanning of the image sequence ( 4 ) predetermined sampling rate is interpolated. Method according to claim 5 or 6, wherein the test signal ( 16 ) with the reference signal ( 15 ) by a window-based comparison ( 32 ) is registered. Method according to one of claims 5 to 7, wherein a further reference signal ( 15 ' ) is generated and from the further reference signal ( 15 ' ) by means of a method of machine learning (ML) the tolerance range ( 38 ) is determined. Method according to claim 8, wherein the respective feature values of at least two reference signals ( 15 . 15 ' ) as coordinates of a data point ( 39 ) and from the data points ( 39 ) of the reference signals ( 15 . 15 ' ) by means of radial basis functions the tolerance range ( 38 ) as a decision boundary ( 38 ) containing most or all data points ( 39 ) is determined. Test device ( 2 ) for a navigation system ( 1 ) of a motor vehicle, wherein the test device ( 2 ) is adapted to perform a method according to any one of the preceding claims.

Images