WO2016097776A1

WO2016097776A1 - Method for changing the content of an image segment

Info

Publication number: WO2016097776A1
Application number: PCT/HU2015/050024
Authority: WO
Inventors: Péter SZARVAS; Szabolcs Mike
Original assignee: Pi Holding Zrt
Priority date: 2014-12-17
Filing date: 2015-12-17
Publication date: 2016-06-23
Also published as: HUP1400600A2

Abstract

The invention relates to a method for the replacement of the content of image segments. The method comprises creating a stereoscopic image pair (14) consisting of a first and second image (14a, 14b) using two cameras (1 2a, 1 2b) of a region of space containing a geometrical object (1 0), in image segments (151) corresponding to a common field of view of the two cameras (1 2a, 1 2b) detecting feature point pairs (1 8) consisting of feature points (1 8a, 1 8b) belonging to first and second object images (1 6a, 1 6b) of the geometrical object (1 0) located in the first and second images (14a, 14b), using the feature point pairs (1 8) for determining a zeroth equation transforming the feature points (18a, 1 8b) of the surface of the geometrical object (1 0) in the first image (14a, 14b) into the corresponding feature points (1 8a, 1 8b) in the second image (14a, 14b), and in the two images (14a, 14b) replacing the content of image segments determined by feature point pairs (1 8) satisfying the zeroth equation and the content of image segments determined by point pairs (28) consisting of further points (28a, 28b, x₁, X₂) satisfying the zeroth equation for other target content.

Description

Method for the replacement of the content of image segments

The invention relates to a method for the replacement of the content of image segments.

High-ranking sports events (the Olympics, world football championships, NBA, etc.) usually attract great international interest. These events are usually broadcast live by the various television channels in several countries at the same time.

The viewer at such events can usually see advertisements and logos at the event venue that are made using a traditional method, i.e. printing technology, or perhaps digital advertising boards. The greatest disadvantage of printed advertising surfaces is that their content is permanent, and the boards are replaced only rarely (every season, or every day at the maximum), furthermore on the basis of their content and location, they are targeted at the viewers at the venue on most occasions. Although digital advertising boards may continuously change their displayed content, these are not suitable for parallel advertising services destined for various markets either.

A further disadvantage of the use of traditional advertising boards may be that due to the movement of the camera their content may become blurred, making them unrecognisable for television viewers.

The greatest disadvantage of printed and on-site digital boards is the aforementioned fixed content, in other words, they are not able to display content that is appropriate for displaying different contents appropriate for the demands of the local markets.

Several methods are known of in the literature during which certain parts of a video recording are identified, then the identified parts are replaced by other pictures and/or videos. The changing of the video parts may also take place in real time, therefore it is possible to replace the physical advertising boards visible in the live broadcasts for virtual advertising boards. As the replaced, outgoing digital signals may be multiple and different to each other, therefore content broadcasting appropriate for the demands of the local markets can be ensured. A further advantage of digital advertising surfaces is that the same offline advertising surface may be sold several times, furthermore, it is worthwhile for the local service providers to advertise as well, as it is not necessary to purchase the advertising right for the entire market, but just for the regions required.

The critical part of the methods dealing with virtual content replacement is the identification of the area to be changed in the recording. There are currently three main techniques used for this, which are the following:

· the area to be digitally changed is physically marked, on the basis of which an image recognition algorithm is able to determine its position in the recording with a pattern recognition procedure

• the essence of the second method is that the floor plan of the broadcast venue (e.g. stadium) and, within this, the position of the area (e.g. advertising board) to be replaced, the position of the camera and its direction are known, then from this information it is determined whether the surface to be replaced falls within the image recorded by the camera and what part of the image it is in

• the third possibility is that signal generators transmitting in a range invisible to the human eye are located on the surface to be replaced, which signals are only visible to the camera, and so the surface can be detected.

Patent document number US6297853B1 presents a method which uses the video signal (pattern recognition) alone to automatically identify given areas in the recording, and then perform the desired modifications on the identified surfaces. The procedure takes into consideration the perspective distortions originating from the various points of view, any obscuring of the surfaces as well as the effects originating from the camera's zoom changes.

Patent number US6208386B1 describes how it is possible to replace real surfaces with virtual surfaces. Furthermore, it calls attention to that procedures using pattern recognition alone may be inaccurate in practice. The reason for this is that pattern recognition requires good quality image frames, which is not always the case during live broadcasts, for example, due to exaggerated occlusion, the - - fast movement of the camera, or fast zoom changes. Due to this a method is proposed in the case of which first of all sensors are used to determine the position of the image portion to be replaced, then the result obtained is made more precise using a pattern recognition procedure.

The essence of the invention described in patent number EP0935886B1 is a signal processing system that has sensors and a camera which are able to identify certain details of a recording using pattern recognition and then change them in real time and then re-broadcast the modified recording. It recognises a given pattern even when it is distorted by using the sensors, and distorts the image/advertisement according to this in order to provide a real experience for the TV viewers. Among the possibilities of use, it mentions surfaces that are only relevant for the TV viewers, such as, for example, the glass wall in ice hockey matches. Due to the fans in the stadium it is not possible to place advertisements onto the glass wall, because this would obscure the match.

In the case of virtual content replacement, it must also be taken into consideration that when an object (for example a player) moves in front of an advertising surface, it must not be covered by a virtual advertisement, instead the advertisement must remain in the background.

In the case of the currently existing solutions, the appropriate handling of occlusions represents a serious problem. In the case of procedures based on pattern recognition alone and signal generators, the method has worse efficiency on the border of the occlusion and may even be inoperable. In certain cases a further problem is presented by that the content replacement does not take place in real time and is not automated, instead it requires the involvement of an operator.

The aim of the invention is to provide a device and method that is free of the disadvantages of the solutions according to the state of the art.

It was recognised that in the case of the use of a 3D (stereoscopic) camera, the above problems may be overcome with surface detection based on stereo pixel correlation.

It was also recognised that the points of the surface of a typically planar advertising board may be determined with great accuracy in the images of 3D cameras by that the coherent planar surface falling in the common field of view of - - the two cameras may be algorithmically detected, as with good approximation the projections of the planar surface in the stereo images are each other's homographic projections. In other words, mathematic correlation may be found between the points of the planar surface projected onto the stereoscopic image pair, which may be used for segmenting the image, i.e. for identifying the image segment to be replaced.

It was also recognised that the procedures used during traditional 2D image content searching may also be used during the correlation of stereoscopic image points, what is more the advantage of stereoscopic images recorded using a stereoscopic camera is that in stereoscopic images the pixels belonging to a given point in space are not arbitrarily distant from each other, therefore a more local search is sufficient. A further advantage of the method is that it is also possible to restore the depth information of the objects.

On the basis of the above recognition the set task is solved with the method according to claim 1 .

Preferred embodiments of the invention are specified in the dependent claims.

Further details of the invention will be explained by way of exemplary embodiments with reference to the drawings. In the drawings

Figure 1 a shows a schematic view of the main elements participating in the method according to the invention,

Figure 1 b shows the unmodified images of the two cameras of the stereoscopic camera in the case of the arrangement according to figure 1 a,

Figure 1 c shows the modified images of the two cameras of the stereoscopic camera in the case of the arrangement according to figure 1 a,

Figure 2 shows a schematic illustration of the first embodiment of the method according to the invention,

Figure 3 shows a schematic illustration of the second embodiment of the method according to the invention.

The main elements and main components participating in the method according to the invention may be seen in figures 1 a, 1 b and 1 c. During the method according to the invention a stereoscopic image pair 14 - containing a first image 14a and a second image 14b - is created with a stereoscopic camera 12 - - containing two cameras from the region of space 1 1 containing the geometrical object 10 in the 3-dimensional space (the prefixes "first" and "second" do not indicate order, they are merely attributes). The first camera 12a and the second camera 12b of the stereoscopic camera 12 are preferably located at a fixed distance from each other, and see the geometrical object 10 from different perspectives. The geometrical object 10 appears in the first image 14a of the stereoscopic camera 12 as a first object image 16a and in the second image 14b as a second object image 16b. The image segments 151 complying to the common field of view of the two cameras 12a, 12b and the image segments 15k outside of the common field of view have been separated from each other by a dotted line in figure 1 b. Feature point pairs 18 belonging to the geometrical object 10 and consisting of feature points 18a and 18b corresponding to each other in the images 14a and 14b may be determined, as will be explained later on. The feature points 18a, 18b are those characteristic areas of the recording that are relatively easy to separate from the other parts of the recording, and their detecting is less sensitive to changes in angle of view and illumination (for example, edges, corners, high-contrast areas).

In a given case there may an occluding object 20 between the stereoscopic camera 12 and the geometrical object 10, which appears as occluding object 20a and 20b in the stereo images. During the method according to the invention the object images 16a and 16b are identified then replaced with other desired virtual object images 17a and 17b (see figure 1 c).

In the following the first embodiment of the method according to the invention is presented with reference to Figures 2a and 2b.

Figure 2 shows a schematic illustration and overview flow chart of the first embodiment of the method according to the invention. As the first step of the method, with the stereoscopic camera 12, a stereoscopic image pair 14 containing the first image 14a and the second image 14b is made of the region of space 1 1 containing the geometrical object 10, and a reference image 14c is also made, in which the geometrical object 10 appears as the reference object image 16c. Preferably the reference image 14c is made in advance either with one of the cameras 12a, 12b or with a separate camera. - -

In the second step preferably pre-processing and other image improvement operations (for example noise filtering, pixel intensity normalisation, masking, etc.) are performed on the recordings made (images 14a, 14b, 14c). The second step does not take place during a single unit of time, for example the desired pre-processing steps may be performed on the reference image 14c in advance (even before the images 14a, 14b are recorded).

In the third step the feature points 18a, 18b, 18c belonging to the geometrical object 10 are searched for in the images 14a, 14b, and 14c and are correlated to each other. In the sense of the present invention point means pixel or pixel range.

The selection of the feature points 18a, 18b, 18c may preferably take place with the help of scale and rotational invariant local feature detectors that tolerate changes in angle of view and illumination well (for example MSER, SIFT, SURF, ORB, etc., which are algorithms well known to a person skilled in the art). The feature detector is used to calculate image descriptors 19c in a known way for the points of the reference object image 16c of the reference image 14c con the basis of the information of the surrounding pixels (or pixel ranges), then on the basis of the image descriptors 19c it is determined with respect to the given image descriptors 19c which points will behave as feature points 18c that are locally invariant and tolerate angle of view and illumination changes well. The feature points 18c may also be determined in other known ways, for example characteristic points (edges, corners, etc.) may be detected using convolution image filters, or by the image segmentation of areas with locally similar structures, or by using local wavelet transformations, etc.

In a similar way image descriptors 19a, 19b are calculated on images 14a and 14b, then the image descriptors 19a, 19b, 19c of the points of images 14a, 14b, 14c are compared to each other per pair with the help of a metric function μ.

The metric function μ allocates measurement numbers to the image descriptor pairs so that the smaller this measurement number is, the more similar the image descriptors 19a, 19b, 19c are considered. Another possible method of the correlation of the image descriptors 19a, 19b, 19c is that only the most similar image descriptors 19a, 19b, 19c are searched for where the metric function μ has a minimum. In the case of a preferable embodiment of the method according to the invention a threshold value may be given to the metric function μ above which the compared feature points 18a, 18b, 18c do not correlate with each other even if the metric function μ has a minimum in the case of the image descriptors 19a, 19b, 19c associated with them - as it may occur that a feature point 18c is not at all visible in one or the other image 14a, 14b (for example, it is obscured), in this case the metric function μ minimum provides a fake point correlation. It may be conceivable that various threshold values are given for the comparison of the feature points 18a, 18b, 18c of the individual images 14a, 14b, 14c, for example, the feature points 18a, 18b of two stereoscopic images 14a, 14b are correlated with each other in the case of a lower threshold value, as the images 14a and 14b are theoretically more similar to each other than to the reference image 14c, which, in a given case, was made in different light and weather conditions.

In the case that several image descriptors 19a, 19b, 19c belong to the given feature point at the same time, then the different image descriptors 19a, 19b, 19c may be preferably weighted, and the metric function μ may be normalised accordingly, i.e. scaled in order for the different image descriptors to be on the same unit scale for comparison.

As a result of the comparison process feature points 18a, 18b corresponding to the feature points 18c are determined in images 14a and in image 14b, which feature points 18a, 18b also correlate to each other in a given case, therefore feature point pairs 18 may be determined.

One possible way of making the method more robust is that the feature points 18a, 18b in images 14a and 14b are not only compared to the feature points 18c in reference image 14c, but to each other as well.

In the case of feature points 18c the corresponding point of which is obscured in either image 14a or 14b, naturally it is only possible to detect the non- obscured feature point 18a or 18b. Such points obscured in the one image 14a or 14b and, however, visible in the other image 14b or 14a also belong to the image segment 15k falling outside of the common field of view, in other words the image segment 15k is not necessarily continuous.

The image descriptors 19a, 19b, 19c, then, make it possible to detect the feature points 18a, 18b, 18c that correlate with each other. Beside this, well- chosen image descriptors 19a, 19b, 19c help overcome obscuring, - - inhomogeneous illumination or other disturbing conditions. In order to select the feature points 18a, 18b, 18c and correlate them with each other, one or, in a given case, several image descriptors maybe used at the same time, which may be known image descriptors, such as SIFT, SURF, LBP, ORB, FREAK, MSER, Shape context, etc.

Naturally other procedures may be considered in order to find and correlate the feature points 18a, 18b, 18c. An embodiment may be conceived in the case of which a reference image 14c is not used at all, instead the feature points 18a, 18b are searched for directly in image 14a and 14b and correlated with each other.

In the fourth step, with the help of at least four feature point pairs 18 consisting of feature points 18a, 18b corresponding to each other and identified in images 14a, 14b, a zeroth equation is determined that transforms the feature points 18a of the object image 16a of the geometrical object 10 appearing in the first image 14a into the corresponding feature points 18b in the second image 14b, in other words a zeroth equation that describes the transformations between the projections of the geometrical object 10. In the case of a preferred embodiment of the method according to the invention, the zeroth equation is a homographic equation that represents the linear transformation between two projected images of a planar surface in a 3-dimensional space, and may be written down in the following form:

λ*ν'=Ηο*ν

where Ho is the 3x3 homography matrix belonging to the linear transformation with eight degrees of freedom, λ is a scalar, and v and v' are the location vectors of the points belonging together of, in the present case, a planar geometrical object 10 (for example, an advertising board) visible in the first image 14a and in the second image 14b, where the x, y coordinates of the location vectors are expressed as the x, y coordinates of the coordinate system fixed to image 14a and 14b, and its z coordinates are an arbitrary value, such as 1 . If the feature points 18a, 18b are individual pixels, the coordinates of the v and v' vectors corresponding to the feature points 18a, 18b are preferably the x and y coordinates of the given pixels, and the z coordinates are selected to have a given value. If the feature points 18a, 18b are not individual pixels, but pixel ranges, then - - the coordinates of the vectors v and v' corresponding to the feature points 1 8a, 18b are preferably the x and y coordinates of a characteristic pixel of the given pixel range, as well as the selected z coordinates. Such a characteristic pixel may be the centre point or the lower right point of the pixel range, etc..

Due to the aforementioned eight degrees of freedom, and due to the fact that the z coordinates do not carry real information, in order to calculate the homography matrix Ho at least four feature point pairs 1 8 that belong together are required. If more feature point pairs 1 8 are available, several point pair fours may be used to calculate the Ho homography matrix, in this way it is possible to filter out the feature point pairs 1 8a, 1 8b incorrectly correlated with each other - for example with a RANSAC (random sample consensus) algorithm - through which the reliability of the method may be improved.

The zero homography equation is only satisfied by those point pairs belonging to each other in images 14a, 14b that also belong to the projections of the geometrical object 1 0, so after calculating the homography matrix Ho and the associated points v, v', it becomes possible to precisely segment the object images 16a, 1 6b corresponding to the planar surface of the geometrical object. The projections of the objects falling outside of the planar surface of the geometrical object 1 0 (for example, occluding object 20, or other foreground or background objects) do not satisfy the zero homography equation.

It should be noted that transformation equations also exist that correlate the projected images of non-planar surfaces with each other, as is obvious for a person skilled in the art.

In the fifth step the points 28a in the vicinity of the feature points 1 8a of the first image 14a are preferably transformed using the zero homography equation into the points 28b of the second image 14b. Image descriptors 29a, 29b are determined for the points 28a, 28b transformed into each other, which may be the same image descriptor functions as used to correlate together the feature points 1 8a, 1 8b, 18c, but they may also be different from them, as in the knowledge of the zero homography equation it is not essentially necessary to look for angle of view and scale invariant image descriptors, therefore simplified image descriptors 29a, 29b may also be used that require less calculation capacity. Following this the image descriptors 29a, 29b are compared with the help of the metric function - - μ. If the value given by the metric function μ for the transformed point pair 28 is, for example, below the previously defined threshold value, then the transformed points 28a, 28b are considered as a corresponding point pair 28 belonging to the geometrical object 10, and, according to that to be explained later on, they are replaced for the appropriate part of the desired virtual object images 17a and 17b.

In the case of another embodiment of the method according to the invention, the point pairs belonging to the surface of the geometrical object 10 and belonging to each other in images 14a and 14b are determined by creating image descriptors 29a, 29b for the points 28a, 28b in the vicinity of the feature points 18a and 18b that have been found earlier, which are then compared to each other with the help of the metric function μ. Following this the point pairs 28 correlated with each other in this way are checked to determine whether they satisfy the zero homography equation. If they do then they are considered as corresponding point pairs belonging to the geometrical object 10, and, according to that to be explained later on, they are replaced for the appropriate part of the virtual object images 17a and 17b.

With the help of the point pairs 28 corresponding to each other the object images 16a and 16b corresponding to the geometrical object 10 may be isolated in images 14a and 14b.

During the sixth step the content of the image segments determined by the object images 16a and 16b identified with the help of the point pairs belonging to each other is replaced for different target content in images 14a and 14b. For example the object images 16a and 16b are replaced for different virtual object images 17a and 17b (for example for the surface of an advertising board displaying a different advertisement), also taking into account the perspective distortions deriving from the different angle of view in a known way. The method according to the invention takes into account in a known way the various visual noises (for example rain, fog, mud splashes, etc.) in the interest of the content replacement being even more realistic.

The steps presented above may also be performed on several stereoscopic image pairs 14 forming a moving image series. As in the video recording the object 10 does not move significantly in two consecutive image frames, the feature points are searched for in the given images 14a and 14b in the - - vicinity of the points with coordinates complying to the feature points 18a, 18b determined in the previous images 14a and 14b of the moving image series. By using several previous stereoscopic image pairs 14 the movement of the feature points 18a, 18b may be tracked using so-called optical flow algorithms (for example Horn-Schunk, Lucas-Kanade), therefore it is not absolutely necessary to determine their coordinates in the way presented earlier.

It is noted that the steps presented here may be preferably performed automatically, without human intervention.

In the case of occlusion, in other words when there is an occluding object 20 between the geometrical object 10 and the stereoscopic camera 12, it may happen that it is not possible to find a point 28b in image 14b with one or more image descriptors 29a corresponding to a point 28a assigned in image 14a. To put it another way, using the Ho homography matrix it projects point 28a to a point 28b so that on comparing one or more of its image descriptors 29b with one or more image descriptors 29a of point 28a the metric function μ gives a higher value than the threshold value 23. As the two cameras 12a, 12b of the stereoscopic camera 12 record the images from slightly different angles of view, there may be some occluded image segments 15k, in other words dead zone ranges that the occluding object 20 obscures in one of the images 14a or 14b of the stereoscopic image pair 14, but not in the other image. In such a case, for example in image 14a, point 28a is a part of object image 16a, while the point 28b determined by the zero homography equation belongs to occluding object image 20b, therefore the image descriptors 29a and 29b differ.

The reference image 14c may be preferably used to handle such cases. Using the correlated, identified feature points 18a, 18b, 18c the first and second homography equations describing the transformation between the projections of, in the present case, planar geometrical object 10 can be seen in images 14c and 14a and in images 14c and 14b, are determined respectively. The latter may be searched for in the same form as the zero homography equation, i.e. with homography matrices Hi , or H2:

λι*ν"=Ηι*ν

A2*V"=H2*V' - - where Hi and H2 are the 3x3 homography matrices belonging to the linear transformation with eight degrees of freedom, λι , and λ2 are scalars, v, v' and v" are the location vectors of the points belonging together of the geometrical object 10 (for example, an advertising board) visible in the first image 14a, the second image 14b and in the reference image 14c respectively. The x, y coordinates of the location vectors are expressed as the x, y coordinates of the coordinate system fixed to the images 14a, 14b and 14c, and their z coordinates are a selected value, such as 1 .

If the feature points 18a, 18b, 18c are individual pixels, then the coordinates of the v, v' and v" vectors corresponding to the feature points 18a, 18b, 18c are preferably the x and y coordinates of the given pixels, and the z coordinates are selected to have a given value. If the feature points 18a, 18b, 18c are not individual pixels, but pixel ranges, then the coordinates of the vectors v, v' and v" corresponding to the feature points 18a, 18b, 18c are preferably the x and y coordinates of a characteristic pixel of the given pixel range, as well as the selected z coordinates. Such a characteristic pixel may be the centre point or the lower right point of the pixel range, etc.

In the knowledge of the first homography equation, the coordinates of the corresponding points v, v" in images 14a, 14c may be determined, while using the second homography equation the coordinates of the corresponding points v', v" in images 14b, 14c may be determined. If then the zero homography equation determines points 28a, 28b the image descriptors 29a, 29b of which do not correlate sufficiently (for example they are above the determined threshold value), then by using the first or the second homography matrix Hi , H2 it may be decided which of the points 28a or 28b belong to the object image 16a or 16b, and which do not, and in a given case it may be determined that none of them do. Using the homography matrix Hi the point 28c corresponding to the point 28a is searched for in reference image 14c. One or more image descriptors 29a and one or more image descriptors 29c of point 28a and 28c are determined, then the image descriptors 29a, 29c are compared with the metric function μ, and from the result of the comparison it is determined whether point 28a belongs to object image 16a, in other words, whether it needs to be replaced. If it is determined that point 28a belongs to object image 16a, then it is expected that point 28b does not, however, - - it is preferable to check this separately with the use of the homography matrix H2, in this way any irregular differences between images 14a and 14b (for example a raindrop on one of the cameras 12a, 12b) can be filtered out. If it is determined that point 28a does not belong to object image 16a, this in itself does not involve that point 28b does, because, for example, both of them may be a part of the occluding object 20a and 20b.

In a given case all three homography equations may be used, which makes the method more robust, and any point coordinate errors can be filtered out better, such as according to the following.

In the case of the method illustrated in figure 3, the point xo of reference image 14c is projected onto point xoi of image 14a using the homography matrix Hi, and projects it onto point X02 of image 14a using the homography matrix H2. Following this, point xoi of image 14a is projected onto point x-12 of image 14b, and point X02 of image 14b is projected onto point X21 of image 14a using the homography matrix Ho and its inverse. Point xi is formed from points xoi and X21 obtained in first image 14a, for example, by averaging or weighted averaging, while using a similar procedure point X2 is formed from points X02 and x-12 obtained in image 14b. Following this, using image descriptor function d image descriptors d(xo), d(xi), d(x2) are determined for points xo, xi and X2, then using the metric function μ belonging to the image descriptors d(xo), d(xi), d(x2), the points xo, xi and X2 are compared pair-by-pair using one or more threshold values for the correlation of the points. Preferably the given threshold value ε is used in order to compare the image descriptors d(xo), d(xi), and d(xo), d(x2) of points xo and xi, and xo and X2, while a lower threshold value ε' is used to compare the image descriptors d(xi), d(x2) of points xi and X2

The possible results are illustrated in Table 1 .

M(d(x₀),d(xi)) M(d(x₀),d(x2)) M(d(xi),d(x2)) replacement replacement

- Χ1 - Χ2

1 . <ε <ε <ε' 1 1

2. <ε <ε >ε' 1 1

3. <ε >ε <ε' 1 1 - -

Table 1

The first three columns of Table 1 show that the distance according to the metric function μ of the image descriptors d(xo), d(xi), d(x2) for the point pairs consisting of points x-i , X2 of the given stereoscopic image 14 obtained in the way described above is smaller or larger than the given threshold value ε, and ε' (the case of equality may be integrated into the smaller or larger case, as desired). The fourth and fifth columns show the decision relating to the replacement of point xi and X2: the value 1 indicates replacement and the value 0 indicates no replacement. In the case of this embodiment an algorithm is used that decides on replacement in the case of a distance between the images 14a and 14b that is smaller than the threshold value ε', except if all of the distances taken with image 14c are greater than the threshold value ε (see row number 7), in other words if comparison with reference image 14c unanimously indicates no replacement should be made (in other words neither xi , nor X2 correlate to xo, however xi and X2 correlate to each other). If the distance of points xi and X2 is greater than the threshold value ε' (in other words the points do not correlate to each other), then it is the comparison with point xo that decides which of xi and X2 belongs to the object image 16a or 16b to be replaced, and in such a case this point xi or X2 is replaced, while the point X2 or xi is not replaced (see rows number 4 and 6). Naturally decision rules that take into account other parameters may also be used, for example the decisions made on the previous images of the image flow may be used with respect to the points corresponding to the points xi and X2.

Various modifications to the above disclosed embodiments will be apparent to a person skilled in the art without departing from the scope of protection determined by the attached claims.

Claims

1 . Method for the replacement of the content of image segments, characterised by creating a stereoscopic image pair (14) consisting of a first and second image (14a, 14b) using two cameras (12a, 12b) of a region of space containing a geometrical object (10), in image segments (151) corresponding to a common field of view of the two cameras (12a, 12b) detecting feature point pairs (18) consisting of feature points (18a, 18b) belonging to first and second object images (16a, 16b) of the geometrical object (10) located in the first and second images (14a, 14b), using the feature point pairs (18) for determining a zeroth equation transforming the feature points (18a, 18b) of the surface of the geometrical object (10) in the first image (14a, 14b) into the corresponding feature points (18a, 18b) in the second image (14a, 14b), and in the two images (14a, 14b) replacing the content of image segments determined by feature point pairs (18) satisfying the zeroth equation and the content of image segments determined by point pairs (28) consisting of further points (28a, 28b, xi , X2) satisfying the zeroth equation for other target content.

2. Method according to claim 1 , characterised by creating a reference image (14c) of the geometrical object (10),

determining feature points (18c) belonging to an object image (16c) of the geometrical object (10) in the reference image (14c),

calculating image descriptors (19c) for the feature points (18c),

calculating image descriptors (19a, 19b) for the individual points of the first and second images (14a, 14b), and

comparing the image descriptors (19c) of the feature points (18c) of the reference image (14c) and the image descriptors (19a, 19b) of the individual points of the first and second image (14a, 14b) with each other pair-by-pair using a metric function (μ) and identifying the feature points (18a, 18b, 18c) that correspond to each other as the point triplets that give the minimum for the metric function (μ).

3. The method according to claim 1 or 2, characterised by determining further point pairs (28) by transforming further points (28a) of the first image (14a) into the points (28b) of the second image (14b) using the zeroth equation, determining image descriptors (29a, 29b) for the points (28a, 28b) transformed into each other, comparing the image descriptors (29a, 29b), and depending on the result of the comparison, considering the points (28a, 28b) transformed into each other as corresponding point pairs (28).

4. The method according to claim 1 or 2, characterised by determining further point pairs (28)by creating image descriptors (29a, 29b) for the points (28a,

28b) in the vicinity of the found feature points (18a, 18b), comparing the image descriptors (29a, 29b) with each other, then calculating whether the corresponding further point pairs (28) satisfy the zeroth equation.

5. The method according to claim 2, characterised by determining a first equation transforming the feature points (18c) of the surface of the geometrical object (10) in the reference image (14c) into the corresponding feature points (18a) in the first image (14a), and in the two images (14a, 14c) replacing the content of the image segments determined by the corresponding feature points (18a, 18c) and further points (28a, 28c) satisfying the first equation for other content,

determining a second equation transforming the feature points (18c) of the surface of the geometrical object (10) in the reference image (14c) into the corresponding feature points (18b) in the second image (14b), and in the two images (14b, 14c) replacing the content of the image segments determined by the corresponding feature points (18b, 18c) and further points (28b, 28c) satisfying the second equation for other content.

6. The method according to claim 5, characterised by determining in the reference image (14c) and in the first image (14a) the further points (28a, 28c) by transforming the further points (28c) of the reference image (14c) into the points (28a) of the first image (14a) using the first equation, determining image descriptors (29a, 29c) for the points (28a, 28c) transformed into each other, comparing the image descriptors (29a, 29c), and depending on the result of the comparison, considering the points (28a, 28c) transformed into each other as corresponding points (28a, 28c), and

determining in the reference image (14c) and in the second image (14b) the further points (28b, 28c) by transforming the further points (28c) of the reference image (14c) into the points (28b) of the second image (14b) using the second equation, determining image descriptors (29b, 29c) for the points (28b, 28c) transformed into each other, comparing the image descriptors (29b, 29c), and depending on the result of the comparison, considering the points (28b, 28c) transformed into each other as corresponding points (28a, 28c).

7. The method according to claim 5, characterised by determining in the reference image (14c) and in the first image (14a) the further points (28a, 28c) by creating image descriptors (29a, 29c) for the points (28a, 28c) in the vicinity of the found feature points (18a, 18c), which are then compared to each other, then calculating whether the corresponding further points (28a, 28c) satisfy the first equation, and

determining in the reference image (14c) and in the second image (14b) the further points (28b, 28c) by creating image descriptors (29b, 29c) for the points (28b, 28c) in the vicinity of the found feature points (18b, 18c), which are then compared to each other, then calculating whether the corresponding further points (28a, 28c) satisfy the second equation

8. The method according to any of claims 1 - 7, characterised by transforming the points of the image segments (15k) of the first image (14a) and of the second image (14b) outside of the common field of view of the two cameras (12a, 12b) with the first equation and the second equation, respectively.

9. The method according to claim 5, characterised by determining the further points (xo, x-i , X2) by

- transforming the given further point (xo) of the reference image (14c) into the first point (xoi) of the first image (14a) using the first equation, - transforming the given further point (xo) of the reference image (14c) into the first point (X02) of the second image (14b) using the second equation,

- transforming the first point (xoi ) of the first image (14a) into the second point (X12) of the second image (14b) using the zeroth equation,

- transforming the first point (X02) of the second image (14b) into the second point (X21 ) of the first image (14a) using the inverse of the zeroth equation,

- creating the further point (x-i ) of the first image (14a) from the first point (xoi ) and the second point (X21 ) of the first image (14a), and

- creating the further point (X2) of the second image (14b) from the first point (X02) and the second point (x-12) of the second image (14b).

10. The method according to any of claims 5 - 9, characterised by comparing the image segments determined to be replaced based on the application of the first and second equations respectively and the image segments determined fto be replaced based on the application of the zeroth equation, and deciding on the replacement of content on the basis of the comparison.

1 1 . The method according to any of claims 1 - 10, characterised by that the geometrical object (10) is a planar surface (Φ) and finding the equation transforming between the first and the second image in the form

where ΗΦ is the 3x3 homography matrix belonging to the linear transformation, λ is a scalar, and v and v' are the location vectors of the corresponding points of the geometrical object in the first and the second image.

12. The method according to any of claims 1 - 1 1 , characterised by performing the method on several stereoscopic image pairs forming a 3- dimensional moving image series, and searching for feature points in a given stereoscopic image pair in the vicinity of points with coordinates corresponding to the coordinates of the feature points (18a, 18b) determined in earlier stereoscopic image pairs of the moving image series.

13. The method according to any of claims 1 - 12, characterised by projecting a 2-dimensional image onto the surface of the geometrical object (10) as target content, which, after performing perspective corrections, is placed into the position of the image segments to be replaced in the two images (14a, 14b).