WO2017120651A1

WO2017120651A1 - Device for creating mosaics of reconstructed images and method for creating a mosaic of reconstructed images

Info

Publication number: WO2017120651A1
Application number: PCT/BR2016/050303
Authority: WO
Inventors: Graciliano MONTEIRO PASSOS
Original assignee: Up Points Serviços Empresariais S.A.
Priority date: 2016-01-13
Filing date: 2016-11-23
Publication date: 2017-07-20
Also published as: BR102016000694A2; BR102016000694A8

Abstract

The present invention relates to a device for creating a mosaic of reconstructed images, comprising an image capture means and a processing means. The image capture means (200) is programmed to continuously capture image frames of a display device (100) with articles (120) at different moments in time. The processing means analyses this series of images and creates a mosaic of reconstructed images that is able to reflect the actual state of the articles in the display device at a given time tn even if the image frame actually captured at the time tn includes obstructions in front of the articles.

Description

"DEVICE FOR CREATING A RECONSTRUCTED IMAGES MOSAIC AND METHOD FOR CREATING A RECONSTRUCTED IMAGES MOSAIC"

FIELD OF INVENTION

The present invention relates to a device and method for creating a reconstructed image mosaic and, more specifically, the present invention relates to a device and methods for creating a reconstructed image mosaic that can be be used in an inventory management system based on object recognition analysis.

BACKGROUND OF THE INVENTION

[002] Inventory control is one of the essential activities related to retail trade. Inefficient inventory management can lead to delays in stock replenishment orders, improper or confusing display of products at points of sale, or even complete stockouts.

[003] For retailers, inventory issues can lead to increased replacement costs (late / unnecessary orders and stock imbalances), loss of revenue due to lack of consumer desired product, or a sloppy store aspect with empty shelves. or no product diversity.

[004] For the supplier, the absence of their product at the point of sale may affect consumer behavior or loyalty - which may choose to buy another brand - and disrupt marketing efforts linked to the point of sale product display layout. . Similarly, lack of inventory control can lead to inefficient logistics planning based on erroneous sales / inventory information.

The most widespread inventory management systems usually involve manual labor, where an employee is assigned to periodically check products on display at points of sale and products in stock, generating data for the preparation of planograms and reports of control.

As known to those skilled in the art, planograms are graphical representations of articles (products) on gondolas or shelves. These representations are used for planning the products on display in the point of sale in order to manage product sales and replacements.

[007] This manual inventory control has been shown to be inefficient: besides being a slow and laborious procedure, it does not produce the expected results. For hand-generated reporting to come true, enough staff would have to be allocated to check the entire point of sale at very short intervals. Of course, this type of check procedure is not feasible.

In addition, even if this optimal check procedure were possible, the time elapsed between physical point-of-sale verification, processing of information gathered with inventory information, reporting, and analysis of reports would already be greater than ideal time for dynamic decision making.

In view of the drawbacks mentioned above, automated inventory management solutions are being developed, generally based on point-of-sale image recognition techniques.

[010] US2003 / 0154141, for example, shows an image recognition based inventory management system. In this system, video cameras are installed in the aisles of the point of sale to transmit data about the products on display. The cameras are linked with image recognition software that recognizes the missing or low quantity products and allows a planogram of the displayed products to be updated. Communication between the cameras and the processing computer is via a LAN or a WAN network. For image recognition to be possible, a central computer also includes a database of stored images of all point-of-sale products.

[01 1] Document PI 0816173-9 is a planogram extraction method based on image processing. For the implementation of the method, one or more image capture devices are installed in inventory environments, object recognition analysis is performed on the captured image, and a planogram is extracted based on the image recognition data.

[012] Although the above solutions are an evolution of control inventory manual, image recognition-based management methods have some drawbacks.

[013] A first of these drawbacks is related to analysis of the image captured at points of sale. For the recognition analysis to be performed, it is necessary that the image captured by the cameras at the point of sale allows the viewing of the articles on shelves or gondolas. Because supermarket aisles or other businesses are places of movement with constant traffic from consumers or employees, there is a high likelihood that captured images will have obstructions that prevent viewing of products.

[014] This inconvenience leads to the need for pre-selection of images to be used for recognition or the need for the captured image to be worked out for cleaning or correction of the captured image.

[015] In an attempt to solve this problem, the method of PI 0816173-9 mentions the possibility of detecting image obstructions during image analysis - several of the prior art obstruction detection techniques are cited. in the document - and discard or ignore the image or part of the image that is jammed.

[016] Another drawback present in object recognition based inventory management methods is related to the transmission of image data.

[017] Known methods involve the transmission of images captured at the point of sale to image analysis servers. It is generally desirable for image capture devices to be wireless and for images to be transmitted to the analysis server over wireless networks, such as LAN, WAN, or other Internet Protocol networks.

[018] As those skilled in the art know, object recognition analysis should be performed on high resolution images, which by definition are heavy images.

[019] Heavy image transmission puts pressure on network upload links. Thus, a limit is created on the number of images and frames that the upload link can support, which imposes restrictions on the number of images that will actually be object of the recognition analysis. Or that is, while the image capture device can capture images every second of the 24 hours of a day, a natural limit is imposed on the amount of images that can actually be sent for object recognition analysis.

[020] Another drawback of inventory management methods based on object recognition in images is the need to respect consumer privacy. In fact, some countries have laws that prevent establishments from obtaining consumer images for any purpose other than for security monitoring alone. Thus, legislation prevents a point-of-sale captured image showing a consumer in the aisle or near a gondola or shelf from being sent to an image analysis server. This kind of legal impediment renders inapplicable some of the solutions known in the art. OBJECTIVES OF THE INVENTION

It is an object of the present invention to provide a device and method of constructing an image mosaic for application in object recognition based inventory management systems that solves problems related to the state of the art.

It is another object of the present invention to provide a device and method of constructing an image mosaic that provides a file size image mosaic suitable for internet traffic.

It is yet another object of the present invention to provide a device and method of constructing an image mosaic that enables the construction of an image mosaic showing the articles being displayed at a point of sale without obstruction of the articles.

It is a further object of the present invention to provide a device and method of constructing an image mosaic that enables the construction of an image mosaic that excludes any persons who might be in range of the image capture devices at the moment. of image capture.

BRIEF DESCRIPTION OF THE INVENTION

[025] The present invention achieves the above objectives by means of a device for creating a mosaic of reconstructed images comprising a image capture medium and a processing medium.

[026] The image capture medium is programmed to continuously capture image frames from a display device with articles at different times of time. Thus, for a series of captured image frames, tn represents a reference time moment, tn- is the past time, and tn + is the future time.

[027] The processing medium analyzes this series of images to identify, for each frame at time tn, background blocks that have changed relative to the other background blocks in the series of images at time tn- and tn +. Background blocks as used herein are blocks that show only the background elements of the picture frame, that is, the display device and the articles packed there.

The mosaic making device of the present invention utilizes such background image blocks to construct, for each time tn, a reconstructed image mosaic. Thus, for reference time tn, this image mosaic includes:

[029] the reproduction of blocks of background elements that showed changes in the series of frames captured at times tn-; and

[030] the inclusion of background blocks that have shown changes in time tn, provided that such background blocks do not change relative to the corresponding background blocks in frames captured at time tn +.

Thus, each reconstructed image mosaic created can reflect the actual state of the articles on the time display tn, even though the effectively captured time frame tn includes obstructions in front of the articles (such obstructions could include, for example, people or objects standing or moving in front of the display device).

[032] The device of the present invention also provides for the creation of metadata associated with each background block that changes relative to the other background blocks in the image series at times tn- and tn +. Such metadata may include, for example, the indication of background change, the presence of a standing shape in front of the display device, or the presence of a moving shape on the device. display

[033] The device of the present invention may also comprise a means for compressing the created reconstructed image mosaics, for example, allowing such mosaics to reach a size suitable for transmission via the internet or wireless networks. Compression also allows the created tiles to take up less storage space and to process them with smaller CPUs.

The present invention also provides for the creation of feature maps associated with the generated reconstructed image mosaics. In an object recognition-based inventory management system, object recognition analysis will be performed on this feature map. It should be noted that the metadata created by the device and method of the present invention can further optimize object recognition analysis, as the metadata can indicate exactly where changes are for each tn buffer, allowing analysis to be performed only for those objects. blocks where there was change.

The further advantages of the device and method of the present invention will be apparent from the detailed description of the exemplary embodiment shown in the figures.

BRIEF DESCRIPTION OF DRAWINGS

[036] The present invention will be described in more detail below with reference to the accompanying drawings in which:

Figure 1 is a schematic illustration of an image recognition based inventory management system utilizing the device according to a preferred embodiment of the present invention;

[038] Figure 2 is a schematic illustration of the logic underlying the image mosaic creation method according to a preferred embodiment of the present invention;

Figure 3 is an example of an image mosaic constructed with the device of the present invention; and

Figure 4 is a schematic illustration of an application of the image mosaic maker in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION [041] The present invention will be described hereinafter based on an example of preferred embodiment.

[1] Figure 1 is a schematic illustration of an inventory management system using the image tessellation device of the present invention.

[043] The figure schematically illustrates an article display device 100 at a point of sale, with a consumer 110 near display device 100. Items or products 120 are disposed on display device 100.

It should be noted that although a shelf 100 is illustrated, the present invention could be used with any kind of packaging or display means for articles or products, such as gondolas, refrigerators, carton displays and the like.

Furthermore, while the present invention is exemplarily described herein in relation to display of sales products, it should be noted that the device and method of the present invention may be applied to any inventory management system where articles are packaged in devices. such as industrial inventory inventories or spare parts inventories.

[046] Turning to Figure 1, one or more mosaic making devices 200 are arranged at the point of sale. Although the figure schematically illustrates only one device 200, it should be understood that a single point of sale environment may include a plurality of mosaic making devices 200.

[047] In addition, it should be understood that the mosaic making device 200 may be statically fixed to a point in the environment or may be on fixed or movable support means.

[048] In the preferred embodiment of the present invention, the moisaic circling device 200 comprises a sequential image capture means, a processing unit, a processing RAM memory, mosaic transmission means and a storage medium which stores the operating system and processing software.

[049] The image capture medium can be, for example, a USB camera, a CMOS (plementary metal oxide semiconductor) or CCD (charged coupled device) sensor with a control coupling module, or any other suitable means of sequential imaging.

[050] The processing unit may be a low-power CPU, such as an ARMv7 architecture CPU. Of course, similar low power processing units could also be used.

Mosaic transmission media may comprise for example, ethernet card, cable, or Wi-Fi module, or wireless media.

The storage device may be, for example, an SSD card or similar storage devices or media.

[053] The mosaic maker 200, the operation of which will be described in detail later, continuously captures real-time image frames from shelf 100 and creates a mosaic of reconstructed images - schematically illustrated with reference numeral 210 - of articles 120 on shelf 100, eliminating any obstructions in front of shelf 100, such as person 110. That is, the image mosaic created is a mosaic illustrating an actual representation of the "background" of the captured image frames.

[054] The tile maker 200 creates a tile or a compressed tile block. This compressed data has a file size suitable for wireless networking. To have an understanding of the efficiency of the technique, 24h capture takes up less than 200MB, and without any compression only 50 frames would take up over 200MB.

[055] The reconstructed background mosaic 210 is sent to a processing system that includes a feature map generator or feature maps 300 for creating a feature map 310.

As known to those skilled in the art, feature maps are maps that extract visual features from an image, such as color, shape and texture, borders, lines, and index such features. Feature maps are common in content-based image retrieval systems (SRIBCs) or image analysis systems for understanding the environment (3D map extraction, motion detection, change detection). [057] The created feature maps are then sent to a feature map server 320 and are accessed for performing object recognition analyzes that will enable inventory and inventory breakdown analyzes.

[058] As shown in Figure 1, the inventory management system further has inventory analysis 400 and stretch break analysis 500 servers. The system further comprises a database 600. Analysis servers 400, 500 and The 600 database can be accessed by 700 users.

Figure 2 is a schematic flow diagram of the operation of the mosaic creation method of the present invention.

[060] Mosaic device 200 captures, in real time, a series of image frames at different time points from shelf 100 with articles 120. The series of image frames over time is illustrated in Figure 2. as t1, t2, t3, t4, t5 and t6, where tn is the capture time. Note that tn can continue indefinitely as picture frames are captured continuously.

Thus, in one embodiment of the present invention, the device captures sequences of 4 frames per second, with T0 = 0s, T1 = 250ms, T2 = 500ms, T3 = 750ms, T4 = 1s, T5 = 1s250ms, T6 = 1, 5s, etc. Thus, the device of the present invention is capable of capturing a large number of frames in a time interval, which allows the mosaic in a given Tn to be created from the temporal analysis of a large number of frames.

[062] The number of frames analyzed for creating a mosaic for a captured series will depend on the processing capacity available on the mosaic maker 200.

[063] As can be seen from the illustration, the example sequence of frames can be defined as follows:

[064] t1: initial captured image frame of the example, where the present article a1 is noted;

[065] t2: time-captured picture frame t2, where article a1 present at t1 is not present on the shelf;

[066] t3: Image frame captured at time t3, where article a1 is still missing; [067] t4: picture frame captured at time t4, where article a1 is still missing but part of the background is obstructed by the presence of a person;

[068] t5: time frame captured at time t5, where part of the background is obstructed by the presence of a person and where it is not possible to know whether article a1 is present or absent; and

[069] t6: picture frame captured at time t6, where it can be seen that article a1 is still missing and article a2, which was present at t4 and obstructed at t5, is missing.

In the mosaic creation method of the present invention, an algorithm is used to do a pixel block analysis of the image sequence. In the embodiment of the present invention, the images are worked in YUV format.

[071] The algorithm used can identify changes in the background and the permanence or not of those changes during that series analyzed. Thus, the algorithm can realize that: there was a change in the bottom of the image (shelf + articles) between t1 and t2; whereas although there is an obstruction by one person at t4, there was no change in the background of the image between t4 and t3; and although it has an obstruction by one person at t5, there was a change in the background between t4 and t5.

[072] This perception in the absence of background change in t4 is only possible because the algorithm does not just do a simple or piecemeal analysis between tn and tn-1 images, the algorithm analyzes the entire frame sequence to realize that changes deep down they were permanent.

Thus, the processing medium of the device of the present invention is capable of analyzing a frame series of images captured at different time points tn-, tn, tn +, where tn is a reference time moment, tn- moments of time in the past and tn + are moments of time in the future.

[074] For each frame at time tn, the background blocks that have changed relative to the other background blocks in the series of images at times tn- and tn + are identified.

[075] The image mosaic is constructed from the reproduction of blocks of background elements that present changes in the series of frames captured at times tn-; and the inclusion of blocks of background elements that presented changes in time tn, provided that such background blocks do not change relative to the background image blocks in frames captured at time tn +.

[076] Figure 3 shows an image mosaic 210 created by the mosaic maker of the present invention for time tn = t5. The mosaic was reconstructed using the block of changes identified in the image captured at time t2 and the block of changes identified in time t5 (note that this block of background elements shows no changes from t6).

[077] In this regard, it should be noted that in this reconstructed background mosaic, the person causing obstruction is not present. That is, the mosaic is not the image frame captured at t5, but rather a representation of the actual condition of articles 120 on shelf 100 at time t5.

[078] This perception of time t5 is only possible because the analysis algorithm considered the sequence that goes beyond t5 and was able to perceive that a person blocking the bottom at t5 removed article a2 from the shelf.

Thus, mosaic 210 created for time t5 contains background blocks that can be generated from image frames captured at different times. In this way, each tile block contains unobstructed, non-moving background elements. Therefore, the mosaic created for time t5 is not an image at time t5 but a reconstruction of the background elements from blocks originating from different frames captured at different times.

[080] The tile maker also allows metadata to be created with information about the blocks used for tile construction. Thus, metadata may contain information about changes in background elements and about movements and shapes in front of that background (eg, people passing, obstructions by objects, etc.).

As illustrated in Figure 2, the method of the present invention allows the creation of reconstructed background mosaic blocks M2, M4, M5, M6 with metadata of information on background changes and presence and movement of shapes in front of the background.

[082] Note that these blocks do not include images at times t2, t4, t5 and t6, but mosaics that reconstruct static background elements at times t2, t4, t5. and t6.

[083] As can be seen from the flowchart of Figure 4, these reconstructed background blocks and metadata are compressed and stored.

[084] Compression is optimized by considering the tile blocks where changes have actually occurred. This allows a reduction in processing capacity, which allows for a reduction in costs and time.

[085] Mosaic compression is preferably performed by an image compression codec that works with YUV pixels. This feature allows a saving in the processing power required by the mosaic maker 20, since the previous step already works with data. in YUV format, without requiring large volume conversions.

[086] Block mosaics where significant changes have occurred are used for feature map extraction, as discussed in relation to Figure 1.

[087] Object recognition analysis is performed on feature maps, considering metadata with information about changes in areas.

Thus, the device and tiling method of the present invention provides, in addition to not only optimizing the file size for data transmission and reducing the processing power required, faster object recognition analysis. it is efficient.

Having described an example of a preferred embodiment of the present invention, it should be understood that the scope of the present invention encompasses other possible variations of the described inventive concept, being limited only by the content of the appended claims, including the possible equivalents thereof.

Claims

1 . Device for creating a mosaic of reconstructed images, comprising an image capture means for continuously capturing image frames of a display device (100) with a plurality of articles (120) and a processing medium that performs a algorithm for:

analyze a frame series of images captured at different time points tn-, tn, tn +, where tn is a reference time moment, tn- are past time moments and tn + are future time moments;

identify, for each frame at time tn, background blocks that have changed relative to the other background blocks in the series of images at times tn- and tn +,

build, for each time tn, a mosaic of reconstructed images with:

- reproducing blocks of background elements that show changes in the series of frames captured at times tn-; and

the inclusion of background blocks that have shown changes in time tn, provided that such background blocks do not change relative to the background blocks in frames captured at times tn +.

Device according to Claim 1, characterized in that the processing means creates for each background block that changes relative to the other background blocks in the image series at times tn- and tn +, a metadata associated with identified change information.

Device according to claim 2, characterized in that the processing means further comprises a means for compressing the created reconstructed image tiles.

Device according to any one of claims 1 to 3, characterized in that the processing medium creates a feature map corresponding to each reconstructed image mosaic created.

Device according to any one of claims 1 to 4, characterized in that the mosaic making device (200) comprises a sequential image capture means, a processing unit, a storage medium, a processing RAM and means for transmitting the tiles.

6. Method for creating a mosaic of reconstructed images, comprising:

continuously capturing image frames from a display device (100) with a plurality of articles (120);

build, for each time tn, a mosaic of reconstructed images with:

Method according to claim 6, characterized in that it further comprises creating a metadata associated with each background block that changes relative to the other background blocks in the image series at times tn- and tn + , where metadata includes information about the identified change.

Method according to claim 7, characterized in that it further comprises the step of compressing the created reconstructed image mosaics.

Method according to any one of claims 6, characterized in that it further comprises:

create a feature map corresponding to each reconstructed image mosaic created.