WO2007122221A1

WO2007122221A1 - Automatic system for processing inspection data

Info

Publication number: WO2007122221A1
Application number: PCT/EP2007/053963
Authority: WO
Inventors: Björn LINDSTRÖM; Mikael Sandberg
Original assignee: Abb Ab
Priority date: 2006-04-24
Filing date: 2007-04-23
Publication date: 2007-11-01
Also published as: SE533637C2; TW200813892A; SE0802132L

Abstract

A method for processing inspection data for a freight container comprising making an image of the container, compressing a digital file containing said image, and storing the image data in a database. In particular the image data is processed to reduce the image size before compression. The image data is also processed to resolve the parts of the image into lines or graphic elements for automatic analysis of a container view to determine visible damage. The container and the damage information may be displayed in 2D or as a 3D image. A system, a graphic user interface and one or more computer programs are also described.

Description

Automatic system for processing inspection data

TECHNICAL FIELD.

The invention relates to a system for processing inspection data for freight containers. The invention concerns a system and a method for processing images of a container which are preferably obtained while the container is being moved by a crane. In particular it is a system and a method to process, store, retrieve and represent inspection data used to record a container condition including any damage to the container.

BACKGROUND ART

A great and growing volume of freight is shipped around the world in standard shipping containers. Transshipment has become a critical function in freight handling. At each point of transfer from one transport means to another, from ship to shore in ports and harbours for example. There is usually a tremendous number of containers that must be unloaded, transferred to a temporary stack, and later loaded on to another ship, back onto the same ship or loaded instead onto another form of transport. Loading and unloading containers to and from a ship takes a great deal of time. The development of automated cranes has improved loading and unloading and made the productivity more predictable, and also eliminated many situations in which port workers have been exposed to danger and injury.

Container handling in ports sometimes results in damage to the outside of the container. Containers may also de damaged in transit. In order to handle shipping containers efficiently it is necessary to record in some way any damage the containers might have suffered. US 2003/0167214, entitled Scheduling method for loading and unloading containers at the terminal; describes a method in which Container ID numbers are tracked during loading on to a ship and unloading off the ship in loading and also used for organising a container storage system. US6,768,931 entitled: Method and apparatus for quay container crane-based automated optical container code recognition with positional identification; assigned to Paceco Corp., describes an optical characteristic system providing container code recognition from a quay container crane of a container identified by a container code to a container inventory management system. It is described in the specification that the system may be used for tracking of all containers with respect to their history, damage, current location, and that the system provides at least one video image for a container, which is compressed.

It is difficult to maintain a record of visible damage, especially considering the large number of containers handled and the volume handled by each port. It is a technical challenge to store images of containers efficiently without using extensive storage systems, and to retrieve information from such systems in a short time. It is also difficult to estimate extent of damage to a container from a still image or other photograph. Video information may be used to document visual appearance, however, storage of video clips normally requires great amounts of system resources for storage and also for retrieval, and such a system would be expensive to use and maintain.

SUMMARY OF THE INVENTION The aim of the present invention is to provide an improved method and system for archiving image data for containers. This and other aims are obtained by a method, and a system characterised by the attached independent claims. Advantageous embodiments are described in sub-claims to the above independent claims .

Another object of the present invention is to provide an improved computer program product and a computer readable medium having a program recorded thereon, for controlling a load control device of a crane. In addition, further and advantageous aspects of the invention are described in relation to an independent claim for a graphical user interface.

The invention comprises a method and system for automatic processing of graphic images of the containers, recorded while the container is being moved by a container crane. Photographed graphic images are processed and stored, normally 5 images per container: Top, Left, Right, front and back. Each group of views, the image data, is assigned an internal database identifier code. Each image of each of the container views is processed to remove unnecessary visual information, before file compression. Each image is then pre-processed for later use in a 3-D rendered image of the container. The images are then ready for storage in the archive system once an ID for the container has been established.

An ID for the container is input into the automatic image archive system. The Container ID is obtained from a record or from a visual analysis system of some sort, such as a tag reader, bar code reader, a laser scanner, an Optical Character Recognition (OCR) process or similar.

The Container ID number input is logically analysed in the automatic image archive system to test if it is of a valid form in terms of number of characters, whether letters or numbers and so on. If the ID number is valid, then a search is made to determine if the container is known to the system.

In cases where the external bar-code or container number is found to be not machine-readable, or not to be a valid number, the image is automatically processed and stored in the archive with an internal identifier only, and the container designated for a manual inspection process .

For valid ID numbers a search is made by the archive system to find if the container is known to the system. Information about each container already handled in a container yard or port is comprised in some way as Property information for each container. Property information is principally text information containing such types of information as container type, container size, container ID number, owner, number of doors, date of last event, etc.

The data for the container information is stored in a predetermined and structured way. The text information is stored in a first database or arrangement of databases. The image data per container is stored in a structured way distributed between a number of second databases. The container information stored in the text database contains a pointer or similar to the address or addresses of the images for the container. The archive holds the information about each container for at least five years, and may thus contain more than one record of handling the same known container. The system is arranged capable of retrieving previous container image data and comparing a picture of the Left side, Right side, back, or front etc of the container to an earlier or a later image.

The processed group of views, image data, for a known container is stored in the archive. When retrieved, the images may be examined as 2-D images. The image data per container may also be provided as a 3-D rendered image of the container. The image may be rotated and manipulated on screen by an operator. The extent and depth of any damage on any one side may be calculated. The extent and depth of any damage extending over more than one side can also be determined and calculated by any of an automatic, semi-automatic or a manually initiated process.

The main advantage is that container damage information is extracted from inspection procedures . Container damage information is compared to previous container damage information so that it is established before or after which dates and events a particular damage to the container was sustained. The damage information is processed, calculated and stored automatically. Information from manual inspections may be input and records updated where necessary. The archive provides a reliable and comprehensive record of date and extent of damage to each or any container handled. Reports may be produced and graphic information in 2-D or 3-D examined and inspected on-line or off- line as necessary.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the method and system of the present invention may be had by reference to the following detailed description when taken in conjunction with the accompanying drawings wherein:

Figure 1 shows in a schematic block diagram a simplified arrangement for a system for processing inspection data for a freight container according to an embodiment of the invention; Figure 2 shows a flowchart diagram of a method for processing inspection data for a freight container according to another aspect of the invention;

Figure 3 shows a flowchart diagram of a method for processing inspection data for a freight container according to another embodiment of the invention;

Figures 4a, 4b show a schematic block diagram of a graphic user interface for displaying and/or handling 2-D displays of container image data according to another aspect of the invention; Figures 5a, 5b show a schematic block diagram of a graphic user interface for displaying and/or handling 3-D displays of container image data according to another aspect of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Figure 1 shows a simplified schematic block diagram of a system for processing inspection data for a freight container. The system diagram is schematically arranged in two parts. A first part C that is based on or around a crane, such as a ship-to- shore (STS) crane 1 arranged on a quayside for loading or unloading containers from a ship, or similar container crane arranged in a container yard. A second part L is arranged some distance from the crane, on the land side, and may be relatively local in the container yard or located remotely.

The figure shows in the crane part C a camera 5 arranged on the crane. The images taken by the crane camera 5 are sent to a image preparation unit 8, together with an internal database object identifier number. The images per container normally comprise five views, these being images of 4 sides and the top of each container. A container identification number, Container ID, is also supplied to the image preparation unit 8 as a string of numbers and letters .

After the image containing the 5 views has been prepared it is sent to a dispatcher unit 7 which normally stores the data in a short term storage device such as Container Information buffer 6 in the vicinity of the crane for up to several days. This is a built-in back-up precaution in case of temporary loss of data transmission between the crane at the quayside and the archive located elsewhere on land. The prepared image is also sent to the second, non-local part L of the system where it is received by a service broker 11. Service broker 11 in the second non-local part L of the system determines what service (s) are required by the data transmission received from the crane side, for the most part services required by the prepared image data included in data transmissions .

The non-local part L also comprises two databases, a first database arrangement 12 for container images and a second database arrangement 13 for text information about each container for which there is image data. The database address of the images for each container is stored with the text information for each container in the second database 13. This address to where the images are stored may be any kind of link or direct or indirect address or similar, and is preferably a pointer. Text information stored per container may comprise information such as: owner, type and number of doors, type of corner cone, cross bars if any and so on. Store handling information may comprise: date last handled, address or pointer to earlier image information, information about any prior events. Records may be held for 5 years or more. A Report service function 16 may be used to select, retrieve information prepare reports containing visual and or text information about one or more containers from the databases 12, 13.

The service broker 11 may send data to the System communicator 16. System communicator 15 may send certain data to the Event handler 18. Event handler 18 may send data to the Routing manager 17, which function determines where the data shall be sent under different circumstances. For example, if a container has a broken seal, and must have and internal inspection by authorised personnel, such as customs personnel for example, the routing manager sends the information to the network addresses, or telephone, SMS or fax numbers for the appropriate authorised persons recorded in the system. The second system part L also comprises other functions such as Database maintenance 50, a Configuration manager 51, an Authentication function 52 and User manager 53. There is also a system log 54. Database maintenance includes handling processes to ensure that records are kept for a predetermined number of years and destroyed when no longer needed. Configuration manager configures functional units for the whole system and facilitates addition or substitution of new hardware and so on. The Authentication function includes configurations for which operations which user may carry out which operations. User manager may use an active directory or other means to maintain contact means for each user on duty or on call. The system log is a detailed, high-resolution log comprising information from, for example, commissioning and maintenance as well as operations, event and alarm information.

Figure 2 shows a flowchart for a method for processing inspection data for a freight container according to an embodiment of the invention. In block 20 the camera (crane camera 5, Fig. 1) sends image data for a container to an image preparation function, Process image data block 22. The system assigns an internal object number to the image data. Image data is processed in block 22 and each view of the container is identified as a front side, or back side, top side, left (short) side, right (short) side. Each view is processed and non- container image data with background images etc is removed or cropped from the view. The image data is then digitally compressed.

A string including a container ID number is received at 22 from an external identifier unit. This unit or information source may be any active process, such as a bar code reader, tag reader, CCD camera image processer, Optical Character Recognition (OCR) reader and so on. The Container ID string is processed at 25 to determine that it includes a valid number and/or combination of letters, numbers, spaces, hyphens etc. In block 29 a validated Container ID is checked to see if there is any record for this container in the Container Information Archive 2 or Container image Archive 1. The string may alternatively be downloaded from a data source held by a ship or shipping company, freight forwarding company, and so on.

If no record, N, the Container ID is stored as a new Container ID in block 30 and sent 34 to the Container Information Archive 2 for storage. If already known, Y, the existing container information is updated block 33 in the Container Information Archive 2 and the updated record stored so that the databases accumulate information over time about a series of handling operations for each container.

If the Container ID number is not of a valid form, indicating that numbers or letters are missing or obliterated or otherwise not distinguishable, then the container, identified by the internal database object number, is placed in a Work Order queue for a manual inspection. The image data is stored in image database 12 associated with the internal database number stored temporarily in the information database 13. After a manual inspection 27, the Container ID is input manually to the system, and checked in block 29 to see if it is already known. If yes, existing information updated 31, if No, then a record for a new container is created 34 as before.

Block 37 Damage inspection provides an answer in block 39 that the container is either damaged Y, and a damage report is generated, or N, not damaged. In both cases, the damage information in the Container information database archive 2 may^¬ be updated.

Block 37 is preferably carried out automatically. Images of the container top or side views from the latest handling are processed, such as rasterised, and each view analysed by a computerised process to determine if there are areas of the container that are significantly different in colour or colour intensity. The computerised processes determine from differently coloured lines or areas of the container that there is a hole or indentation in the side of the container. This may be automatically compared to older images of the same part of the same container. The process then determines if the damage new or old, and the dates between which the damage must have occurred. The images from previous handlings may be combined or superimposed or animated or processed in another way to show the progress and extent of the damage during successive time intervals .

A damage inspection may be signaled or even carried out by an operator examining database information and comparing older views of the same container to present views . The operator may issue a report based on visual and/or numerical and date information provided by the system or he may request a manual inspection if, for example, the damage is extensive, goods may be falling out or leaking out of the container, of if a door seal appears to be broken or lost.

Figure 3 shows a flowchart describing the steps that a computer program may execute in order to make a computer or processor carry out a method for processing inspection data for a freight container according to another embodiment of the invention. In block 22, Process image data, new images from the Container crane are processed. In block 22.3 the images are prepared by removing non-container image information or cropping. In block 22.5 the images are prepared for damage analysis. The images are processed for example by rasterisation, to determine and record parameter values such as colour, hue, intensity, shadow, for small areas of the view, grid elements, or even for individual pixels of the image. From these measurements a computerised process determines any differences in colour, hue, lighting etc that indicate damage or possible damage. One or more edges may be identified in a part of an image and contours may be mapped. Calculations are made of extent and/or depth of damage. The rasterized images of the 2D views of the container sides, top and ends are also prepared suitably so that the views can later be assembled into a 3D model of the container.

In block 22.7 an analysis is carried out of the image on the basis of the values for colour and light etc determined for the graphical image of each container side view. Each side of the container is then compared with a similar view of the same container, photographed during an earlier handling event. Computerised processes compare the new image data with older image date for each view. Changes in container condition, new damage, increased extent of damage and so on are estimated and reported automatically. An event or alarm signal may be generated if the damage is interpreted as being greater in some way than a predetermined norm. For example, a manual inspection request may be generated.

In block 23 a damage report 41' may be generated, and/or an analysis report 41'' may be generated. Images and/or analysis may be output to a computer display 37'' or output as a printout or in some other form. When container information is retrieved it is obtained very rapidly. The required Container ID, a number, is found in the Container Information (text) database (12 very quickly. The property information is read, and retrieved very rapidly because the total amount of data to be searched in the Container information database 12 is relatively small. The pointer or address stored with the text to reach the image data for the container is extracted from the property information. The image address in the image database is accessed and the data read and retrieved. Retrieval of the image data is very rapid because the address where the image data is already known, and the relatively huge amount of data in the image database 11 does not have to be searched.

A graphical user interface (GUI) may be used to display one or more of the values obtained using the system and methods described above. Figure 4a shows a GUI 60a which displays side views of a container. A pointing device or input device P may be used by an operator or manager etc to examine of manipulate 2D images 2D2 of the right side R of the container or other sides or the top T of the container. Figure 4b illustrates a simple comparison of new 2-D image data 2D2 with old image data 2Dl. An old damage dl is found on the old image and a new, larger damage area d23 is found on the new image. This comparison of image data for the same from different dates allows a computer process or a human operator to determine, for example, if new damage has occurred or not.

Figure 5a shows a 3D image 3D2 of a container on a GUI 62a. The image may be rotated about a number axes A, A' and/or re-sized, zoom in, zoom out etc or otherwise manipulated by an input device P. The 3D image is produced as a model with the processed 2D of the container superimposed on a 3-D form. This may be done using a computer graphics modeling technique such as applying rendered 2-D views, for example, to a wire-frame model. The end result is a 3-D image of the container with any visible damage shown in 3-D. Displaying damaged areas that extend over more than one side or edge of the container allows an estimate of the depth of bending or buckling to be calculated. This information has previously not be obtainable with traditional video filming of flat 2-D container images. Similarly, early damage 3dl may be compared on the 3-D image to later damage 3d2. The later image 3D2 may be analysed to calculate the depth of a dent or hole extending over more than one side of the container.

In an advantageous development, words and/or one or more graphical and/or numerical values for damage extent, damage position, damage depth, date of event, or other text information about a known container, may be superimposed or otherwise displayed on an image or part image in 2-D or 3-D of the container.

As mentioned the processing of the container image comprising the container inspection data may be carried out automatically by one or more or the above described computerised processes without any need for supervision by or actions from an operator. For the cases where a Container ID is not valid, or when a Manual Inspection is required work orders for manual intervention may be issued. At any time an operator or other authorised person may access the system to display, view, inspect or analyse live data on-line or off line as required.

In another embodiment of the invention, the image data may be stored distributed over a plurality of databases. One or more schemes such as "data-striping" may be used to store image data distributed across several databases so as to minimise retrieval time. It is advantageous keep the retrieval time low despite the amount of storage space required to store the entire database and maintain image data for a period of five years and more. It should be noted that while the above describes exemplifying embodiments of the invention, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention as defined in the appended claims .

Claims

1. A method for processing inspection data for a freight container, comprising making an image of a container and compressing a digital file containing said image, and storing the image data, in a database, characterised in that the method further comprises receiving an image of said container, processing two or more views (L, R, T, B, F) of said container comprised in said image data and reduce the file size of said image data.

2. A method according to claim 1, comprising the further steps of storing the compressed and processed said image data in a first database (11) distributed over a plurality of database storage structures and/or plurality of first databases.

3. A method according to claim 1, comprising the further steps of recording a storage address or pointer to said image data and storing it with text information about said container in a second database (12) and/or database storage structure.

4. A method according to claim 2 or 3 , comprising the further steps of storing a copy of the compressed and processed said image data in a local buffer database (6) for a period of time of up to a plurality of days.

5. A method according to claim 1, comprising the additional step preceding digital file compression of cropping or otherwise processing the two or more views of said container so as to exclude any visual data external to said container from said image .

6. A method according to claim 1, comprising the further step of dividing up said image data into a plurality of graphic elements for determining graphical parameters of each graphic element .

7. A method according to claim 6, comprising the further step of rasterising said image data into line or grid elements for determining graphical parameters of each graphic element.

8. A method according to claim 1 or 6, comprising the further steps of processing said image data and determining points, lines or areas on a view of said container which may comprise visible damage.

9. A method according to claim 8, comprising the further steps of processing said image data (2D2, 3D2) comparing said image data with previously obtained image data (2Dl, 3Dl) for the same container and determining from any differences between previous image data and present said image data the extent of changes in damage or extent of damage.

10. A method according to claim 9, comprising the further steps of processing said image data and calculating numerical values for damage and for extent of damage in area and/or depth for a 2-D view or a 3-D image.

11. A method according to claims 8-10, comprising the further step of storing the determinations of damage together with the container text information for said container stored in the Container information database (12) .

12. A method according to claim 1, comprising the further steps of processing said image data and arranging a plurality of the views for incorporation in a 3-D image of said container.

13. A method according to claim 1 or 12, comprising the further step of processing or rendering said image data for each view of said container suitable for use in a 3-D image of said container .

14. A method according to claim 1, comprising the further step of retrieving image data and text data for said container from the first and second databases and generating (41, 41', 41'') a report .

15. A method according to claim 1, comprising the further steps of retrieving image data and text data for said container from the first and second databases and providing a 2-D representation and/or 3-D model for said container comprising information about damage comprised in or superimposed on said or numerical information.

16. A method according to claim 1, comprising the further step of analysing a Container ID number, and determining it to be of a valid form.

17. A method according to claim 1, comprising the further steps of analysing a Container ID number, determining it to be of invalid form and creating an order for a manual inspection to determine a valid Container ID.

18. A method according to claim 17, comprising the further step of updating the stored property information data for a container with an updated Container ID number in the Container information text database (12) .

19. A method according to claim 8 or 10, comprising the further steps of processing said image data, determining an absolute or relative numerical value for damage in excess of a predetermined value, and creating a request for a Manual Inspection report.

20. A method according to claim 19, comprising the further step of sending a notice concerning the damage determined for said container to any designated person from the group of: operator, safety inspector, engineer, Customs representative, freight company representative, insurance company representative.

21. A system for processing inspection data for a freight container, said system comprising a database containing an image of a container and a compressed digital file of said image, and one or more computers, characterised in that the system further comprises means (5) for receiving an image of said container, means (8) for processing two or more views (L, R, T, B, F) of said container comprised in said image data reducing the file size of said image data.

22. A system according to claim 21, wherein the image database is arranged as a first database (11) for storing the compressed and processed said image data distributed over a plurality of database storage structures and/or plurality of first databases.

23. A system according to claim 21, wherein said system comprises a second database (12) for storing property information about said container as text information and for storing a storage address or pointer to said image data for said container stored in the first database.

24. A system according to claim 21, wherein said system comprises third database (6) arranged local to the crane for storing a copy of the compressed and processed said image data for a period of time.

25. A system according to claim 21, wherein said system comprises graphic processing means for cropping or otherwise processing the two or more views of said container to exclude any visual data external to said container from said image.

26. A system according to claim 21, wherein said system comprises means for dividing up said image data into a plurality of graphic elements for determining graphical parameters of each graphic element .

27. A system according to claim 21, wherein said system comprises means for rasterising said image data into lines or grid elements for determining graphical parameters of each graphic element.

28. A system according to claim 21, wherein said system comprises means for processing said image data and determining visible points, lines or areas on a view of said container which may be indicate visible damage.

29. A system according to claim 21, wherein said system comprises means for processing said image data and comparing said image data with previously obtained image data for the same container and determining from any differences between previous image data and present said image data the extent of changes in damage or extent of damage.

30. A system according to claim 21, wherein said system comprises hardware and/or software means for processing said image data and calculating numerical values for damage and for extent of damage in area or 2-D or depth or 3-D.

31. A system according to claim 21, wherein said system comprises means for processing said image data and arranging a plurality of the views for incorporation in a 3-D image (3D2) of said container.

32. A system according to claim 21, wherein said system comprises means for processing or rendering said image data for each view of said container suitable for use in a 3-D image (3D2) of said container.

33. A system according to claim 21, wherein said system comprises report means (16) for retrieving image data and text data for said container from the first and second databases and generating a report.

34. A system according to claim 33, wherein the report means for retrieving image data and text data for said container from the first and second databases comprises means for providing a 2-D representation (60a, 60b) and/or 3-D model (62a, 62b) for said container comprising information about damage included in or superimposed on said representations in the form of text, graphic or numerical information.

35. A computer program which when read into a computer or processor will cause the computer or processor to carry out a method according to the steps of any of claims 1-20.

36. A computer readable medium comprising a computer program which when read into a computer or processor will cause the computer or processor to carry out a method according to the steps of any of claims 1-20.

37. A graphical user interface for a system for processing inspection data for a freight container, said system comprising a database containing an image of a container and a compressed digital file of said image, and one or more computers, characterised by a display (60, 62) arranged with means to present and manipulate (P) one or more representations of one or more views of the container comprised in said image data.

38. A graphical user interface to claim 37, characterised by a means to display a part of said image data (2D2, 3D2) together with a part of a previous image data (2Dl, 3Dl) for the container.

39. A graphical user interface to claim 37, characterised by a means to display or manipulate changes or differences between a part of said image data (2D2, 3D2) relative a part of a previous image data (2Dl, 3Dl) for the container.

40. A graphical user interface to claim 37, characterised by means to display information relevant to events or damage included or superimposed on the display of a part of said image data (2D2, 3D2) and/or a part of a previous image data (2Dl, 3Dl) for the container .

41. Use of a system for processing inspection data for a freight container for determining and recording damage to the freight container according to any of claims 21-34.

42. Use of a graphical user interface for a system for processing inspection data for a freight container for investigating, examining and or measuring damage to the freight container according to any of claims 37-40.