WO2008016309A1

WO2008016309A1 - Multi-modal machine-vision quality inspection of food products

Info

Publication number: WO2008016309A1
Application number: PCT/NO2007/000278
Authority: WO
Inventors: Stig Jansson; John Reidar Mathiassen
Original assignee: Sinvent As
Priority date: 2006-08-04
Filing date: 2007-08-03
Publication date: 2008-02-07

Abstract

The invention to automatically recognize shape and surface defects of food products using computer image analysis is characterized by optical data being sampled by at least an optical sensor using combinations of different illumination and sampling modes. This is done to enhance detectability of surface differences, subsurface differences, or shape differences between a defective sample and a non-defective model of the food products. It uses a digital camera or an imaging spectrometer as the optical sensor.

Description

Multi-Modal Machine- Vision Quality Inspection of Food Products

Technical Field

The present invention relates to the field of machine vision. More specifi- cally it relates to machine-vision based processing of food products.

Background and Prior Art

For some years, there have been both an increased demand for, and value attributed to higher quality of mackerel and herring. Typical demands include more precise weight class distribution and less damaged fish. This creates incentives to find more accurate methods for weight and quality grading. The common way of performing weight grading today is to make use of V-belts or rotating rollers that separate the fish into 4-5 weight classes. The accuracy of these methods depends on the body index as well as the condition of the fish texture. These are factors that are directly influenced by how the catching and handling has been carried through, and even with a skilled vessel crew there will be variations in the catch quality. The quality grading today is predominantly done by one to four operators. These grading tasks normally consist of removing non-fish and other fish species, checking for over- or undersized fish in the weight class and removing damaged or defective fish. A common primary processor in the Norwegian industry processes in the range of 90 000 to about 300 000 individual fish per hour. It is not possible to inspect these amounts manually with a sufficient accuracy by only a handful of operators. In addition, manual processing and grading has several other drawbacks. It is influenced by human factors such as mistakes, occasional omission in processing as well as fatigue (Pau and Olafsson, 1991). To meet the increasingly more stringent demands, the primary processor has the choice between more operators or automation of the grading tasks.

The need for automation of basic processing operations to obtain faster processing and a more objective and consistent quality determination has been identified by several researchers already (Strachan and Murray, 1991; Gunnlaugs- son, 1997; Brosnan and Sun, 2004). Recent surveys indicate that the food industry has been rather slow to adapt new automation technologies, yet considers utilizing such technology in the near future (llyukhin et al, 2001a and 2001b). At the moment, the use of machine vision inspection in pelagic industry is tried by several equipment suppliers, among them Cabinplant and Baader, whose systems use just one imaging mode. Robotics is also necessary for automation, but this has predominantly been used to work with homogeneous materials as found in the packaging area in the fishing industry.

Present fish processing is mainly based on the principles of continuous manufacturing where the fish product moves down a manufacturing line to be processed at machine- and manual based specific single operations. Due to a reduction in the cost of elements required for automation, it is now technically feasible to automate tasks in food handling to an increasing extent (Conolly, 2003; llyukhin et. al. 2001a, llyukhin et. al. 2001b). The vast amount of work done in applications similar to the one described below in this invention, suggests that technology for integration of machine vision and robotics has reached the level of maturity that is required in order to solve grading, inspection and processing tasks in the pelagic fish processing industry.

In 2003-2004 several pelagic primary processors took part in a project led by SINTEF Fisheries and Aquaculture, during which the vision of automated quality inspection of whole pelagic fish was born. In general the realization of this vision is still in an early stage, but the present invention permits more automated and cost efficient quality monitoring in food processing.

Summary

The present invention comprises both a method and a system aspect.

The method-aspect of the invention to automatically recognize shape and surface defects of food products 11 using computer image analysis is characterized in that optical data is sampled by at least one optical sensor 5 using combinations of different illumination and sampling modes. This is done to enhance detectability of surface differences, subsurface differences, or shape differences between a defective sample and a non-defective model of the food products. It uses a digital camera or an imaging spectrometer as the optical sensor 5.

The method applies in one embodiment a first illumination and imaging mode where at least one first light source 4 beams substantially in parallel with the axis of view of the optical sensor to illuminate a light pattern onto the food products 11 , resulting in images that measure and/or enhance a 3D-shape of the food products.

In another embodiment the method applies a second illuminating and imaging mode where at least one second light source 6 beams at an substantial angel to the axis of view to illuminate the light pattern onto the food product 11 , resulting in measurement or enhancement of at least one of a UV/VIS/NIR/IR surface, a surface, and internal scattering properties of the food products. In a preferred embodiment of the method a laser is used for one or both of the light sources 4, 6. In a further embodiment of the method at least two lasers are used emitting light at different wavelengths onto the same scan line in the optical sensor field of view for one or both of the light sources 4, 6. According to another embodiment of the invention, light with a wavelength in the 600-700 nm range from at least one of the light sources 4, 6 is emitted. In a further embodiment, the method applies a third illuminating and imaging mode where diffuse light 3 illuminates a light pattern onto the food products 11. The light pattern is at least one of a number of parallel lines. The method at least measures or images surface reflectivity, 3D-shape, and/or diffusivity of the food products 11 or measures surface properties, subsurface properties, or inter- nal scattering properties by using an imaging spectrometer and at least one light source focused into a line.

The method detects abnormality types like surface wounds (broken surface), minor surface damage (scratches on surface), missing parts of the food products, and if the food product is out of a predetermined size range. In a preferred embodiment of the invention, at least two of the illuminating and imaging modes are combined to improve the detectability of the abnormality types.

In further embodiments of the invention a broadband focused line light is used as the light source 4, and a combination of at least one light projector for emitting appropriate illumination patterns and at least one camera for imaging the food products illuminated by the patterns are used to achieve the illumination and imaging modes.

The invention is used for food product like shellfish, whole fish, fillets of fish such as Atlantic salmon, rainbow trout, cod fish, herring and mackerel, meat portions, meat lumps, fruit, and vegetables, specifically is whole salmonid fish such as Atlantic salmon and rainbow trout.

In one embodiment of the invention the food products 11 are conveyed through the field of view of the optical sensor enabling a complete scan of the food products.

The optical sensor 5 of the present invention can in one embodiment be sensitive to light in the 400-1050 nm range.

For further details and aspects of the invention, reference is made to the attached claim set.

Brief Description of the Drawings

Below the present invention will be described with reference to the attached drawings where

Fig 1 shows a perspective view of a system for a conveyor belt, Fig 2 shows the system with a more abstract, non-perspective view

Detailed Description of Preferred Embodiments

Figure 1 shows a preferred embodiment of the invention in a perspective view. A conveyor belt 1 moves the food item 11 - a fish in this example, but also shellfish, meat, fruit and vegetables are possible - from right to left through the field of view of an optical sensor 5 (an electronic camera) mounted above the conveyor belt and connected to a computer 7. Three light sources are shown: (1) A laser 4, mounted directly to the camera 5 achieving a direction of the laser beam substantially in parallel/alignment with the axis of view of the camera 5. (2) A laser 6, mounted in a way that the corresponding beam illuminates the food item at a considerable angel to the axis of view of the camera 5. (3) A diffuse light source 3 mounted typically much closer to the conveyor belt.

The results from the computer 7 calculations performed on the images taken by the camera 5 can be used to manipulate the food product on the belt by some computer controllable manipulating device 2.

Figure 2 shows most of the above described system in a more structural, non-perspective view, also indicating aspects of the illumination itself: (1) a first laser beam 9 projecting on the food item from the laser 4, the beam being substantially in parallel with the axis of view of the camera 5, illuminating a light pattern (f inst one or more parallel lines perpendicular to the conveyor movement direction) onto the food item 11 , resulting in images that enable

5 the software in the computer 7 to measure and enhance the 3D-shape of the food item 11 ,

(2) a second laser beam 10 at an angle relative to the axis of view of the camera 5, illuminating a pattern onto the food item 11 , resulting in images that enable the software in the computer 7 to measure and enhance the UV/VIS/NIR/IR io (ultra-violett, visible, near infra-red, infrared) surface, the surface, and internal scattering properties of the food item; and

(3) a third, diffuse illumination by the light source 3 with a viewing slit perpendicular to the conveyor movement direction where the camera view 8 is indicated to go through the slit along the light source 3 to the food item 11.

15

Multi-modal images are built up from scan lines acquired from the camera 5. These images are transmitted to the computer 7. The software in the computer analyzes these multi-modal images and detects defects in the food products. The processing of images generated by the camera 5 is done in the computer with 20 general off-the-shelf available image processing programs. A signal indicating whether the food product is defect is transmitted from the computer 7 to a manipulation device 2 which in turn acts on the food product item.

There are three imaging modes: 5 A first mode, denoted scatter mode, of the multi-modal images of the food product is created by acquiring several scan lines, one directly on the laser line 9 of laser 4 and one or more with a small offset from the laser line 9 of laser 4.

A second mode, denoted gloss mode, of the multi-modal images is created by acquiring a scan line 8 viewed through the slit of a diffuse light source 3. o A third mode, denoted 3D mode, of the multi-modal images is created by acquiring a 3D range profile by triangulation of the laser line 10 illuminating the food product in an image region in the camera field of view. In one embodiment of the present invention, the food product is whole pelagic fish such as herring and mackerel. In this embodiment the defective fish is characterized by several defect types.

One defect type is superficial wounds indicated by broken fish skin exposing the underlying muscle. A second similar defect type is scratches or scrapes without exposing the underlying muscle. These two defect types can be detected by the present invention using the scatter and gloss modes described above.

A third defect type is characterized by the fish missing large parts or being completely or partially split into several parts. These defect types can be detected by the present invention using the 3D mode described above. Sorting of herring is a specific example where all three modes, - 3D, scatter, gloss - is combined with laser and diffuse illumination and a multi-mode line scan camera to enhance the detectability of defects. Alternatively to the laser illumination, a light projector (e.g. LCD/DLP) can illuminate different patterns onto the food product and a camera takes images of the product with and without these patterns. The pattern-projec- ting-based method is preferable if the food product is not moving in relation to the sensors, while the laser/line based method is better in situations where the food product is moving on - for instance - a conveyor. Based on the pictures - with and without the patterns - the 3D-shape, reflectivity and scattering can be calculated.

A fourth defect type is characterized by the fish being outside a predetermined range of weight or size. These defect types can also be detected by the present invention using the 3D mode described above. In one preferred embodiment the camera 5 has a sensor that is sensitive to visible light in the 400-1050 nm range. In this preferred embodiment, the lasers and the diffuse illumination emit visible light with a wavelength of 600-700 nm. It is obvious to a skilled person that also sensors with sensitivities in other ranges of the electromagnetic spectrum can be used, such as UV, VIS, NIR, SWIR, MWIR LWIR and FIR. The diffuse illumination and lasers will then emit light within the sensors sensitivity range. It's difficult to get uniformly exposed images of fish. This relates to the shape and nature of the surface of fish, which in nature should be invisible for predators coming above, by having a dark and light absorbing pigments in the skin on the back, and for predators coming below, by having a very light and reflective skin on the belly. The present invention therefore uses both gloss and scattering in combination.

The gloss method is particularly effective in the region of the side of the fish and belly but does not work properly on the back of the fish. On the belly and side 5 of the fish without any superficial wounds the light reflection will be even on this part of the fish 11. If a superficial wound or any other abnormal interruption of the reflective surface occurs on the skin, it will create dark regions on the image generated by the camera. The gloss mode is thus effective for generating images that enhance defects on the side and belly of the fish. The image processing io software will detect and evaluate such dark regions and can make the sorting device 2 remove the fish 11.

The scattering method is particularly effective to use on the back of the fish because of high light absorption and little scattering on undamaged fish skin on the back. In open wounds that penetrate parts or all of the fish skin, light will be is scattered from the line light stripe on the fish 11 and tissue will be illuminated beside the light line stripe on the fish 11 and measured in the camera as bright pixels in the scan line next to the light line stripe. In undamaged skin the light will not be scattered as much and thus the scan line next to the light line stripe will appear dark in this case. Thus, in scatter mode images will be acquired in whicho defects on the back of the fish 11 appear brighter than the surrounding skin. The image processing software on the computer 7 will detect and evaluate such bright regions and can make the handling device 2 remove the fish 11.

By using gloss in combination with scattering on target areas on the fish, gloss on belly and side and scattering on the back, the most important areas in the5 quality inspection are covered. In the computer program, selection of the areas at which a method is used in processing of the raw images, is done by finding the food by masking and finding the orientation up and down of the food and further masking areas of interests where the actual processing of the specific method (gloss or scattering) are done. Thus, detecting the head and tail ends and also theo back and belly enables the use of gloss and scatter in the regions of the food where each method is most appropriate for enhancing the defects.

Using the 3D mode makes it possible to correctly orient the fish 11 or shellfish for optimal use of gloss and scatter in the detection of superficial wounds. 3D mode is also used to generate 3D images that can be used to detect gross defects, such as missing heads, tails and split fish.

In another embodiment of the present invention, several or all of the modes are combined to improve the detection of the four above mentioned defect types. In yet another embodiment the food product is whole salmonid fish such as Atlantic salmon and rainbow trout.

In yet another embodiment, the laser 4 is replaced with several lasers emitting light at different wavelengths onto the same scan line in the sensor field of view. In another embodiment the laser 4 is replaced with a broadband focused line light. In these two embodiments the camera may be replaced or supplemented with one or more imaging spectrometers. Using scatter and/or gloss mode of the inspection system will in these embodiments enable detailed quality inspection of food products, including the inspection of properties such as fat content and distribution, water content and distribution, freshness, opaqueness, color and temperature. Further properties, such as presence and distribution of connective tissue, bone, membranes, skin and parasites and can be detected with the inspection system in the present invention. In one embodiment the food product are fillets of fish such Atlantic salmon and rainbow trout, cod fish, herring and mackerel. In a further embodiment the food product is meat portions or meat lumps or fruit or vegetables.

REFERENCES

Barni M, Cappellini V, and Mecocci A 1997, Color-based detection of defects on chicken meat, Image and Vision Computing, 15(7) 549-556 Bordeπas AJ, Gomez-Guillen MC, Hurtado O, and Montero P 1999, Use of image analysis to determine fat and connective tissue in salmon muscle, Eur Food Res Technol, 209 104-107

Brosnan T, and Sun D-W 2002, Inspection and grading of agricultural and food products by computer vision systems - a review, Computers and electronics in agriculture, 36(2-3) 193-213

Cernadas E, Duran ML, and Antequera T 2002, Recognizing marbling in dry-cured Iberian ham by multiscale analysis, Pattern Recognition Letters, 23(1 1) 1311-1321

Connolly C 2003, Automated food handling, Assembly Automation, 23(3) 249-251

Gunnlaugsson GA 1997, Vision technology intelligent fish processing systems, In Luten JB, Børresen T, and Oehlenschlager J (Eds ), Seafood from producer to consumer, integrated approach to quality, Elsevier Science B V , 351-359 Hu B-G, Gosine L, Cao LX, and de Silva CW 1998, Application of a Fuzzy Classification Technique in

Computer Grading of Fish Products, IEEE Transactions on Fuzzy Systems, 6( 1 ) 144- 152

Ilyukhin SV, Haley TA, and Singh RK 2001a, A survey of automation practices in the food industry, Food control, 12(5) 285-296

Ilyukhin SV, Haley TA, Singh RK 2001b, A survey of control system validation practices in the food industry, Food Control, 12(5) 297-304

Jansson S and Mathiassen JR 2006, US Patent Application (Provisional) No 60/835,464

Kroeger M 2003, Image analysis for monitoring the quality of fish, In Luten JB, Oehlenschlager J, and Olafsdottir G (Eds ), Quality offish from catch to consumer Labelling, monitoring and traceability, The Netherlands Wageningen Academic Publishers, 211-224 Mary-Mahe P, Loisel P, Fauconneau B, Haffray P, Brossard D, and Davenel A 2004, Quality traits of brown trouts (Salmo trutta) cutlets described by automated color image analysis, Aquaculture, 232 225-240

Misimi, E, Mathiassen JR, Eπkson U, Skavhaug A 2006 Computer vision based sorting of Atlantic salmon (Salmo salar) according to shape and size In Proceedings of VISAPP International Conference on Computer Vision Theory and Applications, 2006 February 25-28, Setubal, Portugal p 265-270 Patel VC, McClendon RW, and Goodrum JW 1998, Color Computer Vision and Artificial Neural Networks for the Detection of Defects in Poultry Eggs, AI Review, 12(1-3) 163-176

Pau LF, and Olafsson R 1991, Fish quality control by computer vision, Marcel Dekker, New York, 23-38

Shiranita K, Hayashi K, Otsubo A, Miyajima T, and Takiyama R 2000, Grading meat quality by image processing, Pattern Recognition, 33(1) 97-104 Shiranita K, Miya_jima T, and Takiyama R 1998, Determination of meat quality by texture analysis, Pattern

Recognition Letters, 19(14) 1319- 1324

Strachan NJC 1993, Recognition offish species by color and shape, Image and Vision Computing, 11(1) 2- 10

Strachan NJC, and Murray CK 1991, Image analysis in the fish and food industries In Pau LF, and Olafsson R (Eds ) Fish quality control by computer vision, Marcel Dekker, New York, 209-223

Strachan NJC, Nesvadba P, and Allen AR 1990, Fish species recognition by shape analysis of images, Pattern Recognition, 23(5) 539-544

Yoshikawa F, Toraichi K, Wada K, Ostu N, Nakai H, Mitsumoto M, and Katagishi K 2000, On a grading system for beef marbling, Pattern Recognition Letters, 21(12) 1037- 1050

Claims

PAT E N T C LA I M S

1. Method to automatically recognize shape and surface defects of food prod- ucts (11 ) using computer image analysis, characterized in that optical data is sampled by at least one optical sensor (5) with an axis of view and a field of view, using combinations of different illumination and sampling modes to enhance detectability of at least one of a surface difference, - subsurface difference, and shape difference between a defective sample and a non-defective model of said food products.

2. Method according to claim 1 , characterized by using at least one of a digital camera and an imaging spectrometer as said optical sensor (5).

3. Method according to claim 2, characterized by applying a first illumination and imaging mode where at least one first light source (4) beams substantially in parallel with said axis of view to illuminate a light pattern onto said food products (11), resulting in images that at least one of measure and enhance a 3D-shape of said food products.

4. Method according to claim 2, characterized by applying a second illuminating and imaging mode where at least one second light source (6) beams at an substantial angel to said axis of view to illuminate said light pattern onto said food products (11), resulting in measurement or enhancement of at least one of - a UVΛ/IS/NIR/IR surface, surface, and internal scattering properties of said food products.

5. Method according to claim 3 or 4, characterized by using a laser for said at least one of said first light source (4) and said second light source (6).

6. Method according to claim 3 or 4, characterized by using at least two lasers emitting light at different wavelengths onto the same scan line in said optical sensor field of view for said at least one of said first light source (4) and second light source (6).

7. Method according to claim 2, characterized by applying a third illuminating and imaging mode where diffuse light (3) illuminates a light pattern onto said food products (11).

8. Method according to claim 2 or 3 or 7, characterized i n that said light pattern is at least one of a number of parallel lines.

9. Method according to one of the preceding claims, characterized by at least one of measuring and imaging at least one of surface reflectivity,

3D-shape diffusivity of said food products (11).

10. Method according to one of the preceding claims, characterized by measuring at least one of surface properties, - subsurface properties, and internal scattering properties using an imaging spectrometer and at least one light source focused into a line.

11. Method according to one of the preceding claims, characterized by detecting abnormality types like surface wounds (broken surface), minor surface damage (scratches on surface), 5 - missing parts of said food products, and said food product out of a predetermined size range.

12. Method according to claim 11 , characterized i n that at least two of said illuminating and imagingo modes are combined to improve the detectability of said abnormality types.

13. Method according to claim 2, characterized by using a broadband focused line light as said light source (4). 5

14. Method according to claim 3 and 4, characterized by using a combination of at least one light projector for emitting appropriate illumination patterns and at least one camera for imaging said food products illuminated by said patterns to achieve said illumination and0 imaging modes.

15. Method according to claim 14, characterized in that said food product is one of shellfish, 5 - whole fish, fillets of fish such as Atlantic salmon, rainbow trout, cod fish, herring and mackerel, meat portions, meat lumps, o - fruit, and vegetables.

16. Method according to claim 15, characterized in that said food product is whole salmonid fish such as

Atlantic salmon and rainbow trout.

17. Method according to one of the preceding claims, characterized in that said food products (11) are conveyed through said field of view of said optical sensor enabling a complete scan of said food products.

18. Method according to claim 1 , characterized by said optical sensor (5) being sensitive to light in the 400-1050 nm range.

19. Method according to claim one of the preceding claims, characterized by emitting light with a wavelength in the 600-700 nm range from at least one of said first light source (4) and second light source (6).

20. System for automatically recognizing shape and surface defects of food products (11) using computer image analysis, characterized by different illumination means to illuminate said food products and at least one optical sensor (5) with an axis of view and a field of view, sampling optical data in different sampling modes to enhance detectability of at least one of a surface difference, subsurface difference, and shape difference between a defective sample and a non-defective model of said food products.

21. System according to claim 20, characterized i n that said optical sensor (5) is at least one of a digital camera and an imaging spectrometer.

22. System according to claim 21 , characterized by a first illumination and imaging mode where at least one first light source (4) beams substantially in parallel with said axis of view to illuminate a light pattern onto said food products (11), resulting in images that at least one of measure and enhance a 3D-shape of said food products.

23. System according to claim 21 , characterized by a second illuminating and imaging mode where at least one second light source (6) is directed at an substantial angel to said axis of view to illuminate said light pattern onto said food products (11), resulting in measurement or enhancement of at least one of a UV/VIS/NIR/IR surface, surface, and internal scattering properties of said food products.

24. System according to claims 22 or 23, characterized by a laser for said at least one of said first light source (4) and said second light source (6).

25. System according to claims 22 or 23, characterized by at least two lasers for said at least one of said first light source (4) and second light source (6) emitting light at different wavelengths onto the same scan line in said optical sensor field of view.

26. System according to claim 21 , characterized by a third illuminating and imaging mode where diffuse light (3) illuminates a light pattern onto said food products (11).

27. System according to claims 21 or 22 or 26, characterized in that said light pattern is at least one of a number of parallel lines.

28. System according to one of the claims 20 to 27, characterized by being arranged to at least one of measure and image at least one of surface reflectivity, - 3D-shape diffusivity of said food products (11).

29. System according to one of the claims 20 to 27, characterized by being arranged to measure at least one of surface properties, subsurface properties, and internal scattering properties using an imaging spectrometer and at least one light source focused into a line.

30. System according to one of the claims 20 to 27, characterized by being arranged to detect abnormality types like surface wounds (broken surface), minor surface damage (scratches on surface), - missing parts of said food products, and food product out of a predetermined size range.

31. System according to claim 30, characterized by a combination of at least two of said illuminating and imaging modes to improve the detectability of said abnormality types.

32. System according to claim 21 , characterized by a broadband focused line light being that said light source (4).

33. System according to claim 22 and 23, characterized by a combination of at least one light projector for emitting appropriate illumination patterns and at least one camera for imaging said food products illuminated by said patterns to achieve said illumination and imaging modes.

34. System according to claim 33, characterized in that said food product is one of shellfish, - whole fish, fillets of fish such as Atlantic salmon, rainbow trout, cod fish, herring and mackerel, meat portions, meat lumps, - fruit, and vegetables.

35. System according to claim 34, characterized in that said food product is whole salmonid fish such as Atlantic salmon and rainbow trout.

36. System according to one of the claims 20 to 35, characterized by a conveyor arrangement moving said food products (11) through said field of view of said optical sensor enabling a complete scan of said food products.

37. System according to claim 20, characterized by said optical sensor (5) being sensitive to light in the 400-1050 nm range.

38. System according to claims 20 to 37, characterized by at least one of said first light source (4) and second light source (6) emitting light with a wavelength in the 600-700 nm range.