WO2010021972A1

WO2010021972A1 - Surround structured lighting for recovering 3d object shape and appearance

Info

Publication number: WO2010021972A1
Application number: PCT/US2009/054007
Authority: WO
Inventors: Douglas Lanman; Daniel Crispell; Gabriel Taubin
Original assignee: Brown University
Priority date: 2008-08-18
Filing date: 2009-08-17
Publication date: 2010-02-25

Abstract

Disclosed are methods and apparatus for recovering the shape and appearance of an object illuminated by a coded structured light pattern that is observed by a camera, which simultaneously captures multiple views of the object. In one embodiment a Fresnel lens and a data projector are used as the means of illuminating the object with surrounding structured light patterns. The projector projects several patterns and the camera captures images of the object for each pattern. The images are decoded to determine the correspondence between projector scan-lines and camera pixels. Combined with the calibration of the relative position of the various optical elements, including the projector and camera, the ray-plane intersection is used to determine the three-dimensional depth for each part of the surface. An additional image captured under ambient illumination is used to recover the appearance for each surface point for each camera pixel.

Description

SURROUND STRUCTURED LIGHTING FOR RECOVERING 3D OBJECT SHAPE AND APPEARANCE

BACKGROUND OF THE INVENTION

This invention generally relates to systems and methods for recovering the shape and appearance of objects from images captured under structured lighting. More particularly, this invention relates to systems and methods for recovering the shape and appearance of objects, where the systems comprise a single camera and a single projector displaying structured light patterns.

Reconstructing the shape and appearance of three- dimensional (3D) objects is also referred to in the prior art as "3D scanning." Many prior art 3D scanning methods are referred to as triangulation-based methods. Coded structured light is a particularly reliable and inexpensive triangulation-based method for reconstructing the shape and appearance of three dimensional objects from image data. In its simplest form, a coded structured light system comprises a single calibrated projector-camera pair. By illuminating the object surface with a known sequence of coded images, the correspondence between projector pixels and camera pixels can be uniquely identified. One such sequence of coded images is known as Gray codes. Inokuchi et al. originally applied Gray codes to 3D scanning; see S. Inokuchi, K. Sato, and F. Matsuda, "Range imaging system for 3-D object recognition," International Conference on Pattern Recognition, 1984. In this system and method, each pattern is composed of a sequence of black and white stripes oriented along the projector scan-lines. Each projector scan-line corresponds to a projected light plane. By illuminating the object with a temporally-multiplexed sequence of increasingly- fine Gray code patterns, the corresponding projector scan- line can be identified for each camera pixel. Each image pixel defines a camera ray supported by a straight line in 3D. The intersection of each camera rays with its corresponding projected light plane determines the location of a 3D point on the subset of the surface illuminated by the Gray codes. The collection of all these 3D points is the result of the coded structured light method. Alternatively, a polygon mesh is reconstructed by interconnecting these 3D points forming polygon mesh faces .

The main problem with the coded structured light methods as described in the previous paragraph, is that only the portion of the object surface simultaneously illuminated by the Gray codes and visible from the camera can be reconstructed. To reconstruct other parts of the object surface, the projector and camera must me moved so that other parts of the object surface are simultaneously visible and illuminated. In general, this process needs to be repeated several times to cover the whole surface of the object, and then the various 3D scans produced at each relative position of camera and projector with respect to the object, must be merged to produce a single integrated 3D scan.

An alternative to moving the projector and camera is to use mirrors to create virtual structured light projectors and virtual camera views.

The idea of using planar mirrors to create multiple virtual structured light projectors was first presented by Epstein et al . ; see E. Epstein, M. Granger-Piche, and P. Poulin, "Exploiting mirrors in interactive reconstruction with structured light", Vision, Modeling, and Visualization, 2004. In their system, one or more planar mirrors are illuminated by a projector displaying a modified Gray code sequence which is invariant to mirror reflections. By visually tracking the relative camera, projector, and mirror positions and by interactively selecting a conservative object bounding box, the authors mask the projected patterns to ensure that each surface point is illuminated from a single direction in each image. While eliminating the need for multiplexing several projectors to obtain complete object models, this system still suffers from several limitations. Foremost, it increases the number of required patterns since the directly and indirectly viewed portions of the object surface cannot be illuminated simultaneously.

Several prior art methods use planar mirrors to obtain several virtual views of a 3D scene or object from a single image capture. As discussed by Gluckman and Nayar, the virtual cameras created by planar mirror reflections have several benefits over multiple camera systems, including automatic frame synchronization, color calibration, and identical intrinsic calibration parameters; see J. Gluckman and S. Nayar, "Planar Catadioptric Stereo: Geometry and Calibration", International Conference on Computer Vision and Pattern Recognition, 1999. While Gluckman and Nayar restricted their attention to stereo catadioptric systems in that work, Forbes et al . have explored a configuration comprising a mirror surface oriented such that an object placed between the two mirrors will produce one real and four virtual viewpoints, resulting from the first and second reflections; see K. Forbes, F. Nicolls, G. de Jager, and A. Voigt, "Shape-from-silhouette with two mirrors and an uncalibrated camera", European Computer on Computer Vision, 2006. These authors obtained a complete 3D surface model by estimating the approximate visual hull defined by the five object silhouettes. A large number of methods to reconstruct the shape of objects from multiple silhouettes exist in the prior art. These methods are referred to as shape-from-silhouette methods. Some shape- from-silhouette methods approximate the visual hull of the object. The main problem with these methods is that they are unable to estimate the 3D position of points located inside concavities of the object surface. On the other hand, methods based on coded structure lighting can recover those points as long as they are visible from the camera and illuminated by the projector.

In conclusion, there is a need for a 3D scanning method which overcomes the limitations of existing Coded structured light systems and methods, and is able to rapidly capture a full 360 degree 3D scan of an object surface using only a single camera and a single projector without any repositioning of cameras and projectors, or merging of multiple scans.

BRIEF SUMMARY OF THE INVENTION

Disclosed herein is a three-dimensional (3D) scanning apparatus to recover the shape and appearance of an object, comprising a camera, an illuminator, and a mirror surface. The camera and illuminator are placed in front of the object to be 3D scanned, and the mirror surface is placed behind the object. The camera and illuminator are pointed towards the object and the mirror surface. The illuminator projects light patterns on the object and on the mirror surface. The light patterns are composed of projected light rays. The mirror surface is a generalized cylinder with a mirror axis, and the projected light rays are perpendicular to the mirror axis. The illuminator and mirror surface are placed in such a way that most of the surface of the object is illuminated either directly or indirectly. While part of the surface of the object is illuminated by projected light rays emanating directly from the illuminator, other parts of the surface of the object are illuminated indirectly by reflected light rays resulting from projected light rays bouncing off the mirror surface after one or more mirror reflections. The camera has the object and part of the mirror surface simultaneously on view, resulting in the camera simultaneously observing the object from multiple directions. In a more preferred embodiment the mirror surface is composed of a left planar mirror and a right planar mirror. The left planar mirror and right planar mirror meet at the mirror axis, and are separated by a mirror angle of approximately 72 degrees. The camera captures a single camera view which includes one real image and four reflected images.

Disclosed herein is also a method to recover the shape and appearance of an object. The camera is used to capture camera views of the object under a given illumination pattern displayed by the illuminator. In the presently preferred embodiment, the illumination patterns correspond to a sequence of Gray code patterns. In a first step of the method the Gray coded images are captured using the 3D scanning apparatus disclosed herein. In a second step, for each image captured under a given structured light illumination pattern, each pixel is identified as being either illuminated or in shadow. In a third step, after processing multiple such images, the set of thresholded values for each camera pixel are analyzed to determine which projector scanline illuminated each pixel. Equivalently, since each projector scanline corresponds to one of the planes contained in a plurality of parallel light planes emitted by the illuminator, this second step determines which plane of light illuminates a given camera pixel. In a third step the object shape is estimated from the structured light sequence. In the preferred embodiment, ray-plane triangulation is used to reconstruct the three-dimensional coordinate of a point on the object surface for each camera pixel. Furthermore, these points can be analyzed to produce a polygon mesh to represent the object shape.

An optional additional image collected under ambient illumination, with no pattern projected by the illuminator, is used to further estimate the appearance (e.g., color) of the estimated object surface. Furthermore, a plurality of ambient images, collected for various poses of the object and ambient illumination could be collected to obtain a better approximation of the object appearance.

In summary, the set of illumination images and the ambient image provide sufficient information to reconstruct the shape and appearance of a 3D object placed within the system.

It is a first object and advantage of this invention to provide a system and method projecting structured illumination patterns which surround an object.

It is a second object and advantage of this invention to provide an improved system and method to obtain the 3D shape and photometric appearance of an object from one or more images.

It is a further object and advantage of this invention to provide a system and method for deriving the surface shape and appearance of an object from images collected by projecting structured illumination patterns wherein the projector is designed in order to surround an object using a multitude of optical elements.

These together with other objects of the invention, along with various features of novelty that characterize the invention, are pointed out with particularity in the claims annexed hereto and forming a part of this disclosure. For a better understanding of the invention, its operating advantages and the specific objects attained by its uses, reference should be had to the accompanying drawings and descriptive matter in which there is illustrated a preferred embodiment of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above set forth and other features of the invention are made more apparent in the ensuing Detailed Description of the Invention when read in conjunction with the attached Drawings, wherein:

FIG. 1 is an illustration depicting the presently preferred embodiment of a system for obtaining the shape and appearance of an object by capturing a set of images under varying projected structured light illumination patterns using a camera, a standard "pinhole" projector, a mirror surface, and a large aperture converging Fresnel lens .

FIG. 2 is an illustration depicting a general form of a system for obtaining the shape and appearance of an object by capturing a set of images under varying projected structured light illumination patterns.

FIGS. 3a and 3b are diagrams depicting possible geometric properties of the light source and means of bending the light rays in order to form the plurality of first parallel planes shown in FIG. 2.

FIG. 4 is an illustration of the first four Gray code structured light illumination patterns.

FIGS. 5a, 5b, and 5c are illustrations depicting a typical image collected by the invention under preferred illumination, wherein FIG. 5a illustrates a typical object under ambient illumination, FIG. 5b shows the same object under a Gray code structured light illumination as in FIG. 4, and FIG. 5c shows the decoded projector scan-line with darker colors indicating larger scan-lines.

FIG. 6 is a logic flow diagram for the processing of the set of images computed in accordance with FIG. 4 into an estimate of the correspondence between camera pixels and projector scan-lines. FIG. 7 is an illustration depicting a second embodiment of a system for obtaining the shape and appearance of an object by capturing a set of images under varying projected structured light illumination patterns, wherein an array of cameras replaced the single camera of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

A presently preferred first embodiment of the invention is depicted in FIG. 1, a 3D scanning apparatus 100 to recover the shape and appearance of an object 120, comprising a camera 101, an illuminator 110, and a mirror surface 130; the camera 101 and illuminator 110 placed in front of an object 120 to be 3D scanned; the mirror surface 130 beeing placed behind the object 120; the camera 101 and illuminator 110 pointing towards the object 120 and the mirror surface 130; the illuminator 110 projecting light patterns 111 on the object 120 and mirror surface 130; the light patterns 111 composed of projected light rays 112, 113; the mirror surface 130 being a generalized cylinder with a mirror axis 133 as the generalized cylinder axis; the projected light rays 112, 113 being perpendicular to the mirror axis 133; the illuminator 110 and mirror surface 130 placed in such a way that most of the surface of the object 120 is illuminated either directly or indirectly; part of the surface of the object 120 being illuminated by projected light rays 113 emanating directly from the illuminator 110; other parts of the surface of the object 120 being illuminated indirectly by reflected light rays 114 resulting from projected light rays 112 bouncing off the mirror surface 130 after one or more mirror reflections; the camera 101 having the object 120 and part of the mirror surface 130 simultaneously on view, resulting in the camera 101 simultaneously observing the object 120 from multiple directions.

In the prior art, a generalized cylinder is a surface with an axis and a cross section. The axis is a straight line, and the cross section is a planar curve contained in a plane perpendicular to the axis. The generalized cylinder can be described as the result of sweeping the plane containing the cross section curve perpendicularly along the axis. As a result of this process, a point on a generalized cylinder can be described by two parameters: one along the axis, and the second one along the cross section. For example, two intersecting planes define a generalized cylinder where the axis is the intersecting line, and the cross section is the union of two intersecting lines perpendicular to the axis. The surfaces of so called "cylindrical lenses" and "cylindrical mirrors" are other examples of generalized cylinders, where the cross sections are circles or parabolas. It is a well known property that a plane tangent to a point on the surface of a generalized cylinder is parallel to the axis. Equivalently, the normal direction to a point on the surface of a generalized cylinder is a vector perpendicular to the axis. It follows that the reflection of a light ray perpendicular to the axis of a generalized cylinder mirror surface is also perpendicular to the axis. In a more preferred embodiment the mirror surface 130 is composed of a left planar mirror 131 and a right planar mirror 132; the left planar mirror 131 and right planar mirror 132 meeting at the mirror axis 133; the left planar mirror 131 and right planar mirror 132 separated by a mirror angle 134 of approximately 72 degrees; the camera 101 capturing a single camera view 150; the single camera view 150 simultaneously measuring one real image 155 and four reflected images 151, 152, 153, and 154.

Since the projected light ray 112 is perpendicular to the mirror axis 133, and the mirror surface 130 is a generalized cylinder with respect to the mirror axis 133, the reflected light ray 114 is also perpendicular to the mirror axis 133. A key feature of this invention is this property which prevents the reflected light ray 114 from interfering with other projected light rays 112, 113, or other reflected light rays 114 on the surface of the object 120, after they reflect off the mirror surface 130. Through this mechanism, this invention is able to significantly reduce the number of images required to capture the shape and appearance of the object 120, and avoid moving the object 120 with respect to the imaging system.

In a more preferred embodiment the projected light rays 112, 113, are contained in a plurality of parallel light planes 115, so that all the reflected light rays 114 corresponding to projected light rays 112 contained in the same light plane 115, also belong to the same light plane 115, and light rays 112, 113, 114 contained in different first parallel planes do not interfere with each other. In an even more preferred embodiment all the projected light rays 112, 113 belonging to the same parallel light plane 115 are projected with the same color.

In a preferred embodiment the illuminator 110 is an orthographic projector. An orthographic projector emits projected light rays 112, 113 which are all parallel to each other. In a more preferred embodiment the orthographic projector is composed of a pinhole projector and a coaxial convergent lens position so that one of its focal points coincides with the center of projection of the projector.

A prior art method to implement an orthographic projector system using a data projector and a Fresnel lens was introduced by Nayar and Anand; see S. K. Nayar and V. Anand, "Projection Volumetric Display Using Passive Optical Scatterers", 2006. In their application, the orthographic projector illuminated passive optical scatterers to create a volumetric display. While demonstrating the construction of an orthographic projector using a Fresnel lens and a DLP projector, their invention only applies to the display of 3D content; not to 3D scanning methods.

FIG. 2 shows a preferred embodiment of the illuminator 110, comprising a light source 200, and a means of bending 220; the light source 200 emitting light source rays 210; the light source rays 210 being transformed into projected light rays 112, 113 by the means of bending 220; the projected light rays constituting the light pattern 111. In a more preferred embodiment the light source 200 is a data projector driven by a computer. In another more preferred embodiment the means of bending 220 is achieved by using a convergent lens. In an even more preferred embodiment the means of bending 220 is achieved by using a large aperture convergent Fresnel lens. In another more preferred embodiment, the means of bending 220 is achieved by using a cylindrical lens. In another more preferred embodiment, the means of bending 220 is achieved by using a plurality of mirror surfaces. In a more preferred embodiment, the means of bending 220 is achieved by using a plurality of planar mirror surfaces.

FIG. 3a shows a second embodiment of the illuminator 110, wherein components that are also found in FIG. 2 are numbered accordingly. In this embodiment, the light source rays 210 are contained in a plurality of source parallel light planes 300, which are subsequently transformed by the means of bending 220 into the plurality of parallel light planes 115.

FIG. 3b shows a third embodiment of the illuminator 110, wherein components that are also found in FIG. 2 are numbered accordingly. In this embodiment, the light source emits a single source parallel plane 310, which is subsequently transformed by the means of bending 220. In a more preferred embodiment the light source 200 which emits a single source parallel plane 310 is implemented using a laser stripe projector. In an even more preferred embodiment the means of bending 220 include a mechanically-manipulated mirror in order to time- sequentially redirect the single second parallel plane into each plane of the plurality of parallel light planes 115.

The positions of the camera 101, the light source 200, the means of bending 220, and the mirror surface 130 must be calibrated with respect to a coordinate system defined with respect to the center of projection of the camera (or other global reference position) . Any well- known calibration or measurement technique for obtaining camera and projector parameters and measuring object locations may be used. If the positions are suitably calibrated, the object 120 can be located at any point with a reconstruction volume within which points on the object surface are both imaged by the camera and illuminated by the projection system.

Camera 101 is used to capture camera views 150 of object 120 viewed under a given illumination pattern displayed by the illuminator 110. In the presently preferred embodiment, the illumination patterns correspond to a sequence of Gray code patterns, a few of which 400, 410, 420, 430, are depicted in FIG. 4. Each Gray code pattern is composed of alternating black and white stripes of increasingly-fine width. An exemplary Gray coded captured camera view 510 is shown in FIG. 5. These images are processed according to the logic flow diagram depicted in FIG. 6. In step 600 Gray coded images 510 are captured. In step 610, for each image captured under a given structured light illumination pattern, each pixel is identified as being either illuminated or in shadow. Any suitable pixel thresholding operation may be performed in step 610. In step 620, after processing multiple such images, the set of thresholded values for each camera pixel are analyzed to determine which projector scanline illuminated each pixel. Equivalently, since each projector scanline corresponds to one of the planes contained in the plurality of parallel light planes 115 shown in FIG. 1, step 620 determines which plane of light illuminates a given camera pixel. 520 is an exemplary labeling of projected scanlines produced by step 620. Any suitable analysis may be applied to decode the images to determine which plane illuminated a given camera pixel. For a Gray code structured light sequence, the Gray code decoding scheme may be employed. Any sequence which can be decoded in such a manner can be used within our invention. In step 630 the object shape is estimated from the structured light sequence. Any suitable prior-art method may be used for this purpose, such as the Gray code patterns shown in FIG. 4. In the preferred embodiment, ray-plane triangulation is used to reconstruct the three-dimensional coordinate of a point on the object surface for each camera pixel. In such a method, the projector scan-line illuminating a given pixel is determined in step 620. The calibration of the position of the camera, the projector, and the means of bending are used to determine the equation of the planes within the plurality of parallel light planes 115, as well as the equation of the line connecting the center of projection of the camera with a given pixel. The intersection of a plane with a camera ray will provide an estimate for the three-dimensional coordinate of a point on the object surface. This analysis can be applied independently to allow points on the object surface to represent the object shape. Furthermore, these points can be analyzed to produce a polygon mesh to represent the object shape.

An additional image 500 collected under ambient illumination, with no pattern projected by the illuminator 110, is be used to further estimate the appearance (e.g., color) of the estimated object surface. The recovered set of three-dimensional points representing the object surface can be combined with the system calibration, to determine which pixels in the ambient image correspond to a given point on the surface. This correspondence can be used to estimate the appearance of each three-dimensional point. This scheme is known as view-independent texture mapping. In general, any texture mapping or appearance modeling scheme could be applied, such as view-dependent texture mapping. Furthermore, a plurality of ambient images, collected for various poses of the object and ambient illumination could be collected to obtain a better approximation of the object appearance as shown in FIG. 7.

While there is shown and described herein certain specific structure embodying the invention, it will be manifest to those skilled in the art that various modifications and rearrangements of the parts may be made without departing from the spirit and scope of the underlying inventive concept and that the same is not limited to the particular forms herein shown and described except insofar as indicated by the scope of the appended claims .

Claims

WHAT IS CLAIMED:

1. An apparatus for illuminating an object comprising:

(a) a mirror having a surface, the mirror surface being positioned behind the object and being in the form of a generalized cylinder axis, the mirror surface being a generalized cylinder having a mirror axis as the generalized cylinder axis;

(b) an illuminator formed and positioned in front of the object and having an illumination point directed toward the object and the mirror surface so as to project light patterns composed of light rays perpendicularly to the mirror axis onto the object and the mirror surface in such a way that most of the surface of the object is illuminated either directly or indirectly, with part of the surface of the object being illuminated by projected light rays emanating directly from the illuminator, and other parts of the surface of the object being illuminated indirectly by reflected light rays resulting from projected light rays bouncing off the mirror surface after one or more mirror reflections.

2. An apparatus as in claim 1, where the illuminator comprises a light source which emits light source rays, and a means of bending the light source rays to produce the projected light rays.

3. An apparatus as in claim 2, where the light source is a data projector driven by a computer.

4. An apparatus as in claim 2, where the means of bending is a convergent lens.

5. An apparatus as in claim 4, where the convergent lens is a Fresnel lens.

6. An apparatus as in claim 2, where the means of bending is a telecentric lens.

7. An apparatus as in claim 2, where the means of bending is a cylindrical lens.

8. An apparatus as in claim 2, where the means of bending is a plurality of mirror surfaces.

9. An apparatus as in claim 7, where the mirror surfaces are planar mirror surfaces.

10. An apparatus as in claim 2, where the light source rays are contained in a plurality of source parallel planes .

11. An apparatus as in claim 3 where the plurality of source parallel planes is composed of a single source parallel plane.

12. An apparatus as in claim 4 where the means of bending includes a mechanically manipulated mirror in order to time-sequentially redirect the single source parallel plane into each plane of the plurality of parallel light planes .

13. An apparatus as in claim 1, where the illuminator is an orthographic projector.

14. A three-dimensional (3D) scanning apparatus to recover the shape and appearance of an object, comprising:

(b) an illuminator formed and positioned in front of the object and having an illumination point directed toward the object and the mirror surface so as to project light patterns composed of light rays perpendicularly to the mirror axis onto the object and the mirror surface in such a way that most of the surface of the object is illuminated either directly or indirectly, with part of the surface of the object being illuminated by projected light rays emanating directly from the illuminator, and other parts of the surface of the object being illuminated indirectly by reflected light rays resulting from projected light rays bouncing off the mirror surface after one or more mirror reflections; and

(c) a camera, formed and positioned to have the object and part of the mirror surface simultaneously on view, resulting in the camera simultaneously observing the object from multiple directions.

15. A method of recovering the shape of an object, comprising the steps of:

(a) illuminating the object with an apparatus as in claim 1;

(b) capturing a plurality images of the object; each image corresponding to one light pattern;

(c) processing the plurality of images of the object to determine the correspondence between light pattern scanlines and image pixels, each light pattern scanline corresponding to a parallel light plane, each image pixel corresponding to a camera ray; and

(d) estimating the shape of the object from the structured light sequence using ray-plane triangulation, the shape comprising a plurality of shape points, each shape point resulting from the intersection of a parallel light plane and a corresponding camera ray.

16. A method as in claim 15, further comprising the step of reconstructing a polygon mesh from the plurality of shape points.

17. A method as in claim 15, wherein the processing step is based on determining which pixels are either illuminated or in shadow using a pixel thresholding operation .

18. A method as in claim 15, wherein the light pattern is a Gray code sequence, and the processing step is based on the Gray code decoding scheme.

19. A method as in claim 15, wherein each image of the object is captured by one of a plurality of cameras.

20. A method as in claim 15, wherein the plurality of images of the object are captured by a single camera.

21. A method as in claim 15, to recover the shape and appearance of an object, further comprising the step of: capturing an additional image under ambient illumination, with no pattern projected by the illuminator, to further estimate the appearance of the estimated object surface.

22. A method as in claim 21, wherein the reconstructed shape is represented as a plurality of points, and the appearance is represented as a three dimensional color vector per point.

23. A method as in claim 21, wherein the reconstructed shape is represented as a polygon mesh, and the appearance is represented as a three dimensional color vector per polygon mesh vertex.

24. A method as in claim 21, wherein the reconstructed shape is represented as a polygon mesh, and the appearance is represented as a texture mapping.

25. A method as in claim 24, wherein the texture mapping is a view-dependent texture mapping.

26. A method as in claim 15, to recover the shape and appearance of an object, further comprising the steps of: capturing a plurality of additional image under different ambient illuminations, with no pattern projected by the illuminator, to further estimate the appearance of the estimated object surface.

27. A method as in claim 26, wherein the reconstructed shape is represented as a polygon mesh, and the appearance is represented as a view-independent texture mapping; the view-independent texture mapping reconstructed using a photometric stereo algorithm.