US20110222757A1 - Systems and methods for 2D image and spatial data capture for 3D stereo imaging - Google Patents
Systems and methods for 2D image and spatial data capture for 3D stereo imaging Download PDFInfo
- Publication number
- US20110222757A1 US20110222757A1 US12/932,789 US93278911A US2011222757A1 US 20110222757 A1 US20110222757 A1 US 20110222757A1 US 93278911 A US93278911 A US 93278911A US 2011222757 A1 US2011222757 A1 US 2011222757A1
- Authority
- US
- United States
- Prior art keywords
- camera
- cameras
- images
- image
- stereo
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/246—Calibration of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/257—Colour aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/327—Calibration thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
Definitions
- the present disclosure relates generally to creating three-dimensional (3D) stereo images from two-dimensional (2D) photography, and in particular to systems and methods for 2D image capture and spatial data capture for 3D stereo imaging.
- 3D stereo imaging from 2D photography.
- the available 3D technologies do so through interpretive and creative means, or through system configurations that do not capture the true depth and geometry of the original environment using the 2D photography.
- a conventional and popular technology used to make a 3D stereo image is to use two cameras separated by the typical human interocular distance D H (i.e., human eye spacing), which is assumed in the industry to be about 68 mm, though smaller distances D H are often used for reasons described below.
- D H human eye spacing
- the two cameras are then oriented (angled) so that their fields of view converge and overlap at a distance DS where the various objects in the scene being filmed are located.
- the actual image-capture process does not collect a substantial amount of true 3D information for the given scene mainly because the interocular distance D H is too small relative to the distance D S .
- the amount of spatial (3D) data captured by such an arrangement is far smaller than the actual 3D volume of the scene being imaged.
- An aspect of the disclosure is a method of converting 2D images of a scene having therein at least one object to one or more 3D images of the scene.
- the method includes simultaneously capturing at least first and second 2D images of the scene from corresponding at least first and second cameras having respective camera positions and orientations measured relative to a reference coordinate system.
- the method also includes forming a disparity map from the at least first and second 2D images, wherein the disparity map has a gray scale that corresponds to distance information of the at least one object relative to the reference coordinate system.
- the method also includes forming from the disparity map a 3D point cloud P(x,y,z) representative of the at least one object, wherein the point cloud is configured to support first and second virtual cameras to create a stereo camera pair arrangeable in substantially arbitrary virtual locations.
- Another aspect of the disclosure is a method of forming a distance representation of a scene from 2D images of the scene.
- the method includes simultaneously capturing at least first and second 2D images of the scene from corresponding at least first and second cameras having respective camera positions and orientations measured relative to a reference coordinate system.
- the method also includes defining one or more regions of interest in the at least first and second 2D images.
- the method further includes associating differences between pixels in the at least first and second cameras with distances from a reference point, and assigning different gray-scale intensities to different ones of the distances.
- An aspect of the disclosure includes creating a depth map or “disparity map” using gray-scale variations to represent distances from the camera (or reference coordinates) associated with one or more objects in the scene.
- the process of creating the disparity map can be made semi-automatic using image data from multiple calibrated cameras.
- the range data in the disparity map is transformed to match the perspective and viewing frustum of the cinemagraphic (“cine”) camera.
- the 2D photographed image from the cine camera is embedded with the range data at sub-pixel accuracy to facilitate post-production to create an accurate and more continuous 3D stereo pair from this true 3D data.
- the range data can also be expressed as a “point cloud” P(x,y,z) representing spatial positions of thousands of points (pixols) in three-space that make up the 3D scene.
- P(x,y,z) representing spatial positions of thousands of points (pixols) in three-space that make up the 3D scene.
- the position and placement of every pixel in the cine camera in 3-space is computed with 16-bit color information.
- the range data can be expressed as a polygonal mesh PM(x,y,z;RGB) of spatial positions and the corresponding RGB color, which allows the 2D photographed images from the cine camera to be mapped directly on to the mesh in motion as texture.
- the textured 3D geometry can then be re-imaged with virtual cameras in post-production, setting the appropriate interocular distance and convergence values for the scene, thus creating a true left and a true right eye stereo pair for theatrical viewing.
- Multiple cameras with relatively large separations i.e., typically much greater that the interocular separation of about 65 mm
- One or more of the cameras serve as reference or “witness” cameras that allow for accurate range data determination using photogrammetry techniques to calculate the object distances.
- the witness cameras are synchronized with respect to the shutter of the main cine camera.
- CG visual effect elements is simplified by the present disclosure because the systems and methods result in the creation of a virtual 3D geometry of the location and allows for dual virtual cameras to be placed substantially arbitrarily in the virtual 3D space.
- FIG. 1 is a generalized schematic diagram of the 3D stereo imaging system of the present disclosure
- FIG. 2 is a close-up, top-down schematic diagram of an example camera system that includes three cameras, namely a left camera, a center camera and a right camera;
- FIG. 3 is a close-up, top-down view of the overlapping fields of view FOV of the three cameras of the camera system shown in FIG. 2 , and illustrates a cross-section view of the volume of space covered by the camera system, wherein a scene that includes a number of different objects within the volume;
- FIG. 4 is a top-down view and FIG. 5A is a front-on view of an example camera support device in the form of a rail system configured to allow the cameras of the camera system to pan horizontally (parallel view) and to have a select convergence angle ⁇ C ;
- FIG. 5B is similar to FIG. 5A and illustrates an example embodiment that includes two cine cameras configured as a stereo rig to capture conventional 3D stereo images of a scene;
- FIG. 6 is a front-on view that illustrates an example camera mounting device that supports the camera support device of FIG. 4 and FIG. 5 ;
- FIG. 7A is similar to FIG. 6 except illustrating an example where the camera mounting device includes separate mounts for each camera;
- FIG. 7B is a plan view of the camera layout of FIG. 7A , better illustrating that the cameras can be spaced relatively far apart from one another relative to conventional stereo imaging that employs two cinematography cameras on a stereo camera rig;
- FIG. 8 is a front-on view of the camera system and illustrates an example embodiment where the camera system includes an illumination system configured to illuminate objects in the scene;
- FIG. 9 is a schematic diagram of an example camera system light source that generates textured light and includes a laser and a diffraction grating to generated multiple diffracted light beams;
- FIG. 10 is a schematic diagram of an example camera registration system
- FIG. 11A is a rear-end view of the camera registration system of FIG. 10 showing registration indicia on the back surface of the camera support device that allows for a registration camera to capture the registration information for the cameras in the camera system;
- FIG. 11B is a top-down view of the camera registration system of FIG. 10 showing how the registration camera captures an image of the camera support device to capture the camera system registration information;
- FIG. 12 shows a flow diagram that sets forth an example of an initial post-processing method for creating corrected viewable image data
- FIG. 13A shows a flow diagram that sets forth an example post-processing method for converting the 2D corrected viewable image data formed by carrying the steps of the flow diagram of FIG. 11 into 3D stereo image data;
- FIG. 13B shows a flow diagram that sets forth an example method of performing image rectification for 2D images of a scene captured with different cameras from different positions and orientations;
- FIG. 14A is an example black and white 2D image that includes a scene with a objects in the form of a tree, a person and wall sections that reside at a variety of distances from the camera;
- FIG. 14B is an example disparity map for the 2D image of FIG. 14A , illustrating distance information relative to the camera in the form of a corresponding gray scale, with lighter gray scale being associated with closer distances and darker gray scale being associated with farther distances;
- FIG. 14C is an example point cloud formed from the disparity map of FIG. 14B ;
- FIG. 14D is a close-up view of the point cloud of FIG. 14C showing more detail of the man in the point cloud;
- FIG. 15A depict an example disparity map for an object in the form of a bas relief sculpture
- FIG. 15B depicts example the point cloud data based for the bas relief sculpture on the disparity map of FIG. 15A ;
- FIG. 16 shows details of the region A identified in FIG. 15B and shows how the point cloud is formed by points, with region B showing a close-up portion of region A that schematically illustrates the points making up the point cloud;
- FIG. 17 is a flow diagram that sets forth an example method of forming a 3D image or a 3D movie from the point cloud data and the camera positions used to capture the image data used to form the point cloud;
- FIG. 18 is a diagram that schematically illustrates the method step of defining virtual camera positions with respect to a point cloud formed using the methods described herein, and also schematically illustrates the addition to the point cloud of a visual effect.
- the present disclosure relates generally to creating three-dimensional (3D) stereo images from two-dimensional (2D) photography, and in particular to systems and methods for 2D image capture and post-processing for 3D stereo imaging.
- the disclosure sets forth an overview of the 3D stereo imaging system and its components. The various components of the 3D stereo imaging system are then described in greater detail. Then, a variety of embodiments of the methods of the disclosure based on the operation of the 3D stereo imaging system are described.
- the terms “right” and “left” as applied to the witness cameras are relative to the 3D imaging system and its view of the scene.
- FIG. 1 is a generalized schematic diagram of the 3D stereo imaging system (“3D system”) 10 of the present disclosure.
- 3D system 10 includes a camera system 100 that has two or more cameras 102 spaced apart from its adjacent camera by respective distances DS, which need not be the same.
- FIG. 2 is a close-up, top-down schematic diagram of an example camera system 100 that includes three cameras 102 , namely a left camera 102 L, a center camera 102 C and a right camera 102 R (Cartesian coordinates are shown for reference).
- one of cameras 102 is a cinemagraphic camera (“cine camera” or alternatively, “motion picture camera” or “production camera”) while the other one or more cameras are reference or “witness” cameras, e.g., cameras 102 L and 102 R.
- the cine camera 102 C (assumed hereinbelow to be center camera 102 C for convenience) is thus used to capture images of the scene as contemplated by a director while the witness cameras are used to capture the 3D information relative to the cine camera.
- Cameras 102 are operably connected to one another via communication lines 104 , which may be optical or electrical or wireless, for example. Cameras 102 may also be mechanically connected to one another via a camera support device 110 .
- Each camera 102 includes an imaging lens 120 with an associated field of view FOV, an image sensor 126 , camera electronics 130 and a computer-readable medium 132 , which in an example is in the form of a removable memory storage device (“removable memory”), such as a memory CF card, flash memory stick, CD ROM, etc.
- Computer-readable medium 132 is referred to below as removable memory for the sake of description. The operation of cameras 102 is synchronized as described in greater detail below.
- 3D system 10 further includes a data reading device 140 for reading data stored in removable memories 132 .
- data reading device 140 includes a computer 150 .
- Computer 150 includes a port 152 , a processor 154 and a memory module 156 .
- Port 152 is configured to receive removable memory 132 and read and store the contents of the removable memory in memory module 156 .
- 3D system 10 also includes a removable-memory storage unit 180 where removable memories 152 are physically stored after use, as described below.
- An example removable-memory storage device 180 is a secure drawer or safe.
- 3D system 10 further includes a primary data storage unit 200 such as a server, and an optional back-up or secondary storage unit 200 ′, such as another server.
- primary data storage unit 200 and secondary storage unit 200 ′ are operably linked via a communication link 202 , and further in the example are configured to store a petabyte or more of data.
- secondary storage unit 200 ′ is mirrored to primary storage unit 200 .
- Primary data storage unit 200 optionally includes an antenna system 158 for wireless EM data communication with computer 150 .
- Primary data storage unit 200 may be operably linked to computer 150 via Ethernet cable 160 .
- cameras 102 each capture a raw 2D image, or a sequence of raw images of a scene over a volume 250 (see FIG. 3 ).
- the captured raw images are filtered (e.g., pre-Bayer filtered) by respective camera electronics 126 .
- the filtered images are then time-coded, frame-coded and camera-coded and stored in their respective removable memories 132 .
- Removable memories 132 are removed from their respective cameras 102 (e.g., when the removable memories are full or nearly full) and with reference to arrow AR 1 , are transferred to the location of computer 150 . Each removable memory 132 is then inserted into port 152 and computer 150 reads the filtered image data stored therein, and stores the filtered image data in memory module 156 . With reference to arrow AR 2 , the removable memories are then delivered to removable-memory storage unit 180 for storage.
- the filtered image data in memory module 156 of computer 150 is pre-processed by processor 154 to create viewable images.
- the viewable images have the largest possible dynamic range and are color-corrected (neutral).
- the pre-processing by processor 154 may also be performed to enhance the captured scene details. The pre-processing may be performed later on in the same computer 150 or on a different computer.
- the viewable image data stored in memory module 156 is transferred to primary data storage unit 200 , e.g., via Ethernet cable 160 or via wireless EM antennae systems 158 and wireless signals 159 .
- Primary data storage unit 200 optionally communicates with secondary data storage unit 200 ′ to mirror the storage of the viewable image data for redundancy.
- the viewable image data include 2D range data from each of the one or more witness cameras 102 and also includes 2D scene data (cinemagraphic images) from the cine camera 102 C.
- the viewable image data stored in data storage unit 200 includes the full volumetric (3D) information of the captured scene within volume 250 , which allows for robust post-processing of the viewable images.
- the viewable image data allows for substantially arbitrary virtual camera placement in the captured scene during post-processing, which greatly enhances the ability to create visual effects using CG animation in forming 3D stereo images, including 3D stereo movies.
- computer 150 is programmable with instructions (e.g., software) embodied in a computer-readable medium to cause the computer to perform the various functions and carry out various aspects of the methods described herein in greater detail below.
- computer 150 is a dedicated computer used only to read filtered image data from removable memories 132 , pre-process the filtered data to form viewable image data, and transfer the viewable image data to primary data storage unit 200 .
- Non-volatile media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) operating as a server platform.
- Volatile media include dynamic memory, such as main memory of a computer platform.
- Physical transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a bus within a computer system.
- Carrier-wave transmission media can take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications.
- computer 150 includes an antenna system 158 for wireless electromagnetic (EM) data transmission via EM waves 159 , and also in an example includes an Ethernet cable 160 for wired data transmission via EM data signals S 1 .
- Computer-readable media therefore include, for example: a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, less commonly used media such as punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, so-called “thumb drives,” any other memory chip or cartridge, a carrier-wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer can read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.
- each camera 102 in camera system 100 has a field of view FOV centered on its optical axis, with left camera 102 L having a field of view FOV-L and an optical axis AL, center camera 102 C having a field of view FOV-C and an optical axis AC, and right camera 102 R having a field of view 102 R and an optical axis AR.
- FIG. 3 is a close-up view of the overlapping fields of view FOV of the three cameras 102 of camera system 100 , and illustrates a top-down view of a scene that includes example objects 240 in the form of a picnic table 2401 , a garbage can 240 G, a barbeque 240 B, a shrub 2405 and a block wall 240 W.
- the fields of view FOV are three-dimensional frustums, each covering a volume of space.
- the field of view FOV in the X-Z plane need not be the same in the Y-Z plane, and the fields of view of the cameras can be different.
- Cameras 102 are configured so that their fields of view FOV all start to overlap at some distance D 1 from center camera 102 C, with the overlapping fields of view defining a common volume 250 of space associated with a scene being imaged by each camera from a different orientation (i.e., convergence angle).
- a cross-section of the volume 250 is shown in FIG. 3 .
- Outer boundary 260 is generally defined by the camera spacings DSL and DSR and convergence angles ⁇ CL and ⁇ CR and represents the limit at which accurate 3D volumetric information can be obtained with cameras 102 . Having such an outer boundary on volume 250 usually does not present a problem for most 3D stereo imaging applications because the emphasis of the 3D effect is usually on the objects within the foregrounds of the volume while distant objects are generally treated as background.
- camera spacings DS are defined by the object-to-camera distance.
- a good ratio of the object-to-camera distance DC as compared to the camera spacing DS i.e., DS/DC
- DS/DC A good ratio of the object-to-camera distance DC as compared to the camera spacing DS
- DS/DC the ratio of the object-to-camera distance DC as compared to the camera spacing DS
- the distance between human eyes is about 68 mm
- the distance normally used between conventional stereo cameras is sometimes much less. This is because the stereographer is often asked to compromise between creating good stereo separation of objects in a scene and producing comfortable viewing stereo without causing the many artifacts that come with an extended stereo effect.
- a good rule of thumb for a normal the interaxial distance is between 19 mm and 38 mm. As discussed below, this base distance is much too small to calculate accurate range or point cloud data. This is important in creating the depth map used to convert 2D to 3D, but also for being able to combine visual effects and CG character animation in native stereo plates.
- the use of one or more witness cameras with a stereo production camera rig, but with the witness cameras being spaced apart from the conventional stereo production rig allows for creating 3D point cloud data that can be used for post-production visual effects and animation and for motion capture.
- the cine camera as described herein can, in one example, be a conventional stereo production rig having the aforementioned typical interaxial distance.
- cameras 102 The operation of cameras 102 is synchronized via camera electronics 130 .
- one of the cameras such as center camera 102 C, includes a clock 131 and an electronic interface as part of its camera electronics 130 , with the clock and electronic interface being used to synchronize the frames of the cameras.
- witness cameras 102 are shutter-phased to cine camera 102 C. In examples, global or rolling shutter can be used.
- the point of trigger (exposure) for witness cameras 102 is determined when the shutter of cine camera 102 C is at the center of the “effective” shutter open position.
- Witness cameras 102 preferably shoot with the shortest possible shutter settings to minimize “motion blur,” and overall exposure is controlled by varying the stop in lens 120 or by adjusting the system gain via camera electronics 130 .
- the use of time-code slates and the recording of time code as metadata make it possible to synchronize and identify the corresponding cine images captured by cine camera 102 C and the reference images captured by witness cameras 102 .
- one or more of cameras 102 are high-resolution color machine-vision cameras. As discussed above, cameras 102 need not be the same, though for cameras having image sensors 126 with different pixel sizes, rectification of the pixel sizes should be performed as part of the post-processing for optimum results.
- An example camera 102 is a digital camera suitable for both still and motion pictures.
- An example camera 102 has a 5K (e.g., 5120 pixels by 2700 pixels) or greater image sensor 126 , with lens 120 having 18 stops of dynamic range and a rolling shutter.
- a suitable witness camera 102 includes the EPIC or SCARLET cameras available from Red Camera Company, Lake Forest, Calif. Such cameras can be used for just the one or more witness cameras 102 , just the cine camera 102 C, or can be used for both witness and cine cameras so that all of the cameras are the same. It is anticipated that the digital technology for cameras 102 will continue to develop and improve, e.g., will have increasing numbers of pixels, better lenses, more robust camera electronics, etc.
- camera electronics 130 includes wavelet compression to optimize image data storage in removable memory 132 .
- Camera electronics 130 also optionally includes filtering capability for filtering the raw captured image so that a filtered image is stored in removable memory 132 .
- a removal memory 132 that provides 64 gigabits (Gb) of storage can store about 6 minutes worth of filtered images.
- image sensors 130 of witness cameras 102 are sensitive to near-IR wavelengths (850 nm to 940 nm) so that a potentially dark scene can be illuminated with near infrared light.
- cine camera 102 C can be protected from viewing the IR illumination through the use of an IR rejection filter, such as the Tiffen IR ND Filter.
- witness cameras 102 are equipped with a variety of matched focal length lenses 120 W that match the horizontal view of the cine lenses 120 C.
- the choice of lens 120 W for the witness cameras 102 is determined by the choice of lens for the cine camera 102 , subject the criteria that the horizontal field of view witness lens 120 W is greater than or equal to the horizontal field of view of the cine lens 120 C.
- FIG. 4 is a top-down view
- FIG. 5A is a front-on view of an example camera support device 110 in the form of a rail system configured to allow cameras 102 to pan horizontally (parallel view), and to have select convergence angles ⁇ C (e.g., ⁇ CL and ⁇ CL for the three-camera configuration).
- An example camera support device 110 includes a rail 300 to which cameras 102 can be mounted.
- Witness cameras 102 each include a mounting device 310 that allows for the witness cameras to be positioned along rail 300 and fixed thereto at desired positions, i.e., at a select distance DS from cine camera 102 C.
- rail 300 includes a track 302 sized to accommodate a complimentary groove (not shown) in mounting device 310 .
- Mounting device also is configured to allow for each corresponding witness camera 102 to rotate about its Y-axis, i.e., about axis ALY for left witness camera 102 L and about axis ARY for right witness camera 102 R.
- camera support device 310 is configured so that the convergence angles ⁇ C can be set to within a few arc-seconds of a desired or target convergence angle.
- the convergence angles ⁇ C can be manually set by the operator depending on the 3D requirements of the scene.
- a high-precision rotation motor is used to automatically set the convergence angle.
- the convergence angle ⁇ C for each witness camera 102 is adjustable in small angular increments, e.g., 5 degree increments, using for example a precise mechanical pin registration system for accurate and repeatable setting of the convergence angle.
- mounting device 310 includes a plate 320 having holes 322 formed therein that locks the mounting device to rail 300 via pins 324 that engage corresponding holes 326 in rail 300 .
- Holes 322 are configured to accurately correspond to select convergence angles ⁇ C .
- holes 326 are also formed at precise distances from cine camera 102 C.
- the length L of rail 300 can be any practical length, with 1 meter to 6 meters being an exemplary range of lengths. In an example embodiment, rail 300 is telescopic so that length L can be adjusted.
- rail 300 is configured to that it experiences minimum bending, flexing and vibration under operating conditions, with the result that the relative positions and orientations (convergence angles) of cameras 102 are precisely maintained at all times with substantially no relative movement or vibration during the operation of 3D system 10 .
- Rail 300 is preferably made of one or more light-weight materials, with aluminum and carbon fiber being two exemplary materials.
- light-weight materials with aluminum and carbon fiber being two exemplary materials.
- FIG. 5B is similar to FIG. 5A and illustrates an example embodiment where 3D system includes two witness cameras 102 L and 102 R, and two cine cameras 102 CL and 102 CR configured as a stereo camera rig 1025 .
- This configuration allows for the stereo camera rig 102 S to obtain conventional 3D stereo images while the witness cameras 102 L and 102 R capture the spatial data for the scene necessary for forming the point cloud or polygonal mesh and providing robust 3D stereo post-processing.
- the stereo rig 102 S may also include a beam-splitter arrangement that allows for the two cinematography images to be captured at positions separated by the interocular distance of about 68 mm.
- FIG. 6 illustrates an example configuration where camera mounting device 310 includes a base support 350 , such as a tripod stand or like base support structure. Note, however, that cameras 102 need not be supported by a common camera mounting device 310 . Thus, with reference to FIG. 7 , an example embodiment of camera system 100 has each camera 102 supported by its own base support so that camera support device 110 is not needed. This configuration is workable as long as the relative positions and orientations of cameras 102 are accurately known.
- the relative positions and orientation of cameras 102 can be established, for example, using laser-based position and alignment devices 352 , which can be interfaced with camera electronics 130 . Laser beams 353 associated with laser-based position and alignment devices 352 are shown as dotted lines in FIG. 7 .
- Wireless communication can also be used between the camera electronics 130 , as illustrated by wireless signals 131 .
- the cameras can be quite far apart, as better illustrated in the plan view of FIG. 7B .
- 3D system 10 of FIGS. 7A and 7B supports an embodiment where cine camera 102 C is “free-roaming,” meaning that it can be positioned independently of the witness cameras 102 L and 102 R. This allows for generating point cloud data (discussed in greater detail below) for a completely separate 2D or 3D cine camera. This point cloud can be referenced back to the free roaming cine camera and used by visual effects to accurately position additional CG characters and environments into complex 3D stereo scenes. Cameras 102 would be in wireless communication to sync up all camera shutters. Post-processing as described in detail below is then used to connect the cine camera view to the point cloud data, which is taken from a different perspective.
- cameras 102 can be in wireless communication to sync up all camera shutters. Post-processing, as described in detail below, is then used to connect the cine camera view to the point cloud data, which is taken from a different perspective. Since the witness Cameras 102 L and 102 R are shutter-phased with the production 3D stereo cameras, the creation of volumetric point cloud data (which is created on a frame-by-frame basis) is like having a 3D scan of the scene (e.g., the set and actors therein) per frame. This animating point cloud not only records accurately the 3-space of fixed architecture, but also records the movement and position of actors and objects in motion within the scene.
- the production 3D stereo cameras are than virtually tracked and locked to this point cloud on a per frame basis using a 3D computer software package, such as MAYA® point-cloud software, available from Autodesk, San Rafael, Calif.
- a 3D computer software package such as MAYA® point-cloud software, available from Autodesk, San Rafael, Calif.
- the six axes of tracked translations of the camera are then key-framed into a virtual facsimile of the production 3D camera rig, with the view of these virtual cameras locked to the animated point cloud.
- This animated point cloud is extremely helpful to the animation “Layout Department,” which is involved in adding additional CG elements to the live action stereo plates, such as CG characters and enhancing the live action environment with additional CG sets and architecture.
- the point cloud gives precise 3D positions of all objects in the scene, thus allowing the CG additions to be made to the live action stereo plate in an efficient and accurate manner.
- the animated point cloud is a robust solution because each point in the point cloud can be loaded virtually into the 3D volume of the motion capture stage. This point cloud allows the operators and actors to reference their position and timings to the animated point cloud, thereby recreating the original objects in the photographed scene.
- the virtual production camera with its key-framed positional translations can also be imported into the motion capture volume. This virtual camera allows the motion capture operators to view the motion capture actors through the same effective virtual lens as what was used to shoot the original live action background.
- An additional use of the point cloud in a motion capture volume is to convert this point cloud into a “bird's eye” view from the top to be projected in scale on the floor of the motion capture volume.
- An orthographic view from above of the point cloud can be graphically simplified into basic animated artwork that represents critical objects or timings of objects in a scene that actors must accurately interact with.
- This simplified graphic representation of the point cloud can be projected with a video projector onto the floor of the motion capture volume.
- This graphic representation of the point cloud data can perform as an interactive guide for the motion capture actors to help position their actions for both position and timings to photographed elements critical to the scene.
- FIG. 8 illustrates an example embodiment where camera system 100 includes an illumination system 400 .
- illumination system 400 includes one or more light sources 402 that in one example emit light 404 in the near IR wavelength range of 850 nm to 940 nm.
- IR light source 402 include light-emitting diodes (LEDs), laser diodes or near-IR lasers.
- illumination system 400 is configured to illuminate the scene in volume 250 when production lighting is less than optimal.
- IR illumination system 400 is configured so that the amount of illumination 404 from light sources 402 can be controlled.
- light source 402 emit IR light 404 and the amount of IR light level is balanced with the production lighting (which is typically white light) used for the particular scene.
- illumination system 400 is configured to generate “textured light” 404 that includes a spatial intensity variation that result in a light pattern when an object 240 in scene 242 is illuminated. Textured light 404 is used to highlight objects 240 that might otherwise lack sufficient texture to be able to determine the objects range and the range of features within the object.
- FIG. 9 illustrates an example light source 402 that generates textured light 404 .
- Light source 402 of FIG. 9 includes a laser 420 and a diffraction grating 424 arranged to receive light 422 from the laser and diffract light 422 to form textured light 404 comprising multiple diffracted light beams.
- laser 420 operates in the near IR so that textured light 404 is visible to witness cameras 102 but is made invisible to the cine camera 102 C using a IR-rejection filter.
- laser 420 operates in a narrow band of visible light, e.g., 589 nm (yellow-orange), which is visible to witness cameras 102 but is blocked from being detected by the cine camera using a correspond narrow-band rejection filter (e.g., a Didymium-coated filter).
- a correspond narrow-band rejection filter e.g., a Didymium-coated filter.
- Such narrow-band filtering in the visible does not substantially affect the ability of cine camera 102 C to record live-action color 2D image.
- lasers 420 are pulsed (e.g., strobed) in synchronization with the shuttering of the witness camera shutters and cine camera shutters.
- the laser pulse duration is preferably kept as short as possible to reduce any distracting effects to the case and crew involved with imaging scene 240 .
- light sources 402 are synchronized with the same camera electronics 130 that control the operation of the cine camera 102 C and witness cameras 102 .
- an aspect of the disclosure includes measuring one or more camera parameters for cameras 102 and performing a camera calibration so that differences between cameras that could impact the post-processing of viewable images is reduced and preferably substantially eliminated.
- Example camera parameters include lens distortion, sensor position, principal point and nodal point locations, and F-stop vs. focal length variations.
- the camera parameters can be obtained from the camera manufacturer for the given lenses.
- the difference in camera parameters between cameras 102 can be reduced or eliminated during the post-processing steps since the viewable images from the different cameras are still available for processing, and the measured camera parameters directly relate to the captured viewable images for the particular camera.
- each camera 102 is mounted on the camera support device and then the camera spatial coordinates (x,y,z) and the corresponding camera angular coordinates ( ⁇ , ⁇ , ⁇ ) (i.e., rotation angles about x, y and z, respectively) are established for each camera relative to a reference (“world”) coordinate system.
- FIG. 10 is a schematic diagram of an example camera registration system 500 .
- Camera registration system includes camera system 100 with cameras 102 mounted on camera support device 110 .
- a jig 301 that includes reference objects 502 is arranged relative to camera system 100 .
- Reference objects 502 have select heights, sizes and positions.
- Test images are captured by each camera 102 and the camera orientation established relative to a reference coordinate system, which is shown based at left witness camera 102 by way of example.
- Camera parameters such as distortion can also be measured using jig 500 .
- FIG. 11A is a rear-end view of the camera registration system of FIG. 10 showing registration indicia 320 on the back surface 301 of camera support device 110 .
- FIG. 11B is a top-down view of the camera registration system of FIG. 11A and shows how a registration camera 321 with a field of view FOV-A captures an image of camera support device 110 to collect camera registration information.
- FIG. 12 shows a flow diagram 550 that sets forth an example of an initial post-processing method for creating corrected viewable image data.
- the method comprises a number of steps 551 through 555 .
- step 551 the viewable image data is accessed, and in step 552 the camera calibration (based on the measured camera parameters) is applied to the viewable image data to reduce or eliminate differences in the viewable images due to differences between cameras 102 .
- step 553 the camera registration is applied to the camera-calibrated viewable image data formed in step 552 to reduce or eliminate camera registration errors between cameras 102 .
- Step 554 is an optional step of adding information (e.g., notes, comments, etc.) to the various image frames.
- step 555 a first output file denoted FILE-1.ccc that includes corrected viewable image data.
- the corrected viewable image data is in the form of frames F for each of the witness cameras 102 (e.g., frames F WL and F WR for the left and right witness cameras 102 L and 102 R) and the cine camera 102 C (e.g., frames F C ).
- the corrected viewable image data for each frame F is 2D in spatial coordinates and also includes R-G-B color information, i.e., F(x,y;RGB), with x and y corresponding to the pixel spatial coordinates of the corresponding image sensor 130 .
- the various frames F can be stored in a first output file FILE 1 , which in an example is in “curtain call native bitmap graphic format,” i.e., FILE-1.ccc.
- FIG. 13A shows a flow diagram 580 that sets forth an example post-processing method (with steps 581 through 585 ) for converting the 2D corrected viewable image data in first output FILE-1.ccc into 3D stereo image data.
- the first output file FILE-1.ccc is accessed and in step 582 image rectification is performed.
- Image rectification involves ensuring that each of the images for each frame F has substantially the same format. For example, if cameras 102 reside in different planes, the images (frames) will be skewed relative to one another. The process of image rectification seeks to remove this skew, or at least reduce it so that it is insubstantial. Also, slight magnification differences between the cameras can result in differences in the size of the images.
- the image rectification process ensures that the image sizes are the same to within sub-pixel resolution. Similarly, differences in the images caused by slight differences in the fields of view of the cameras 102 can be reduced or eliminated during the image rectification process. Thus, image rectification is directed to reducing or eliminating one or more differences in the captured images so that all of the images have substantially the same fundamental format (shape, size, etc.).
- An example image rectification process is provided in the flow diagram 582 of FIG. 13B , discussed in greater detail below.
- stereo matching is performed on the image-rectified, corrected viewable image data formed in 582 .
- Stereo matching involves correlating the pixel information between the various images, e.g., matching the left witness camera frames F wL (x L ,y L ;RGB), the right witness camera frames F WR (x R ,y R ;RGB) and the cine camera frames F C (x C ,y C ;RGB) so that the pixels for the different frames are correlated.
- step 584 the witness camera viewable image data is related to the cine camera viewable image data from 583. This is accomplished using a processed called “roto.”
- FIG. 14A is an example 2D black and white image 600 from cine camera 102 C.
- 2D image 600 includes objects 240 - 1 and 240 - 2 (portions of a brick wall in the background), object 240 - 3 (a person) and object 240 - 4 (a portion of a tree in the foreground).
- Each of these objects has been rotoed (i.e., selected as an ROI), as indicated by the dark-line outline of each of these objects.
- Only the selected ROIs of the viewable images are processed while the remaining portions of the viewable images are treated as background.
- the roto process can be done manually for an initial frame and then carried through automatically for multiple frames.
- the process associated with step 584 includes performing a roto on the cine camera viewable images to define select ROIs to create a rotoed cine camera viewable image RC.
- This rotoed viewable image RC is then transformed to the left witness camera 102 L to form a left witness camera rotoed image RL.
- This rotoed image is then transformed to the right witness camera to form a left witness camera rotoed image RR.
- This process results in three rotoed viewable images RL, RC and RR having corresponding or matched ROIs, where the rotoed portions (ROIs) match even though the viewing angles for cameras 102 L, 102 C and 102 R were different.
- the rotoed viewable images formed in step 584 are used to create a “disparity map” 600 that associates gray-scale pixel intensities in the rotoed viewable images with distance (z) from the cine camera 102 C.
- FIG. 14B is the corresponding disparity map 610 to 2D image 600 of FIG. 14A and shows in gray scale the corresponding objects, which are identified in the disparity map as 240 ′- 1 through 240 ′- 4 .
- the remaining portion of the disparity map has a uniform gray background.
- the gray scale is such that the lighter the color, the closer the object is to camera 120 C.
- the tree object 240 - 4 is in the foreground
- the wall objects 240 - 1 and 240 - 2 are in the background
- the person object 240 - 3 is at an intermediate distance between the wall object and the tree object.
- the disparity map is formed from the rotoed images (ROIs) in one example by comparing the difference ( ⁇ x, ⁇ y) for each pixel in left witness camera frames F WL and the right witness camera frames F WR and associating the difference with a distance z.
- the gray-scale intensities in the image are correlated to distance z, with objects closer to the camera having a lighter gray-scale level than objects farther away, as illustrated in FIG. 14B .
- disparity map 610 can have a very large gray-scale dynamic range, e.g., 0 to 65,000. This allows for the disparity map to include a tremendous amount of distance information for the various objects (including portions of objects) in the scene for a given frame.
- the gray scale for two or more disparity map frames is normalized so that a sequence of frames has a common reference for distance information, i.e., a certain gray scale represents the same distance in one frame as it does in another frame. This is particularly useful when the gray scale can have a large dynamic range such as the aforementioned 0 to 65,000 range.
- FIG. 14C is an example point cloud formed from the disparity map of FIG. 14B .
- FIG. 14D is a close-up view of the point cloud of FIG. 14C showing more detail of the man 240 ′- 3 in the point cloud.
- FIG. 15A depicts an example disparity map 240 DM for an object in the form of a has relief sculpture.
- 3D image data is calculated, whereby the 2D images from cine camera 102 C are embedded with the range data at sub-pixel accuracy, creating a voxel-based image.
- the 3D image data can be expressed as a “point cloud” P(x,y,z), where each element in the point cloud is a “pixol” representing a point in 3D space.
- the point cloud P is typically comprised of thousands of points.
- FIG. 15B An example point cloud 240 PC formed from the has relief sculpture disparity map 240 DM of FIG. 15A is shown in FIG. 15B .
- a region A of point cloud 240 P is denoted by a dashed-line rectangle.
- FIG. 16 is a close-up view of region A that shows point cloud 240 P being comprised of thousands of points 241 in the point cloud.
- a close-up inset B shows a magnified small region of region A that more clearly shows points 241 . Note that point cloud 240 P looks remarkably like an actual image or picture.
- the 3D image data can also be expressed as a polygonal mesh PM(x,y,z;RGB), wherein each element in the mesh is a “voxol” defined by three spatial coordinates and corresponding RGB values.
- the 3D image data is calculated from the disparity map using triangulation techniques known in the art of photogrammetry (i.e., photogrammetric triangulation).
- This process solves the position of the six camera parameters (i.e., spatial coordinates (x,y,z) and angular coordinates ( ⁇ , ⁇ , ⁇ ) using the calibrated camera coefficients to compute the 3D coordinates. This process is described in greater detail below in connection with the image rectification process.
- step 587 the 3D image data is outputted to a file, e.g., FILE-3D, which can be in one of the commonly used graphics formats such as ASCII.
- FILE-3D includes either the point cloud data or the polygonal mesh data.
- step 582 - 1 involves accessing the left and right input images (frames) F WL an F WR from first output file FILE-1.ccc, as well as accessing the camera calibration data.
- the rotation angles for the left and right cameras are respectively defined by M 1 and M 2 .
- the rectification rotation matrix R for the image pair (frame pair) can be calculated as follows. First, let e 1 , e 2 and e 3 be three 3D vectors, given by as follows:
- e 1 e 1 /(magnitude of e 1 )
- e 2 e 2 /(magnitude of e 2 )
- e 3 e 3 /(magnitude of e 3 )
- the rectification rotation matrix R is formed by column vectors e 1 , e 2 and e 3 .
- step 582 - 3 the average focal length AFL is calculated as follows:
- ALF (( CC 1 /PS 1)+( CC 2 /PS 2))*(size percentage)/2
- CC 1 and CC 2 are the camera constants for the first and second cameras and PS 1 and PS 2 are the pixel sizes for the images for the first and second cameras, and the size percentage.
- the size percentage can be selected to be 100% of the actual image size or any other percentage, usually smaller (e.g., 50%), which makes the point cloud smaller and more dense.
- step 582 - 4 the average principal point APP is calculated as follows:
- PP 1 and PP 2 are the principal points for the first and second cameras.
- step 582 - 6 for each input image and for each pixel (x, y) in the input image, the image point relative to project center (i.e., principal point) and with respect to the image coordinates using camera calibration parameters A, B, K 1 , K 2 , P 1 , P 2 is calculated as follows:
- x′′ x′+A*y′+r*x′+P 2 *t+P 1*( s+ 2 *x′ ⁇ 2)
- y′′ y′+B*x′+r*y′+P 1 *t+P 2*( s+ 2 *y′ ⁇ 2).
- APPx and APPy are the x and y coordinates of the average principle point.
- step 582 - 9 involves finding the position, size and color of the rectified left and right ROIs.
- This involves an inverse-mapping process for each rectified image and for each pixel (x, y) in the rectified ROI in the rectified image.
- the rectified image is inverse-mapped back to the input image by iteratively moving towards the closest pixel in input image, and then finding the sub-pixel ratios.
- the closest pixel and sub-pixel ratios are used to determine the color of that pixel.
- the output comprises left and right rectified images with ROIs, with color values (RGB) at each pixel, namely voxels (x,y;RGB).
- FIG. 17 sets forth a flow diagram of an example method of forming 3D stereo image or movie.
- step 601 the point cloud data and the camera positions used to obtain the point cloud data are provided.
- step 602 first and second virtual camera positions are selected, thereby defining a 3D stereo pair. This replicates a stereo rig and allows for arbitrary 3D virtual filming of the scene as a post-processing step.
- step 602 is schematically represented in FIG. 18 , which shows a point cloud PC and first and second virtual cameras CV 1 and CV 2 .
- a second technique to create a 3D stereo separate pair (Left and Right eyes) from a 2D image is to horizontally displace (warp) the pixels in a horizontal direction by a specific amount. This amount differs between the two left and right eyes, creating the illusion of stereo depth. The amount of horizontal displacement is informed by both the range data of the corresponding pixol and the stereo volume required for the creative specifics of the shot.
- Step 603 includes the optional step of combining the point cloud with at least one visual effect to enhance the 3D stereo image or 3D stereo movie formed as described above.
- Example visual effects include computer graphics, computer animation, hand-drawn animation, computer-generated environments, computer-generated or hand-drawn imagery, computer-generated objects, and the like.
- a visual effect means any additional information added to the point cloud.
- FIG. 18 illustrates schematically the addition to point cloud PC of a visual effect VE in the form of an animated dragon by way of a simple example.
- the 3D stereo image or movie is created using the selected positions of the first and second virtual cameras.
- 3D stereo images are created (left and right eyes) from 2D imagery using one of two primary techniques.
- the first method is to re-create a virtual version of a two-camera 3D stereo rig in CG.
- This virtual 3D stereo rig has the same controls at its physical (real) counterpart for creating stereo photography, i.e., convergence and setting of the interaxial distance between virtual cameras.
- the 2D imagery is projected as a texture map onto a polygonal geometry mesh, which is created from the point cloud data embedded in each frame. This texture map of the 2D live action is physically displaced and projected onto this geometry.
- the 2D imagery has now been given shape by the geometry, which is a true 3D view of the scene as photographed from point of view of the production camera and one or more attendant witness cameras.
- the virtual 3D stereo rig is then used to virtually re-photograph this frame, thereby producing a discreet left and right eye element for stereo viewing.
- a second method is to use the depth information stored in each pixol, which contains the X, Y, Z information for the voxel-based image.
- a 3D stereo image can be created from a 2D image by re-positioning the pixels of the image in a horizontal manner, thereby creating two separate images (left & right eyes). The amount of horizontal displacement is informed by both the range data of the corresponding pixol and the stereo volume required for the creative specifics of the shot.
- forming the 3D stereo movie includes making one or more critical 3D-stereo-related decisions such as camera position, orientations, convergence angles, panning, etc., in post-production rather at the time of capturing the 2D images.
- an aspect of the disclosure includes relaxing the conditions under which the original 2D cinemagraphic images are captured. This simplifies the filming process for the movie on location and allows for much greater creative freedom to shape the movie in the post-production phase. This also saves substantial time and cost in making a 3D stereo movie.
- ROIs Regions of Interest
- Average focal length :((camera constant/pixel size of first image)+(camera constant/pixel size of first image))*(size percentage)/2
- x′′ :x′+A*y′+r*x′+P 2 *t+P 1*( s+ 2 *x′ ⁇ 2)
- y′′ :y′+B*x′+r*y′+P 1 *t+P 2*( s+ 2 *y′ ⁇ 2).
- x -coord :( x -coord of average principal point) ⁇ (average focal length)* x′′′/z′′′
- y -coord :( y -coord of average principal point) ⁇ (average focal length)* y′′′/z′′′
- Using 1.5 find the position and size of the rectified left and right ROIs 2. For each rectified image, for each pixel (x, y) in the rectified ROI on the image, inverse-map it back to the input image by iteratively moving towards the closest pixel in input image, and then finding the subpixel ratios. 3. For each rectified image, for each pixel (x,y), use the closest pixel and subpixel ratios to determine the color of that pixel.
- x -subpixel-coord : x +( x -coord of principal point)/(pixel size);
- y -subpixel-coord : y +( y -coord of principal point)/(pixel size);
Abstract
Systems and methods for 2D image and spatial data capture for 3D stereo imaging are disclosed. The system utilizes a cinematography camera and at least one reference or “witness” camera spaced apart from the cinematography camera at a distance much greater that the interocular separation to capture 2D images over an overlapping volume associated with a scene having one or more objects. The captured image date is post-processed to create a depth map, and a point cloud is created form the depth map. The robustness of the depth map and the point cloud allows for dual virtual cameras to be placed substantially arbitrarily in the resulting virtual 3D space, which greatly simplifies the addition of computer-generated graphics, animation and other special effects in cinemagraphic post-processing.
Description
- This application claims the benefit of priority under 35 U.S.C. §119(e) of U.S. Provisional Application Ser. No. 61/312,330, filed on Mar. 10, 2010, which application is incorporated by reference herein.
- The present disclosure relates generally to creating three-dimensional (3D) stereo images from two-dimensional (2D) photography, and in particular to systems and methods for 2D image capture and spatial data capture for 3D stereo imaging.
- The process of “converting” 2D photographed images into three-dimensional 3D stereo images (left eye and right eye pairs) for the motion picture and television industry is extremely labor intensive, time consuming, financially expensive, and has the added problem of being geometrically inaccurate to the original scene.
- Current technologies allow for the creation of 3D stereo imaging from 2D photography. However, the available 3D technologies do so through interpretive and creative means, or through system configurations that do not capture the true depth and geometry of the original environment using the 2D photography. For example, a conventional and popular technology used to make a 3D stereo image (or a 3D movie, which is a sequence of 3D stereo images) is to use two cameras separated by the typical human interocular distance DH (i.e., human eye spacing), which is assumed in the industry to be about 68 mm, though smaller distances DH are often used for reasons described below. The two cameras are then oriented (angled) so that their fields of view converge and overlap at a distance DS where the various objects in the scene being filmed are located. While this allows for creation of a 3D effect, the actual image-capture process does not collect a substantial amount of true 3D information for the given scene mainly because the interocular distance DH is too small relative to the distance DS. Said differently, the amount of spatial (3D) data captured by such an arrangement is far smaller than the actual 3D volume of the scene being imaged.
- This lack of accurate volumetric data and true 3D geometry provides significant problems and challenges when visual effects such as computer-generated elements need to be added to the photographed or filmed scenes. Complex visual effects scenes in 3D stereo that incorporate live action captured using a conventionally two-
camera 3D imaging system require critical stereo decisions to be made at the time of filming, such as the aforementioned convergence angles and interocular distances. - In addition, certain types of image post-processing require the full 3D spatial data to be captured to facilitate removing artifacts in the captures scene. For example, in movies it not uncommon to have to remove an unwanted jet contrail from an outdoor scene. In 2D movie (cinemagraphic) post-processing, this is a straightforward operation. However, in 3D movie post-processing, the jet contrail is also in 3D and thus is much more difficult to remove.
- Because of the limitations of present-
day 3D imaging technology, critical 3D-stereo-related decisions must made at the time of shooting rather than in post-production. However, it would be much preferred to be able to make such decisions in post-processing to optimize the camera positions relative to the surrounding cuts of the film. The addition of visual effects in the form of computer-graphics (CG) environments and CG characters into scenes that have been originally shot in 2D and converted into 3D stereo further complicates matters and poses great technical and financial challenges to visual effects post-production. - It would thus be of tremendous benefit to be able to reduce the time and expense presently associated with adding CG environments and CG characters to a 3D stereo movie as part of the movie post-processing.
- An aspect of the disclosure is a method of converting 2D images of a scene having therein at least one object to one or more 3D images of the scene. The method includes simultaneously capturing at least first and second 2D images of the scene from corresponding at least first and second cameras having respective camera positions and orientations measured relative to a reference coordinate system. The method also includes forming a disparity map from the at least first and second 2D images, wherein the disparity map has a gray scale that corresponds to distance information of the at least one object relative to the reference coordinate system. The method also includes forming from the disparity map a 3D point cloud P(x,y,z) representative of the at least one object, wherein the point cloud is configured to support first and second virtual cameras to create a stereo camera pair arrangeable in substantially arbitrary virtual locations.
- Another aspect of the disclosure is a method of forming a distance representation of a scene from 2D images of the scene. The method includes simultaneously capturing at least first and second 2D images of the scene from corresponding at least first and second cameras having respective camera positions and orientations measured relative to a reference coordinate system. The method also includes defining one or more regions of interest in the at least first and second 2D images. The method further includes associating differences between pixels in the at least first and second cameras with distances from a reference point, and assigning different gray-scale intensities to different ones of the distances.
- An aspect of the disclosure includes creating a depth map or “disparity map” using gray-scale variations to represent distances from the camera (or reference coordinates) associated with one or more objects in the scene. The process of creating the disparity map can be made semi-automatic using image data from multiple calibrated cameras. The range data in the disparity map is transformed to match the perspective and viewing frustum of the cinemagraphic (“cine”) camera. The 2D photographed image from the cine camera is embedded with the range data at sub-pixel accuracy to facilitate post-production to create an accurate and more continuous 3D stereo pair from this true 3D data.
- The range data can also be expressed as a “point cloud” P(x,y,z) representing spatial positions of thousands of points (pixols) in three-space that make up the 3D scene. In a particular example, the position and placement of every pixel in the cine camera in 3-space is computed with 16-bit color information.
- Alternatively, the range data can be expressed as a polygonal mesh PM(x,y,z;RGB) of spatial positions and the corresponding RGB color, which allows the 2D photographed images from the cine camera to be mapped directly on to the mesh in motion as texture. The textured 3D geometry can then be re-imaged with virtual cameras in post-production, setting the appropriate interocular distance and convergence values for the scene, thus creating a true left and a true right eye stereo pair for theatrical viewing.
- Multiple cameras with relatively large separations (i.e., typically much greater that the interocular separation of about 65 mm) are configured to capture 2D images over their respective field of views and over an overlapping volume associated with a scene having one or more objects. One or more of the cameras serve as reference or “witness” cameras that allow for accurate range data determination using photogrammetry techniques to calculate the object distances. The witness cameras are synchronized with respect to the shutter of the main cine camera.
- The addition of CG visual effect elements is simplified by the present disclosure because the systems and methods result in the creation of a virtual 3D geometry of the location and allows for dual virtual cameras to be placed substantially arbitrarily in the virtual 3D space.
- It is to be understood that both the foregoing general description and the following detailed description present embodiments of the disclosure, and are intended to provide an overview or framework for understanding the nature and character of the disclosure as it is claimed. The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated into and constitute a part of this specification. The drawings illustrate various embodiments of the disclosure and together with the description serve to explain the principles and operations of the disclosure.
-
FIG. 1 is a generalized schematic diagram of the 3D stereo imaging system of the present disclosure; -
FIG. 2 is a close-up, top-down schematic diagram of an example camera system that includes three cameras, namely a left camera, a center camera and a right camera; -
FIG. 3 is a close-up, top-down view of the overlapping fields of view FOV of the three cameras of the camera system shown inFIG. 2 , and illustrates a cross-section view of the volume of space covered by the camera system, wherein a scene that includes a number of different objects within the volume; -
FIG. 4 is a top-down view andFIG. 5A is a front-on view of an example camera support device in the form of a rail system configured to allow the cameras of the camera system to pan horizontally (parallel view) and to have a select convergence angle θC; -
FIG. 5B is similar toFIG. 5A and illustrates an example embodiment that includes two cine cameras configured as a stereo rig to capture conventional 3D stereo images of a scene; -
FIG. 6 is a front-on view that illustrates an example camera mounting device that supports the camera support device ofFIG. 4 andFIG. 5 ; -
FIG. 7A is similar toFIG. 6 except illustrating an example where the camera mounting device includes separate mounts for each camera; -
FIG. 7B is a plan view of the camera layout ofFIG. 7A , better illustrating that the cameras can be spaced relatively far apart from one another relative to conventional stereo imaging that employs two cinematography cameras on a stereo camera rig; -
FIG. 8 is a front-on view of the camera system and illustrates an example embodiment where the camera system includes an illumination system configured to illuminate objects in the scene; -
FIG. 9 is a schematic diagram of an example camera system light source that generates textured light and includes a laser and a diffraction grating to generated multiple diffracted light beams; -
FIG. 10 is a schematic diagram of an example camera registration system; -
FIG. 11A is a rear-end view of the camera registration system ofFIG. 10 showing registration indicia on the back surface of the camera support device that allows for a registration camera to capture the registration information for the cameras in the camera system; -
FIG. 11B is a top-down view of the camera registration system ofFIG. 10 showing how the registration camera captures an image of the camera support device to capture the camera system registration information; -
FIG. 12 shows a flow diagram that sets forth an example of an initial post-processing method for creating corrected viewable image data; -
FIG. 13A shows a flow diagram that sets forth an example post-processing method for converting the 2D corrected viewable image data formed by carrying the steps of the flow diagram ofFIG. 11 into 3D stereo image data; -
FIG. 13B shows a flow diagram that sets forth an example method of performing image rectification for 2D images of a scene captured with different cameras from different positions and orientations; -
FIG. 14A is an example black and white 2D image that includes a scene with a objects in the form of a tree, a person and wall sections that reside at a variety of distances from the camera; -
FIG. 14B is an example disparity map for the 2D image ofFIG. 14A , illustrating distance information relative to the camera in the form of a corresponding gray scale, with lighter gray scale being associated with closer distances and darker gray scale being associated with farther distances; -
FIG. 14C is an example point cloud formed from the disparity map ofFIG. 14B ; -
FIG. 14D is a close-up view of the point cloud ofFIG. 14C showing more detail of the man in the point cloud; -
FIG. 15A depict an example disparity map for an object in the form of a bas relief sculpture; -
FIG. 15B depicts example the point cloud data based for the bas relief sculpture on the disparity map ofFIG. 15A ; -
FIG. 16 shows details of the region A identified inFIG. 15B and shows how the point cloud is formed by points, with region B showing a close-up portion of region A that schematically illustrates the points making up the point cloud; -
FIG. 17 is a flow diagram that sets forth an example method of forming a 3D image or a 3D movie from the point cloud data and the camera positions used to capture the image data used to form the point cloud; and -
FIG. 18 is a diagram that schematically illustrates the method step of defining virtual camera positions with respect to a point cloud formed using the methods described herein, and also schematically illustrates the addition to the point cloud of a visual effect. - The various elements depicted in the drawing are merely representational and are not necessarily drawn to scale. Certain sections thereof may be exaggerated, while others may be minimized. The drawing is intended to illustrate an example embodiment of the disclosure that can be understood and appropriately carried out by those of ordinary skill in the art.
- The present disclosure relates generally to creating three-dimensional (3D) stereo images from two-dimensional (2D) photography, and in particular to systems and methods for 2D image capture and post-processing for 3D stereo imaging. The disclosure sets forth an overview of the 3D stereo imaging system and its components. The various components of the 3D stereo imaging system are then described in greater detail. Then, a variety of embodiments of the methods of the disclosure based on the operation of the 3D stereo imaging system are described. The terms “right” and “left” as applied to the witness cameras are relative to the 3D imaging system and its view of the scene.
- Various algorithms used to carry out the systems and methods of the invention are described herein along the way, and are also set forth in more detail in an “algorithms” section toward the end of this Detailed Description.
-
FIG. 1 is a generalized schematic diagram of the 3D stereo imaging system (“3D system”) 10 of the present disclosure.3D system 10 includes acamera system 100 that has two ormore cameras 102 spaced apart from its adjacent camera by respective distances DS, which need not be the same.FIG. 2 is a close-up, top-down schematic diagram of anexample camera system 100 that includes threecameras 102, namely aleft camera 102L, acenter camera 102C and aright camera 102R (Cartesian coordinates are shown for reference). - In an example, one of
cameras 102 is a cinemagraphic camera (“cine camera” or alternatively, “motion picture camera” or “production camera”) while the other one or more cameras are reference or “witness” cameras, e.g.,cameras cine camera 102C (assumed hereinbelow to becenter camera 102C for convenience) is thus used to capture images of the scene as contemplated by a director while the witness cameras are used to capture the 3D information relative to the cine camera. -
Cameras 102 are operably connected to one another viacommunication lines 104, which may be optical or electrical or wireless, for example.Cameras 102 may also be mechanically connected to one another via acamera support device 110. Eachcamera 102 includes animaging lens 120 with an associated field of view FOV, animage sensor 126,camera electronics 130 and a computer-readable medium 132, which in an example is in the form of a removable memory storage device (“removable memory”), such as a memory CF card, flash memory stick, CD ROM, etc. Computer-readable medium 132 is referred to below as removable memory for the sake of description. The operation ofcameras 102 is synchronized as described in greater detail below. -
3D system 10 further includes adata reading device 140 for reading data stored inremovable memories 132. In an example,data reading device 140 includes acomputer 150.Computer 150 includes a port 152, aprocessor 154 and amemory module 156. Port 152 is configured to receiveremovable memory 132 and read and store the contents of the removable memory inmemory module 156. -
3D system 10 also includes a removable-memory storage unit 180 where removable memories 152 are physically stored after use, as described below. An example removable-memory storage device 180 is a secure drawer or safe. -
3D system 10 further includes a primarydata storage unit 200 such as a server, and an optional back-up orsecondary storage unit 200′, such as another server. In an example, primarydata storage unit 200 andsecondary storage unit 200′ are operably linked via a communication link 202, and further in the example are configured to store a petabyte or more of data. In an example,secondary storage unit 200′ is mirrored toprimary storage unit 200. Primarydata storage unit 200 optionally includes anantenna system 158 for wireless EM data communication withcomputer 150. Primarydata storage unit 200 may be operably linked tocomputer 150 viaEthernet cable 160. - In the general operation of
3D system 10,cameras 102 each capture a raw 2D image, or a sequence of raw images of a scene over a volume 250 (seeFIG. 3 ). In an example, the captured raw images are filtered (e.g., pre-Bayer filtered) byrespective camera electronics 126. The filtered images are then time-coded, frame-coded and camera-coded and stored in their respectiveremovable memories 132. -
Removable memories 132 are removed from their respective cameras 102 (e.g., when the removable memories are full or nearly full) and with reference to arrow AR1, are transferred to the location ofcomputer 150. Eachremovable memory 132 is then inserted into port 152 andcomputer 150 reads the filtered image data stored therein, and stores the filtered image data inmemory module 156. With reference to arrow AR2, the removable memories are then delivered to removable-memory storage unit 180 for storage. - Meanwhile, the filtered image data in
memory module 156 ofcomputer 150 is pre-processed byprocessor 154 to create viewable images. In an example, the viewable images have the largest possible dynamic range and are color-corrected (neutral). Further, the pre-processing byprocessor 154 may also be performed to enhance the captured scene details. The pre-processing may be performed later on in thesame computer 150 or on a different computer. - The viewable image data stored in
memory module 156 is transferred to primarydata storage unit 200, e.g., viaEthernet cable 160 or via wirelessEM antennae systems 158 and wireless signals 159. Primarydata storage unit 200 optionally communicates with secondarydata storage unit 200′ to mirror the storage of the viewable image data for redundancy. The viewable image data include 2D range data from each of the one ormore witness cameras 102 and also includes 2D scene data (cinemagraphic images) from thecine camera 102C. - Once all of the viewable image data is collected in primary
data storage unit 200, it is post-processed as described in greater detail below. However, unlike data collected usingprior art 3D systems, the viewable image data stored indata storage unit 200 includes the full volumetric (3D) information of the captured scene withinvolume 250, which allows for robust post-processing of the viewable images. In particular, the viewable image data allows for substantially arbitrary virtual camera placement in the captured scene during post-processing, which greatly enhances the ability to create visual effects using CG animation in forming 3D stereo images, including 3D stereo movies. - In an example,
computer 150 is programmable with instructions (e.g., software) embodied in a computer-readable medium to cause the computer to perform the various functions and carry out various aspects of the methods described herein in greater detail below. In an example,computer 150 is a dedicated computer used only to read filtered image data fromremovable memories 132, pre-process the filtered data to form viewable image data, and transfer the viewable image data to primarydata storage unit 200. - As used herein, the term “computer-readable medium” refers to any medium such as
removable memory 132 andmemory module 156 that can store data and instructions, and that optionally participates in providing instructions toprocessor 154 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) operating as a server platform. Volatile media include dynamic memory, such as main memory of a computer platform. Physical transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media can take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. In an example,computer 150 includes anantenna system 158 for wireless electromagnetic (EM) data transmission via EM waves 159, and also in an example includes anEthernet cable 160 for wired data transmission via EM data signals S1. - Common forms of computer-readable media therefore include, for example: a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, less commonly used media such as punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, so-called “thumb drives,” any other memory chip or cartridge, a carrier-wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer can read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.
- With reference to
FIG. 2 , eachcamera 102 incamera system 100 has a field of view FOV centered on its optical axis, withleft camera 102L having a field of view FOV-L and an optical axis AL,center camera 102C having a field of view FOV-C and an optical axis AC, andright camera 102R having a field ofview 102R and an optical axis AR. -
FIG. 3 is a close-up view of the overlapping fields of view FOV of the threecameras 102 ofcamera system 100, and illustrates a top-down view of a scene that includes example objects 240 in the form of a picnic table 2401, agarbage can 240G, abarbeque 240B, a shrub 2405 and ablock wall 240W. The fields of view FOV are three-dimensional frustums, each covering a volume of space. The field of view FOV in the X-Z plane need not be the same in the Y-Z plane, and the fields of view of the cameras can be different.Cameras 102 are configured so that their fields of view FOV all start to overlap at some distance D1 fromcenter camera 102C, with the overlapping fields of view defining acommon volume 250 of space associated with a scene being imaged by each camera from a different orientation (i.e., convergence angle). A cross-section of thevolume 250 is shown inFIG. 3 . - In an example,
volume 250 is in part defined by anouter boundary 260 at a distance DB=DV+D1 fromcenter camera 102C.Outer boundary 260 is generally defined by the camera spacings DSL and DSR and convergence angles θCL and θCR and represents the limit at which accurate 3D volumetric information can be obtained withcameras 102. Having such an outer boundary onvolume 250 usually does not present a problem for most 3D stereo imaging applications because the emphasis of the 3D effect is usually on the objects within the foregrounds of the volume while distant objects are generally treated as background. - In an example, camera spacings DS are defined by the object-to-camera distance. A good ratio of the object-to-camera distance DC as compared to the camera spacing DS (i.e., DS/DC) is about 1:6, with about 1:10 representing an example minimum ratio. However, greater ratios, such as 1:4, can be used.
- It is noted here that even though the distance between human eyes is about 68 mm, the distance normally used between conventional stereo cameras (called the “interaxial distance”) is sometimes much less. This is because the stereographer is often asked to compromise between creating good stereo separation of objects in a scene and producing comfortable viewing stereo without causing the many artifacts that come with an extended stereo effect. A good rule of thumb for a normal the interaxial distance is between 19 mm and 38 mm. As discussed below, this base distance is much too small to calculate accurate range or point cloud data. This is important in creating the depth map used to convert 2D to 3D, but also for being able to combine visual effects and CG character animation in native stereo plates. In an example, the use of one or more witness cameras with a stereo production camera rig, but with the witness cameras being spaced apart from the conventional stereo production rig, allows for creating 3D point cloud data that can be used for post-production visual effects and animation and for motion capture. In other words, the cine camera as described herein can, in one example, be a conventional stereo production rig having the aforementioned typical interaxial distance.
- The operation of
cameras 102 is synchronized viacamera electronics 130. In an example, one of the cameras, such ascenter camera 102C, includes aclock 131 and an electronic interface as part of itscamera electronics 130, with the clock and electronic interface being used to synchronize the frames of the cameras. In an example,witness cameras 102 are shutter-phased tocine camera 102C. In examples, global or rolling shutter can be used. - In an example, the point of trigger (exposure) for
witness cameras 102 is determined when the shutter ofcine camera 102C is at the center of the “effective” shutter open position.Witness cameras 102 preferably shoot with the shortest possible shutter settings to minimize “motion blur,” and overall exposure is controlled by varying the stop inlens 120 or by adjusting the system gain viacamera electronics 130. The use of time-code slates and the recording of time code as metadata make it possible to synchronize and identify the corresponding cine images captured bycine camera 102C and the reference images captured bywitness cameras 102. - In an example, one or more of
cameras 102 are high-resolution color machine-vision cameras. As discussed above,cameras 102 need not be the same, though for cameras havingimage sensors 126 with different pixel sizes, rectification of the pixel sizes should be performed as part of the post-processing for optimum results. - An
example camera 102 is a digital camera suitable for both still and motion pictures. Anexample camera 102 has a 5K (e.g., 5120 pixels by 2700 pixels) orgreater image sensor 126, withlens 120 having 18 stops of dynamic range and a rolling shutter. Asuitable witness camera 102 includes the EPIC or SCARLET cameras available from Red Camera Company, Lake Forest, Calif. Such cameras can be used for just the one ormore witness cameras 102, just thecine camera 102C, or can be used for both witness and cine cameras so that all of the cameras are the same. It is anticipated that the digital technology forcameras 102 will continue to develop and improve, e.g., will have increasing numbers of pixels, better lenses, more robust camera electronics, etc. - In an example,
camera electronics 130 includes wavelet compression to optimize image data storage inremovable memory 132.Camera electronics 130 also optionally includes filtering capability for filtering the raw captured image so that a filtered image is stored inremovable memory 132. In an example, aremoval memory 132 that provides 64 gigabits (Gb) of storage can store about 6 minutes worth of filtered images. - In an example,
image sensors 130 ofwitness cameras 102 are sensitive to near-IR wavelengths (850 nm to 940 nm) so that a potentially dark scene can be illuminated with near infrared light. In this case,cine camera 102C can be protected from viewing the IR illumination through the use of an IR rejection filter, such as the Tiffen IR ND Filter. - In an example,
witness cameras 102 are equipped with a variety of matched focal length lenses 120W that match the horizontal view of the cine lenses 120C. In an example, the choice of lens 120W for thewitness cameras 102 is determined by the choice of lens for thecine camera 102, subject the criteria that the horizontal field of view witness lens 120W is greater than or equal to the horizontal field of view of the cine lens 120C. - As discussed above, in an example,
cameras 102 are operably supported by acamera support device 110.FIG. 4 is a top-down view andFIG. 5A is a front-on view of an examplecamera support device 110 in the form of a rail system configured to allowcameras 102 to pan horizontally (parallel view), and to have select convergence angles θC (e.g., θCL and θCL for the three-camera configuration). - An example
camera support device 110 includes arail 300 to whichcameras 102 can be mounted.Witness cameras 102 each include a mountingdevice 310 that allows for the witness cameras to be positioned alongrail 300 and fixed thereto at desired positions, i.e., at a select distance DS fromcine camera 102C. In an example,rail 300 includes atrack 302 sized to accommodate a complimentary groove (not shown) in mountingdevice 310. Mounting device also is configured to allow for eachcorresponding witness camera 102 to rotate about its Y-axis, i.e., about axis ALY forleft witness camera 102L and about axis ARY forright witness camera 102R. - In an example,
camera support device 310 is configured so that the convergence angles θC can be set to within a few arc-seconds of a desired or target convergence angle. In an example, the convergence angles θC can be manually set by the operator depending on the 3D requirements of the scene. In another example, a high-precision rotation motor is used to automatically set the convergence angle. - In an example embodiment, the convergence angle θC for each
witness camera 102 is adjustable in small angular increments, e.g., 5 degree increments, using for example a precise mechanical pin registration system for accurate and repeatable setting of the convergence angle. Thus, in the example illustrated inFIG. 4 andFIG. 5 , mountingdevice 310 includes aplate 320 havingholes 322 formed therein that locks the mounting device to rail 300 viapins 324 that engage correspondingholes 326 inrail 300.Holes 322 are configured to accurately correspond to select convergence angles θC. In an example, holes 326 are also formed at precise distances fromcine camera 102C. The length L ofrail 300 can be any practical length, with 1 meter to 6 meters being an exemplary range of lengths. In an example embodiment,rail 300 is telescopic so that length L can be adjusted. - In an example embodiment,
rail 300 is configured to that it experiences minimum bending, flexing and vibration under operating conditions, with the result that the relative positions and orientations (convergence angles) ofcameras 102 are precisely maintained at all times with substantially no relative movement or vibration during the operation of3D system 10.Rail 300 is preferably made of one or more light-weight materials, with aluminum and carbon fiber being two exemplary materials. One skilled in the art will appreciated that various mounting configurations forcameras 102 are possible and that the example configurations described herein are provided by way of example. -
FIG. 5B is similar toFIG. 5A and illustrates an example embodiment where 3D system includes twowitness cameras stereo camera rig 102S to obtain conventional 3D stereo images while thewitness cameras stereo rig 102S may also include a beam-splitter arrangement that allows for the two cinematography images to be captured at positions separated by the interocular distance of about 68 mm. -
FIG. 6 illustrates an example configuration wherecamera mounting device 310 includes abase support 350, such as a tripod stand or like base support structure. Note, however, thatcameras 102 need not be supported by a commoncamera mounting device 310. Thus, with reference toFIG. 7 , an example embodiment ofcamera system 100 has eachcamera 102 supported by its own base support so thatcamera support device 110 is not needed. This configuration is workable as long as the relative positions and orientations ofcameras 102 are accurately known. The relative positions and orientation ofcameras 102 can be established, for example, using laser-based position andalignment devices 352, which can be interfaced withcamera electronics 130.Laser beams 353 associated with laser-based position andalignment devices 352 are shown as dotted lines inFIG. 7 . Wireless communication can also be used between thecamera electronics 130, as illustrated by wireless signals 131. - In
FIG. 7A , the cameras can be quite far apart, as better illustrated in the plan view ofFIG. 7B . - The configuration of
3D system 10 ofFIGS. 7A and 7B supports an embodiment wherecine camera 102C is “free-roaming,” meaning that it can be positioned independently of thewitness cameras Cameras 102 would be in wireless communication to sync up all camera shutters. Post-processing as described in detail below is then used to connect the cine camera view to the point cloud data, which is taken from a different perspective. - In an example,
cameras 102 can be in wireless communication to sync up all camera shutters. Post-processing, as described in detail below, is then used to connect the cine camera view to the point cloud data, which is taken from a different perspective. Since thewitness Cameras production 3D stereo cameras, the creation of volumetric point cloud data (which is created on a frame-by-frame basis) is like having a 3D scan of the scene (e.g., the set and actors therein) per frame. This animating point cloud not only records accurately the 3-space of fixed architecture, but also records the movement and position of actors and objects in motion within the scene. - The
production 3D stereo cameras are than virtually tracked and locked to this point cloud on a per frame basis using a 3D computer software package, such as MAYA® point-cloud software, available from Autodesk, San Rafael, Calif. The six axes of tracked translations of the camera are then key-framed into a virtual facsimile of theproduction 3D camera rig, with the view of these virtual cameras locked to the animated point cloud. - This animated point cloud is extremely helpful to the animation “Layout Department,” which is involved in adding additional CG elements to the live action stereo plates, such as CG characters and enhancing the live action environment with additional CG sets and architecture. The point cloud gives precise 3D positions of all objects in the scene, thus allowing the CG additions to be made to the live action stereo plate in an efficient and accurate manner.
- The motion capture of live (e.g., human or animal) actors to be combined digitally with backgrounds shot with either 2D or 3D stereo cameras presents a significant challenge when attempting to align the motion capture volume to the photographed plate. Since true spatial data of the photographed scene is not embedded within the initial photography, other techniques must be used to properly align the actors' position and their actions within the motion capture volume.
- The animated point cloud is a robust solution because each point in the point cloud can be loaded virtually into the 3D volume of the motion capture stage. This point cloud allows the operators and actors to reference their position and timings to the animated point cloud, thereby recreating the original objects in the photographed scene. The virtual production camera with its key-framed positional translations can also be imported into the motion capture volume. This virtual camera allows the motion capture operators to view the motion capture actors through the same effective virtual lens as what was used to shoot the original live action background.
- An additional use of the point cloud in a motion capture volume is to convert this point cloud into a “bird's eye” view from the top to be projected in scale on the floor of the motion capture volume. An orthographic view from above of the point cloud can be graphically simplified into basic animated artwork that represents critical objects or timings of objects in a scene that actors must accurately interact with. This simplified graphic representation of the point cloud can be projected with a video projector onto the floor of the motion capture volume. This graphic representation of the point cloud data can perform as an interactive guide for the motion capture actors to help position their actions for both position and timings to photographed elements critical to the scene.
-
FIG. 8 illustrates an example embodiment wherecamera system 100 includes anillumination system 400. In an example,illumination system 400 includes one or morelight sources 402 that in one example emit light 404 in the near IR wavelength range of 850 nm to 940 nm. Examples of IRlight source 402 include light-emitting diodes (LEDs), laser diodes or near-IR lasers. In an example,illumination system 400 is configured to illuminate the scene involume 250 when production lighting is less than optimal. In an example,IR illumination system 400 is configured so that the amount ofillumination 404 fromlight sources 402 can be controlled. In an example,light source 402 emit IR light 404 and the amount of IR light level is balanced with the production lighting (which is typically white light) used for the particular scene. - In an example,
illumination system 400 is configured to generate “textured light” 404 that includes a spatial intensity variation that result in a light pattern when anobject 240 inscene 242 is illuminated.Textured light 404 is used to highlightobjects 240 that might otherwise lack sufficient texture to be able to determine the objects range and the range of features within the object. -
FIG. 9 illustrates an examplelight source 402 that generatestextured light 404.Light source 402 ofFIG. 9 includes alaser 420 and adiffraction grating 424 arranged to receive light 422 from the laser and diffract light 422 to formtextured light 404 comprising multiple diffracted light beams. In an example,laser 420 operates in the near IR so thattextured light 404 is visible to witnesscameras 102 but is made invisible to thecine camera 102C using a IR-rejection filter. In another example,laser 420 operates in a narrow band of visible light, e.g., 589 nm (yellow-orange), which is visible to witnesscameras 102 but is blocked from being detected by the cine camera using a correspond narrow-band rejection filter (e.g., a Didymium-coated filter). Such narrow-band filtering in the visible does not substantially affect the ability ofcine camera 102C to record live-action color 2D image. - In an example,
lasers 420 are pulsed (e.g., strobed) in synchronization with the shuttering of the witness camera shutters and cine camera shutters. The laser pulse duration is preferably kept as short as possible to reduce any distracting effects to the case and crew involved withimaging scene 240. In an example,light sources 402 are synchronized with thesame camera electronics 130 that control the operation of thecine camera 102C andwitness cameras 102. - It is anticipated that many 3D stereo imaging applications that use
3D system 10 will require high-quality imaging. Because the post-processing methods described in detail below involve comparing high-resolution images from different cameras, small differences in the imaging quality oflenses 120 andimage sensors 130 can result in errors when the pre-processed images are combined. For example, differences in distortion between cameras can lead to difficulties in correlating pixel positions of the different cameras for a given object in the scene because the distortion will cause an unintended shift in pixel positions for different cameras imaging the same object from different directions. Thus, knowing the camera distortions allows for correcting the captured images prior to attempting camera pixel correlations and performing post-processing the viewable images. - Accordingly, an aspect of the disclosure includes measuring one or more camera parameters for
cameras 102 and performing a camera calibration so that differences between cameras that could impact the post-processing of viewable images is reduced and preferably substantially eliminated. Example camera parameters include lens distortion, sensor position, principal point and nodal point locations, and F-stop vs. focal length variations. - In example, at least some of the camera parameters can be obtained from the camera manufacturer for the given lenses. The difference in camera parameters between
cameras 102 can be reduced or eliminated during the post-processing steps since the viewable images from the different cameras are still available for processing, and the measured camera parameters directly relate to the captured viewable images for the particular camera. - Once the above-mentioned parameters for
cameras 102 are measured, the cameras need to be arranged oncamera support device 110 so that the relative camera orientations are well defined. Thus, in an example camera registration is performed wherein eachcamera 102 is mounted on the camera support device and then the camera spatial coordinates (x,y,z) and the corresponding camera angular coordinates (ω,φ,κ) (i.e., rotation angles about x, y and z, respectively) are established for each camera relative to a reference (“world”) coordinate system. -
FIG. 10 is a schematic diagram of an examplecamera registration system 500. Camera registration system includescamera system 100 withcameras 102 mounted oncamera support device 110. Ajig 301 that includes reference objects 502 is arranged relative tocamera system 100. Reference objects 502 have select heights, sizes and positions. Test images are captured by eachcamera 102 and the camera orientation established relative to a reference coordinate system, which is shown based atleft witness camera 102 by way of example. Camera parameters such as distortion can also be measured usingjig 500. -
FIG. 11A is a rear-end view of the camera registration system ofFIG. 10 showingregistration indicia 320 on theback surface 301 ofcamera support device 110.FIG. 11B is a top-down view of the camera registration system ofFIG. 11A and shows how aregistration camera 321 with a field of view FOV-A captures an image ofcamera support device 110 to collect camera registration information. -
FIG. 12 shows a flow diagram 550 that sets forth an example of an initial post-processing method for creating corrected viewable image data. The method comprises a number ofsteps 551 through 555. - Thus in
step 551, the viewable image data is accessed, and instep 552 the camera calibration (based on the measured camera parameters) is applied to the viewable image data to reduce or eliminate differences in the viewable images due to differences betweencameras 102. - Next in
step 553, the camera registration is applied to the camera-calibrated viewable image data formed instep 552 to reduce or eliminate camera registration errors betweencameras 102. - Step 554 is an optional step of adding information (e.g., notes, comments, etc.) to the various image frames.
- The result of the above steps forms in (step) 555 a first output file denoted FILE-1.ccc that includes corrected viewable image data. The corrected viewable image data is in the form of frames F for each of the witness cameras 102 (e.g., frames FWL and FWR for the left and
right witness cameras cine camera 102C (e.g., frames FC). - The corrected viewable image data for each frame F is 2D in spatial coordinates and also includes R-G-B color information, i.e., F(x,y;RGB), with x and y corresponding to the pixel spatial coordinates of the
corresponding image sensor 130. The various frames F can be stored in a first output file FILE1, which in an example is in “curtain call native bitmap graphic format,” i.e., FILE-1.ccc. -
FIG. 13A shows a flow diagram 580 that sets forth an example post-processing method (withsteps 581 through 585) for converting the 2D corrected viewable image data in first output FILE-1.ccc into 3D stereo image data. With reference to flow diagram 580, instep 581, the first output file FILE-1.ccc is accessed and instep 582 image rectification is performed. Image rectification involves ensuring that each of the images for each frame F has substantially the same format. For example, ifcameras 102 reside in different planes, the images (frames) will be skewed relative to one another. The process of image rectification seeks to remove this skew, or at least reduce it so that it is insubstantial. Also, slight magnification differences between the cameras can result in differences in the size of the images. - Thus, the image rectification process ensures that the image sizes are the same to within sub-pixel resolution. Similarly, differences in the images caused by slight differences in the fields of view of the
cameras 102 can be reduced or eliminated during the image rectification process. Thus, image rectification is directed to reducing or eliminating one or more differences in the captured images so that all of the images have substantially the same fundamental format (shape, size, etc.). An example image rectification process is provided in the flow diagram 582 ofFIG. 13B , discussed in greater detail below. - Next in
step 583, stereo matching is performed on the image-rectified, corrected viewable image data formed in 582. Stereo matching involves correlating the pixel information between the various images, e.g., matching the left witness camera frames FwL(xL,yL;RGB), the right witness camera frames FWR(xR,yR;RGB) and the cine camera frames FC(xC,yC;RGB) so that the pixels for the different frames are correlated. This means, for example, that anobject 240 in scene 242 (seeFIG. 3 ) represented by one set of pixels in the left witness camera frames FWD can now be correlated to a corresponding set of pixels in the cine camera frames FC and the right witness camera frames FWR. - Next in
step 584, the witness camera viewable image data is related to the cine camera viewable image data from 583. This is accomplished using a processed called “roto.” - With reference to
FIG. 14A , the roto process involves defining and carving out one or more select portions of the viewable image referred to as “regions of interest,” or ROIs.FIG. 14A is an example 2D black andwhite image 600 fromcine camera 102C.2D image 600 includes objects 240-1 and 240-2 (portions of a brick wall in the background), object 240-3 (a person) and object 240-4 (a portion of a tree in the foreground). Each of these objects has been rotoed (i.e., selected as an ROI), as indicated by the dark-line outline of each of these objects. - Only the selected ROIs of the viewable images are processed while the remaining portions of the viewable images are treated as background. The roto process can be done manually for an initial frame and then carried through automatically for multiple frames.
- Thus, the process associated with
step 584 includes performing a roto on the cine camera viewable images to define select ROIs to create a rotoed cine camera viewable image RC. This rotoed viewable image RC is then transformed to theleft witness camera 102L to form a left witness camera rotoed image RL. This rotoed image is then transformed to the right witness camera to form a left witness camera rotoed image RR. This process results in three rotoed viewable images RL, RC and RR having corresponding or matched ROIs, where the rotoed portions (ROIs) match even though the viewing angles forcameras - Next in
step 585, the rotoed viewable images formed instep 584 are used to create a “disparity map” 600 that associates gray-scale pixel intensities in the rotoed viewable images with distance (z) from thecine camera 102C.FIG. 14B is thecorresponding disparity map 610 to2D image 600 ofFIG. 14A and shows in gray scale the corresponding objects, which are identified in the disparity map as 240′-1 through 240′-4. The remaining portion of the disparity map has a uniform gray background. The gray scale is such that the lighter the color, the closer the object is to camera 120C. Thus, the tree object 240-4 is in the foreground, the wall objects 240-1 and 240-2 are in the background, and the person object 240-3 is at an intermediate distance between the wall object and the tree object. -
Disparity map 610 generally provides range data for the rotoed viewable images and comprises one or more frames FD(x,y,z)=FD(x,y; gray scale) (FIG. 14B can thus be thought of as an example disparity map frame). The disparity map is formed from the rotoed images (ROIs) in one example by comparing the difference (Δx, Δy) for each pixel in left witness camera frames FWL and the right witness camera frames FWR and associating the difference with a distance z. The gray-scale intensities in the image are correlated to distance z, with objects closer to the camera having a lighter gray-scale level than objects farther away, as illustrated inFIG. 14B . In an example,disparity map 610 can have a very large gray-scale dynamic range, e.g., 0 to 65,000. This allows for the disparity map to include a tremendous amount of distance information for the various objects (including portions of objects) in the scene for a given frame. - In an example embodiment, the gray scale for two or more disparity map frames is normalized so that a sequence of frames has a common reference for distance information, i.e., a certain gray scale represents the same distance in one frame as it does in another frame. This is particularly useful when the gray scale can have a large dynamic range such as the aforementioned 0 to 65,000 range.
-
FIG. 14C is an example point cloud formed from the disparity map ofFIG. 14B .FIG. 14D is a close-up view of the point cloud ofFIG. 14C showing more detail of theman 240′-3 in the point cloud. -
FIG. 15A depicts an example disparity map 240DM for an object in the form of a has relief sculpture. - With reference again to flow diagram 580 of
FIG. 13A , next instep 586 3D image data is calculated, whereby the 2D images fromcine camera 102C are embedded with the range data at sub-pixel accuracy, creating a voxel-based image. The 3D image data can be expressed as a “point cloud” P(x,y,z), where each element in the point cloud is a “pixol” representing a point in 3D space. The point cloud P is typically comprised of thousands of points. - An example point cloud 240PC formed from the has relief sculpture disparity map 240DM of
FIG. 15A is shown inFIG. 15B . A region A of point cloud 240P is denoted by a dashed-line rectangle.FIG. 16 is a close-up view of region A that shows point cloud 240P being comprised of thousands ofpoints 241 in the point cloud. A close-up inset B shows a magnified small region of region A that more clearly shows points 241. Note that point cloud 240P looks remarkably like an actual image or picture. - The 3D image data can also be expressed as a polygonal mesh PM(x,y,z;RGB), wherein each element in the mesh is a “voxol” defined by three spatial coordinates and corresponding RGB values.
- In an example, the 3D image data is calculated from the disparity map using triangulation techniques known in the art of photogrammetry (i.e., photogrammetric triangulation). This process solves the position of the six camera parameters (i.e., spatial coordinates (x,y,z) and angular coordinates (ω,φ,κ) using the calibrated camera coefficients to compute the 3D coordinates. This process is described in greater detail below in connection with the image rectification process.
- In
step 587, the 3D image data is outputted to a file, e.g., FILE-3D, which can be in one of the commonly used graphics formats such as ASCII. FILE-3D includes either the point cloud data or the polygonal mesh data. - The
image rectification step 582 discussed above in connection with flow diagram 580 is now discussed in greater detail by expandingstep 582 into a flow diagram set forth inFIG. 13B . Thus, step 582-1 involves accessing the left and right input images (frames) FWL an FWR from first output file FILE-1.ccc, as well as accessing the camera calibration data. - In step 582-2, for each input image (frame), a rotation matrix R is computed from the image coordinates to the reference (world) coordinates using rotation angles M=(ω,φ,κ) in the calibration data for each camera. The rotation angles for the left and right cameras are respectively defined by M1 and M2. The rectification rotation matrix R for the image pair (frame pair) can be calculated as follows. First, let e1, e2 and e3 be three 3D vectors, given by as follows:
- e1=projection center of second camera−projection center of first camera
- e1=e1/(magnitude of e1)
- e2=sum of third column of the rotation matrix of the two images
- e2=e2×e1, where × stands for cross product
- e2=e2/(magnitude of e2)
- e3=e1×e2
- e3=e3/(magnitude of e3)
- Then the rectification rotation matrix R is formed by column vectors e1, e2 and e3.
- Next, in step 582-3, the average focal length AFL is calculated as follows:
-
ALF=((CC1/PS1)+(CC2/PS2))*(size percentage)/2 - where CC1 and CC2 are the camera constants for the first and second cameras and PS1 and PS2 are the pixel sizes for the images for the first and second cameras, and the size percentage. The size percentage can be selected to be 100% of the actual image size or any other percentage, usually smaller (e.g., 50%), which makes the point cloud smaller and more dense.
- Next in step 582-4, the average principal point APP is calculated as follows:
-
APP=((PP1/PS1)+(PP2/{S2))*(size percentage)/2 - where PP1 and PP2 are the principal points for the first and second cameras.
- Next in step 582-5, for each image, the product matrices P1 and P2 are calculated using the rotation matrices, i.e., the angle matrices M1 and M2, and the rectification rotation matrix R, via the relationships P1=R*M1; P2=: R*M2.
- Next in step 582-6, for each input image and for each pixel (x, y) in the input image, the image point relative to project center (i.e., principal point) and with respect to the image coordinates using camera calibration parameters A, B, K1, K2, P1, P2 is calculated as follows:
- Let x′=x*(PS)−(PPx) where PS=pixel size, PPx=x-coordinate of the principal point PP
- Let y′=y*(PS)−(PPy) PPy=x-coordinate of the principal point PP
- Let radial distortion factor=r=K1*s+K2*s*s, where s=x′2 y′2
- Let tangential distortion factor=t=2*x′*y′
- Then the x, y, z-coordinates of image point are given by:
-
x″=x′+A*y′+r*x′+P2*t+P1*(s+2*x′̂2), -
y″=y′+B*x′+r*y′+P1*t+P2*(s+2*y′̂2). -
z″=camera constant - Next in 582-7, the image point IP′″=(x′″, y′″, z′″) relative to project center and with respect to the world coordinates is calculated by applying the matrix P1 or P2 on the column vector of IP″=(x″, y″ and z″), i.e., IP′″=P1*IP″ or IP′″=P2*IP″.
- Next in 582-8, the new pixel coordinates for the rectified image are computed as follows:
-
x=(APPx)−(AFL)*x′″//z′″ -
y=(APPy)−(AFL)*y′″/z′″ - where APPx and APPy are the x and y coordinates of the average principle point.
- Then step 582-9 involves finding the position, size and color of the rectified left and right ROIs. This involves an inverse-mapping process for each rectified image and for each pixel (x, y) in the rectified ROI in the rectified image. Specifically, the rectified image is inverse-mapped back to the input image by iteratively moving towards the closest pixel in input image, and then finding the sub-pixel ratios. For each pixel (x,y) in the rectified image, the closest pixel and sub-pixel ratios are used to determine the color of that pixel. The output comprises left and right rectified images with ROIs, with color values (RGB) at each pixel, namely voxels (x,y;RGB).
- An aspect of the disclosure involves making a 3D stereo image or a 3D stereo movie (i.e., a sequence of 3D stereo images) using the point cloud data, range data, or an image-based displacement map.
FIG. 17 sets forth a flow diagram of an example method of forming 3D stereo image or movie. Instep 601, the point cloud data and the camera positions used to obtain the point cloud data are provided. Then, instep 602, first and second virtual camera positions are selected, thereby defining a 3D stereo pair. This replicates a stereo rig and allows for arbitrary 3D virtual filming of the scene as a post-processing step. Thisstep 602 is schematically represented inFIG. 18 , which shows a point cloud PC and first and second virtual cameras CV1 and CV2. - A second technique to create a 3D stereo separate pair (Left and Right eyes) from a 2D image is to horizontally displace (warp) the pixels in a horizontal direction by a specific amount. This amount differs between the two left and right eyes, creating the illusion of stereo depth. The amount of horizontal displacement is informed by both the range data of the corresponding pixol and the stereo volume required for the creative specifics of the shot.
- Step 603 includes the optional step of combining the point cloud with at least one visual effect to enhance the 3D stereo image or 3D stereo movie formed as described above. Example visual effects include computer graphics, computer animation, hand-drawn animation, computer-generated environments, computer-generated or hand-drawn imagery, computer-generated objects, and the like. Generally, a visual effect means any additional information added to the point cloud.
FIG. 18 illustrates schematically the addition to point cloud PC of a visual effect VE in the form of an animated dragon by way of a simple example. - In
step 604, the 3D stereo image or movie is created using the selected positions of the first and second virtual cameras. In post-production, 3D stereo images are created (left and right eyes) from 2D imagery using one of two primary techniques. The first method is to re-create a virtual version of a two-camera 3D stereo rig in CG. This virtual 3D stereo rig has the same controls at its physical (real) counterpart for creating stereo photography, i.e., convergence and setting of the interaxial distance between virtual cameras. The 2D imagery is projected as a texture map onto a polygonal geometry mesh, which is created from the point cloud data embedded in each frame. This texture map of the 2D live action is physically displaced and projected onto this geometry. The 2D imagery has now been given shape by the geometry, which is a true 3D view of the scene as photographed from point of view of the production camera and one or more attendant witness cameras. The virtual 3D stereo rig is then used to virtually re-photograph this frame, thereby producing a discreet left and right eye element for stereo viewing. - A second method is to use the depth information stored in each pixol, which contains the X, Y, Z information for the voxel-based image. As described above, a 3D stereo image can be created from a 2D image by re-positioning the pixels of the image in a horizontal manner, thereby creating two separate images (left & right eyes). The amount of horizontal displacement is informed by both the range data of the corresponding pixol and the stereo volume required for the creative specifics of the shot.
- In an example, forming the 3D stereo movie includes making one or more critical 3D-stereo-related decisions such as camera position, orientations, convergence angles, panning, etc., in post-production rather at the time of capturing the 2D images. Thus, an aspect of the disclosure includes relaxing the conditions under which the original 2D cinemagraphic images are captured. This simplifies the filming process for the movie on location and allows for much greater creative freedom to shape the movie in the post-production phase. This also saves substantial time and cost in making a 3D stereo movie.
- Several algorithms related to carrying out different aspects of the invention are now described below in more detail and in an abbreviated format that is familiar to those skilled in the art.
- Input:
- Left and right input images
ROIs on the input images
Calibration data for the left and right cameras - Output:
- Left and right rectified image of ROI with color values at each pixel
- Process:
- 1. Map the ROIs on input images into ROIs on rectified images:
1.1 For each input image, compute rotation matrix from image to world using Omega, Phi and Kappa in the calibration data for each camera. Call them M1 and M2.
1.2 Compute the rectification rotation matrix R for the image pair:
Let e1, e2 and e3 be three 3D vectors, given by:
e1=: projection centre of second camera−projection centre of first camera
e1=: e1/(magnitude of e1)
e2=: sum of third column of the rotation matrix of the two images found in 1.1
e2=: e2×e1, where × stands for cross product
e2=: e2/(magnitude of e2)
e3=: e1×e2
e3=: e3/(magnitude of e3) - Then the rectification rotation matrix is formed by column vectors e1, e2 and e3
-
Average focal length=:((camera constant/pixel size of first image)+(camera constant/pixel size of first image))*(size percentage)/2 -
Average principal point=:((principal point/pixel size of first image)+(principal point/pixel size of first image))*(size percentage)/2 - 1.4 For each image, find product matrices:
-
P0=:R*M0; P1=:R*M1 - 1.5. For each input image, for each pixel (x, y) in the image:
1.5.1 Compute image point relative to project centre (w.r.t. image) using camera calibration parameters A, B, K1, K2, P1, P2:
Let x′=: x*(pixel size)−(x coordinate of principal point)
Let y′=: y*(pixel size)−(y coordinate of principal point)
Let radial distortion factor=r=: K1*s+K2*s*s, where s=x′̂2+y′̂2
Let tangential distortion factor=t=: 2*x′*y′
Then the x, y, z-coordinates of image point are given by: -
x″=:x′+A*y′+r*x′+P2*t+P1*(s+2*x′̂2), -
y″=:y′+B*x′+r*y′+P1*t+P2*(s+2*y′̂2). -
z″=:camera constant - 1.5.2 Compute image point (x′″, y′″, z′″) relative to project centre (w.r.t. world) by applying the matrix P0 or P1 on the column vector of x″, y″ and z″
1.5.3 Compute new pixel coordinates on rectified image as: -
x-coord=:(x-coord of average principal point)−(average focal length)*x′″/z′″ -
y-coord=:(y-coord of average principal point)−(average focal length)*y′″/z′″ - 1.6 Using 1.5, find the position and size of the rectified left and right ROIs
2. For each rectified image, for each pixel (x, y) in the rectified ROI on the image, inverse-map it back to the input image by iteratively moving towards the closest pixel in input image, and then finding the subpixel ratios.
3. For each rectified image, for each pixel (x,y), use the closest pixel and subpixel ratios to determine the color of that pixel. - Input:
- Left and right rectified images of ROI
Disparity values at each pixel in left rectified image (defined as difference between x-subpixel-coordinate of a pixel on left rectified image, and that of the corresponding pixel on the right rectified image) - Calibration data for the left and right cameras
- Threshold for distance between a pair of rays
- Output:
- Point cloud with (x, y, z) world coordinates of the point corresponding to each pixel in the left rectified image
- Process:
- 1. For each input image, compute rotation matrix from image to world using Omega, Phi and Kappa. Call them M1 and M2.
2. For each pixel on the left rectified image:
2.1 Inverse-map the pixel back to the left input image (see Rectification Step 2)
2.2 Find the corresponding pixel on the right rectified image by disparity value, and inverse-map it back to the right input image
2.3 For each of left and right pixels found above:
2.3.1 Compute image point relative to projection centre (w.r.t. image) (see Rectification Step 1.5.1)
2.3.2 Compute image point relative to projection centre (w.r.t. world) by applying rotation matrix on column vector of the result of 2.3.1. Denote these relative position vectors by p1 and p2 for left and right pixels respectively.
2.4 Find the point of intersection by analytic geometry:
Let d=: project centre of second camera−projection centre of first camera
Let the common normal be n=: p1×p2
Let a1=: (d×p2)·n/(n·n) and a2=: (d×p1)·n/(n·n) where · stands for dot product.
So, the spatial intersection for left and right are given by (q1+q2)/2, where -
q1=:(projection centre of first camera)−a1*q1 -
q2=:(projection centre of first camera)−a2*p2 - only if the distance between q1 and q2 is less than the given threshold.
- Input:
- Left and right rectified images of ROI
Set D of disparity values at each pixel in left rectified image
Calibration data for the left and right cameras
Point cloud C formed from the left and right rectified images by Spatial Intersection algorithm above
Middle image
Calibration data for the middle camera
Aberration correction parameters for middle camera
Threshold for distance between adjacent subpixels in middle image - Output:
- Disparity values for each pixel in the middle image (if that pixel maps inside the point cloud)
Point cloud with (x, y, z) world coordinates of the point corresponding to each pixel in the middle image - Process:
- 1. Map the given point cloud C into the middle image:
1.1 Compute rotation matrix from middle image to world using Omega, Phi and Kappa in the calibration data for the middle camera.
1.2 For each vertex in C, find its corresponding subpixel coordinates on the middle image:
1.2.1 Transform the position P of the vertex to the corresponding point Q the middle image (both in world coordinates):
Let p=: position vector of P relative to projection centre (w.r.t. world)=position of P−projection centre
Then position vector of Q is obtained by applying rotation matrix on p.
1.2.2 The image subpixel coordinates of the pixel corresponding to P is given by: -
x-subpixel-coord=:x+(x-coord of principal point)/(pixel size); -
y-subpixel-coord=:y+(y-coord of principal point)/(pixel size); - where x=:−(x coord of Q)/(z coord of Q)*(camera constant of middle image)+aberration correction
and y=:−(y coord of Q)/(z coord of Q)*(camera constant of middle image)+aberration correction
1.2.3 Record the vertex in C and the disparity values in D for subpixel.
1.3 The set S of middle image subpixels formed in 1.2 is then processed to remove isolated subpixels whose distance from its nearest subpixel exceeds the given threshold.
1.4 For each pixel in the middle image, find three subpixels in S closest to it, and the points in C and disparities in D for these three subpixels, recorded in 1.2.3. Then use a real interpolation to determine the (x, y, z) of the point and disparity for that pixel. - It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the spirit and scope of the disclosure. Thus it is intended that the present disclosure cover the modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalents, which claims are incorporated into and constitute part of this specification.
Claims (21)
1. A method of converting two-dimensional (2D) images of a scene having therein at least one object to one or more three-dimensional (3D) images of the scene, comprising:
simultaneously capturing at least first and second 2D images of the scene from corresponding at least first and second cameras having respective camera positions and orientations measured relative to a reference coordinate system;
forming a disparity map from the at least first and second 2D images, wherein the disparity map has a gray scale that corresponds to distance information of the at least one object relative to the reference coordinate system; and
forming from the disparity map a 3D point cloud P(x,y,z) representative of the at least one object, wherein the point cloud is configured to support first and second virtual cameras to create a stereo camera pair arrangeable in substantially arbitrary virtual locations.
2. The method of claim 1 , further comprising forming the disparity map using photogrammetric triangulation operation on the at least first and second 2D images.
3. The method of claim 1 , further comprising:
defining the first and second virtual cameras for the 3D point cloud to define a stereo camera pair.
4. The method of claim 1 , further comprising:
defining one or more regions of interest in one of the at least first and second 2D images;
forming the disparity map for the one or more regions of interest;
transforming the one or more regions of interest to conform to the other of the least first and second 2D images; and
forming the point cloud to include the one or more regions of interest.
5. The method of claim 1 , further comprising:
calibrating the at least first and second cameras for one or more camera parameters.
6. The method of claim 5 , further comprising:
performing a registration operation for the at least first and second cameras relative to a reference coordinate system.
7. The method of claim 3 , further comprising:
combining the point cloud with at least one visual effect.
8. The method of claim 7 , further comprising forming the at least one visual effect as:
a) one or more computer-generated objects;
c) one or more animated objects; and
b) one or more computer-generated environments.
9. The method of claim 7 , further comprising:
forming a 3D stereo movie from the combined point cloud and the at least-one visual effect.
10. The method of claim 7 , wherein the 3D stereo movie includes a plurality of 3D stereo frames, and further comprising providing different positions for the first and second virtual cameras for at least two of the frames.
11. The method of claim 7 , further comprising defining for the first and second virtual camera an interocular distance and convergence values for the scene, thereby creating a true left and a true right eye for the stereo pair for theatrical viewing of the 3D movie.
12. The method of claim 1 , further comprising:
supporting the at least first and second cameras on a camera support device.
13. The method of claim 1 , further comprising:
collecting an amount of image data in respective removable memories respectively operably arranged in the at least first and second cameras; and
downloading the image data from the removable memories to a memory module.
14. The method of claim 1 , further comprising employing three cameras, and using one of the three cameras as a cinemagraphic camera and the other two cameras for capturing spatial date of the scene.
15. A method of forming a distance representation of a scene from two-dimensional (2D) images of the scene, comprising:
simultaneously capturing at least first and second 2D images of the scene from corresponding at least first and second cameras having respective camera positions and orientations measured relative to a reference coordinate system;
defining one or more regions of interest in the at least first and second 2D images;
associating differences between pixels in the at least first and second cameras with distances from a reference point; and
assigning different gray-scale intensities to different ones of the distances.
16. The method of claim 15 , further comprising transforming the at least one or more regions of interest from the at least first and second 2D images so that like regions of interest in the different images match despite being taken with different cameras at different orientations.
17. The method of claim 16 , further comprising:
forming from the disparity map a 3D point cloud P (x,y,z) representing spatial positions of points in three-space that make up the 3D scene.
18. The method of claim 17 , further comprising defining a stereo pair of first and second virtual cameras.
19. The method of claim 18 , further comprising forming a 3D stereo movie from the 3D point cloud and the stereo pair of first and second virtual cameras.
20. The method of claim 15 , further comprising forming from the disparity map a polygonal mesh PM(x,y,z;RGB) of spatial positions and corresponding RGB color.
21. The method of claim 20 , further comprising:
mapping 2D photographed images on to the polygonal mesh to form a textured 3D geometry;
defining first and second virtual cameras for the polygonal mesh; and
re-imaging the textured 3D geometry with the first and second virtual cameras to form a 3D movie, including setting an interocular distance and convergence values for the scene, thereby creating a true left and a true right eye stereo pair for theatrical viewing of the 3D movie.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/932,789 US20110222757A1 (en) | 2010-03-10 | 2011-03-07 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US14/089,300 US8867827B2 (en) | 2010-03-10 | 2013-11-25 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US14/481,949 US20140375775A1 (en) | 2010-03-10 | 2014-09-10 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US31233010P | 2010-03-10 | 2010-03-10 | |
US12/932,789 US20110222757A1 (en) | 2010-03-10 | 2011-03-07 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/089,300 Continuation US8867827B2 (en) | 2010-03-10 | 2013-11-25 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
Publications (1)
Publication Number | Publication Date |
---|---|
US20110222757A1 true US20110222757A1 (en) | 2011-09-15 |
Family
ID=44560007
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/932,789 Abandoned US20110222757A1 (en) | 2010-03-10 | 2011-03-07 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US14/089,300 Expired - Fee Related US8867827B2 (en) | 2010-03-10 | 2013-11-25 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US14/481,949 Abandoned US20140375775A1 (en) | 2010-03-10 | 2014-09-10 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/089,300 Expired - Fee Related US8867827B2 (en) | 2010-03-10 | 2013-11-25 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US14/481,949 Abandoned US20140375775A1 (en) | 2010-03-10 | 2014-09-10 | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
Country Status (1)
Country | Link |
---|---|
US (3) | US20110222757A1 (en) |
Cited By (114)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102592284A (en) * | 2012-02-27 | 2012-07-18 | 上海交通大学 | Method for transforming part surface appearance three-dimensional high-density point cloud data into grayscale image |
CN102750730A (en) * | 2012-06-15 | 2012-10-24 | 北京理工大学 | Characteristic-maintained point cloud data compacting method |
US20130063363A1 (en) * | 2011-09-09 | 2013-03-14 | Dreamworks Animation Llc | Minimal parallax coincident digital drawing and display surface |
WO2013052089A1 (en) * | 2011-10-03 | 2013-04-11 | Multiple Interocular 3-D, L.L.C. | Stereoscopic three-dimensional camera rigs |
US20130095920A1 (en) * | 2011-10-13 | 2013-04-18 | Microsoft Corporation | Generating free viewpoint video using stereo imaging |
US20130135441A1 (en) * | 2011-11-28 | 2013-05-30 | Hui Deng | Image Depth Recovering Method and Stereo Image Fetching Device thereof |
US20130141433A1 (en) * | 2011-12-02 | 2013-06-06 | Per Astrand | Methods, Systems and Computer Program Products for Creating Three Dimensional Meshes from Two Dimensional Images |
US8485430B2 (en) | 2011-12-06 | 2013-07-16 | Honeywell International, Inc. | Hand held bar code readers or mobile computers with cloud computing services |
US20130286004A1 (en) * | 2012-04-27 | 2013-10-31 | Daniel J. McCulloch | Displaying a collision between real and virtual objects |
US20130293681A1 (en) * | 2010-12-23 | 2013-11-07 | Fastree3D S.A. | 2d/3d real-time imager and corresponding imaging methods |
CN103595935A (en) * | 2013-10-15 | 2014-02-19 | 深圳市掌网立体时代视讯技术有限公司 | Dynamic compensation method and system for three-dimensional video image dead pixels |
CN103679127A (en) * | 2012-09-24 | 2014-03-26 | 株式会社理光 | Method and device for detecting drivable area of road pavement |
US8754929B1 (en) * | 2011-05-23 | 2014-06-17 | John Prince | Real time vergence control for 3D video capture and display |
US20140210940A1 (en) * | 2013-01-31 | 2014-07-31 | Navteq B.V. | Stereo panoramic images |
CN104008189A (en) * | 2014-06-11 | 2014-08-27 | 义乌市勘测设计研究院 | Measurable street view establishing method with automatic information pushing function |
US20140307058A1 (en) * | 2013-04-15 | 2014-10-16 | Microsoft Corporation | Robust stereo depth system |
US8908041B2 (en) | 2013-01-15 | 2014-12-09 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US20140362186A1 (en) * | 2012-01-04 | 2014-12-11 | The Trustees Of Dartmouth College | Method and apparatus for calibration of stereo-optical three-dimensional surface-mapping system |
WO2015017855A1 (en) | 2013-08-02 | 2015-02-05 | Xactware Solutions, Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US9064254B2 (en) | 2012-05-17 | 2015-06-23 | Honeywell International Inc. | Cloud-based system for reading of decodable indicia |
US9086582B1 (en) | 2014-08-20 | 2015-07-21 | David Kind, Inc. | System and method of providing custom-fitted and styled eyewear based on user-provided images and preferences |
US9092683B2 (en) | 2012-07-10 | 2015-07-28 | Honeywell International Inc. | Cloud-based system for processing of decodable indicia |
US20150279038A1 (en) * | 2014-04-01 | 2015-10-01 | Gopro, Inc. | Image Sensor Read Window Adjustment for Multi-Camera Array Tolerance |
US20150317412A1 (en) * | 2014-05-05 | 2015-11-05 | Microsoft Corporation | Fabricating three-dimensional objects with embossing |
WO2015142412A3 (en) * | 2014-03-20 | 2015-11-12 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US9197885B2 (en) | 2014-03-20 | 2015-11-24 | Gopro, Inc. | Target-less auto-alignment of image sensors in a multi-camera system |
CN105191287A (en) * | 2013-03-08 | 2015-12-23 | 吉恩-鲁克·埃法蒂卡迪 | Method of replacing objects in a video stream and computer program |
WO2016022008A1 (en) * | 2014-08-08 | 2016-02-11 | Samsung Electronics Co., Ltd. | Method and apparatus for environmental profile generation |
US20160057406A1 (en) * | 2013-04-09 | 2016-02-25 | Vit | Three-dimensional image acquisition system |
US20160065858A1 (en) * | 2014-09-03 | 2016-03-03 | Fuji Xerox Co., Ltd. | Methods and systems for sharing views |
US9325917B2 (en) | 2014-03-20 | 2016-04-26 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US20160119541A1 (en) * | 2014-10-24 | 2016-04-28 | Bounce Imaging, Inc. | Imaging systems and methods |
US9336592B2 (en) | 2012-02-03 | 2016-05-10 | The Trustees Of Dartmouth College | Method and apparatus for determining tumor shift during surgery using a stereo-optical three-dimensional surface-mapping system |
GB2497517B (en) * | 2011-12-06 | 2016-05-25 | Toshiba Res Europe Ltd | A reconstruction system and method |
US20160173757A1 (en) * | 2014-12-15 | 2016-06-16 | Samsung Electro-Mechanics Co., Ltd. | Camera module |
US9406343B1 (en) * | 2015-01-26 | 2016-08-02 | Bryan T. Evans | Method of tracking for animation insertions to video recordings |
US9501700B2 (en) | 2012-02-15 | 2016-11-22 | Xactware Solutions, Inc. | System and method for construction estimation using aerial images |
US9558386B2 (en) | 2012-05-15 | 2017-01-31 | Honeywell International, Inc. | Encoded information reading terminal configured to pre-process images |
US9563958B1 (en) | 2015-07-02 | 2017-02-07 | Yesvideo, Inc. | 2D/3D image scanning and compositing |
AT517656A1 (en) * | 2015-08-20 | 2017-03-15 | Ait Austrian Inst Of Tech G M B H | Photometric Stereomatching |
WO2017112905A1 (en) * | 2015-12-22 | 2017-06-29 | Google Inc. | Capture and render of virtual reality content employing a light field camera array |
US9697604B2 (en) | 2014-01-28 | 2017-07-04 | Altek Semiconductor Corp. | Image capturing device and method for detecting image deformation thereof |
US20170214899A1 (en) * | 2014-07-23 | 2017-07-27 | Metaio Gmbh | Method and system for presenting at least part of an image of a real object in a view of a real environment, and method and system for selecting a subset of a plurality of images |
JP2017535805A (en) * | 2014-10-02 | 2017-11-30 | シゼイ シジブイ カンパニー リミテッド | Multi-camera rig |
US20170359561A1 (en) * | 2016-06-08 | 2017-12-14 | Uber Technologies, Inc. | Disparity mapping for an autonomous vehicle |
US20170374360A1 (en) * | 2016-06-28 | 2017-12-28 | Magic Leap, Inc. | Camera calibration system, target, and process |
US9858673B2 (en) | 2012-08-21 | 2018-01-02 | Fotonation Cayman Limited | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints |
US9864921B2 (en) | 2011-09-28 | 2018-01-09 | Fotonation Cayman Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
US9888194B2 (en) | 2013-03-13 | 2018-02-06 | Fotonation Cayman Limited | Array camera architecture implementing quantum film image sensors |
US9898856B2 (en) | 2013-09-27 | 2018-02-20 | Fotonation Cayman Limited | Systems and methods for depth-assisted perspective distortion correction |
US9917998B2 (en) | 2013-03-08 | 2018-03-13 | Fotonation Cayman Limited | Systems and methods for measuring scene information while capturing images using array cameras |
US9924092B2 (en) | 2013-11-07 | 2018-03-20 | Fotonation Cayman Limited | Array cameras incorporating independently aligned lens stacks |
US9936148B2 (en) | 2010-05-12 | 2018-04-03 | Fotonation Cayman Limited | Imager array interfaces |
US9955070B2 (en) | 2013-03-15 | 2018-04-24 | Fotonation Cayman Limited | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US9986224B2 (en) | 2013-03-10 | 2018-05-29 | Fotonation Cayman Limited | System and methods for calibration of an array camera |
US10009538B2 (en) | 2013-02-21 | 2018-06-26 | Fotonation Cayman Limited | Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information |
WO2018125369A1 (en) * | 2016-12-30 | 2018-07-05 | Google Llc | Multi-view scene flow stitching |
US10027901B2 (en) | 2008-05-20 | 2018-07-17 | Fotonation Cayman Limited | Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras |
EP3358846A1 (en) * | 2017-02-06 | 2018-08-08 | Robo-Team Home Ltd. | Method and device for stereoscopic vision |
US10089740B2 (en) | 2014-03-07 | 2018-10-02 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
US10091405B2 (en) | 2013-03-14 | 2018-10-02 | Fotonation Cayman Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US10119808B2 (en) | 2013-11-18 | 2018-11-06 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
CN108810571A (en) * | 2017-05-04 | 2018-11-13 | 汤姆逊许可公司 | The method and apparatus for coding and decoding two-dimensional points cloud |
US10127682B2 (en) | 2013-03-13 | 2018-11-13 | Fotonation Limited | System and methods for calibration of an array camera |
US10142560B2 (en) | 2008-05-20 | 2018-11-27 | Fotonation Limited | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US10182216B2 (en) | 2013-03-15 | 2019-01-15 | Fotonation Limited | Extended color processing on pelican array cameras |
US10218889B2 (en) | 2011-05-11 | 2019-02-26 | Fotonation Limited | Systems and methods for transmitting and receiving array camera image data |
US10250871B2 (en) | 2014-09-29 | 2019-04-02 | Fotonation Limited | Systems and methods for dynamic calibration of array cameras |
US10261219B2 (en) | 2012-06-30 | 2019-04-16 | Fotonation Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US10306120B2 (en) | 2009-11-20 | 2019-05-28 | Fotonation Limited | Capturing and processing of images captured by camera arrays incorporating cameras with telephoto and conventional lenses to generate depth maps |
US10311649B2 (en) | 2012-02-21 | 2019-06-04 | Fotonation Limited | Systems and method for performing depth based image editing |
US10334241B2 (en) | 2012-06-28 | 2019-06-25 | Fotonation Limited | Systems and methods for detecting defective camera arrays and optic arrays |
US20190219902A1 (en) * | 2016-09-28 | 2019-07-18 | Jacek LIPIK | Scanner, specifically for scanning antique books, and a method of scanning |
CN110060291A (en) * | 2019-04-04 | 2019-07-26 | 长春理工大学 | It is a kind of consider people because stereopsis in distance calculation method |
US10366472B2 (en) | 2010-12-14 | 2019-07-30 | Fotonation Limited | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US10375302B2 (en) | 2011-09-19 | 2019-08-06 | Fotonation Limited | Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures |
US10390005B2 (en) | 2012-09-28 | 2019-08-20 | Fotonation Limited | Generating images from light fields utilizing virtual viewpoints |
US10412368B2 (en) * | 2013-03-15 | 2019-09-10 | Uber Technologies, Inc. | Methods, systems, and apparatus for multi-sensory stereo vision for robotics |
US10455218B2 (en) | 2013-03-15 | 2019-10-22 | Fotonation Limited | Systems and methods for estimating depth using stereo array cameras |
US10462362B2 (en) | 2012-08-23 | 2019-10-29 | Fotonation Limited | Feature based high resolution motion estimation from low resolution images captured using an array source |
US10542247B2 (en) | 2017-12-20 | 2020-01-21 | Wistron Corporation | 3D image capture method and system |
CN110728740A (en) * | 2018-07-13 | 2020-01-24 | 辉达公司 | Virtual photogrammetry |
CN110832547A (en) * | 2017-02-08 | 2020-02-21 | 约夫·舍菲 | System and method for generating stereo paired images of virtual objects |
US10568535B2 (en) | 2008-05-22 | 2020-02-25 | The Trustees Of Dartmouth College | Surgical navigation with stereovision and associated methods |
US10602200B2 (en) | 2014-05-28 | 2020-03-24 | Lucasfilm Entertainment Company Ltd. | Switching modes of a media content item |
US10674138B2 (en) | 2013-03-15 | 2020-06-02 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
CN111340871A (en) * | 2019-12-31 | 2020-06-26 | 东软集团股份有限公司 | Depth map prediction method, depth map prediction device, storage medium and electronic equipment |
US10708492B2 (en) | 2013-11-26 | 2020-07-07 | Fotonation Limited | Array camera configurations incorporating constituent array cameras and constituent cameras |
CN111735451A (en) * | 2020-04-16 | 2020-10-02 | 中国北方车辆研究所 | Point cloud matching high-precision positioning method based on multi-source prior information |
CN111902072A (en) * | 2017-12-01 | 2020-11-06 | 索罗新公司 | System and method for ultrasound tissue screening |
US10967862B2 (en) | 2017-11-07 | 2021-04-06 | Uatc, Llc | Road anomaly detection for autonomous vehicle |
US11012676B2 (en) | 2017-12-13 | 2021-05-18 | Google Llc | Methods, systems, and media for generating and rendering immersive video content |
CN113008265A (en) * | 2012-06-05 | 2021-06-22 | 苹果公司 | Virtual camera for 3D maps |
US11094113B2 (en) | 2019-12-04 | 2021-08-17 | Geomni, Inc. | Systems and methods for modeling structures using point clouds derived from stereoscopic image pairs |
CN113409450A (en) * | 2021-07-09 | 2021-09-17 | 浙江大学 | Three-dimensional reconstruction method for chickens containing RGBDT information |
US11182960B2 (en) * | 2012-05-09 | 2021-11-23 | Ncam Technologies Limited | System for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera |
US20210390724A1 (en) * | 2020-06-15 | 2021-12-16 | Zebra Technologies Corporation | Three-Dimensional Sensor Acuity Recovery Assistance |
US11238603B2 (en) * | 2016-06-08 | 2022-02-01 | Amazon Technologies, Inc. | Selectively paired imaging elements for stereo images |
US11270110B2 (en) | 2019-09-17 | 2022-03-08 | Boston Polarimetrics, Inc. | Systems and methods for surface modeling using polarization cues |
US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
US11302012B2 (en) | 2019-11-30 | 2022-04-12 | Boston Polarimetrics, Inc. | Systems and methods for transparent object segmentation using polarization cues |
WO2022237544A1 (en) * | 2021-05-11 | 2022-11-17 | 梅卡曼德(北京)机器人科技有限公司 | Trajectory generation method and apparatus, and electronic device and storage medium |
US11510600B2 (en) | 2012-01-04 | 2022-11-29 | The Trustees Of Dartmouth College | Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance |
US11525906B2 (en) | 2019-10-07 | 2022-12-13 | Intrinsic Innovation Llc | Systems and methods for augmentation of sensor systems and imaging systems with polarization |
US11564639B2 (en) | 2013-02-13 | 2023-01-31 | The Trustees Of Dartmouth College | Method and apparatus for medical imaging using differencing of multiple fluorophores |
US11580667B2 (en) | 2020-01-29 | 2023-02-14 | Intrinsic Innovation Llc | Systems and methods for characterizing object pose detection and measurement systems |
US11671717B2 (en) * | 2017-10-20 | 2023-06-06 | Lucasfilm Entertainment Company Ltd. | Camera systems for motion capture |
US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US11797863B2 (en) | 2020-01-30 | 2023-10-24 | Intrinsic Innovation Llc | Systems and methods for synthesizing data for training statistical models on different imaging modalities including polarized images |
US11937951B2 (en) | 2013-02-13 | 2024-03-26 | The Trustees Of Dartmouth College | Method and apparatus for medical imaging using differencing of multiple fluorophores |
US11953700B2 (en) | 2020-05-27 | 2024-04-09 | Intrinsic Innovation Llc | Multi-aperture polarization optical systems using beam splitters |
US11954886B2 (en) | 2021-04-15 | 2024-04-09 | Intrinsic Innovation Llc | Systems and methods for six-degree of freedom pose estimation of deformable objects |
US11956609B2 (en) | 2012-06-05 | 2024-04-09 | Apple Inc. | Context-aware voice guidance |
Families Citing this family (43)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2208354A4 (en) | 2007-10-10 | 2010-12-22 | Gerard Dirk Smits | Image projector with reflected light tracking |
US20110222757A1 (en) * | 2010-03-10 | 2011-09-15 | Gbo 3D Technology Pte. Ltd. | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
JP5444505B2 (en) * | 2011-05-03 | 2014-03-19 | オリンパスイメージング株式会社 | Stereoscopic image processing apparatus and stereoscopic image processing method |
EP2808645B1 (en) * | 2012-01-23 | 2019-02-20 | Nec Corporation | Camera calibration device, camera calibration method, and camera calibration program |
KR20140121107A (en) * | 2013-04-05 | 2014-10-15 | 한국전자통신연구원 | Methods and apparatuses of generating hologram based on multi-view |
US11019258B2 (en) | 2013-08-21 | 2021-05-25 | Verizon Patent And Licensing Inc. | Aggregating images and audio data to generate content |
US9451162B2 (en) | 2013-08-21 | 2016-09-20 | Jaunt Inc. | Camera array including camera modules |
US9210321B2 (en) * | 2013-12-05 | 2015-12-08 | Here Global B.V. | Method and apparatus for a shutter animation for image capture |
US9911454B2 (en) | 2014-05-29 | 2018-03-06 | Jaunt Inc. | Camera array including camera modules |
US10368011B2 (en) | 2014-07-25 | 2019-07-30 | Jaunt Inc. | Camera array removing lens distortion |
US11108971B2 (en) | 2014-07-25 | 2021-08-31 | Verzon Patent and Licensing Ine. | Camera array removing lens distortion |
US9774887B1 (en) | 2016-09-19 | 2017-09-26 | Jaunt Inc. | Behavioral directional encoding of three-dimensional video |
US10440398B2 (en) | 2014-07-28 | 2019-10-08 | Jaunt, Inc. | Probabilistic model to compress images for three-dimensional video |
US10186301B1 (en) * | 2014-07-28 | 2019-01-22 | Jaunt Inc. | Camera array including camera modules |
US10701426B1 (en) | 2014-07-28 | 2020-06-30 | Verizon Patent And Licensing Inc. | Virtual reality system including social graph |
US9363569B1 (en) | 2014-07-28 | 2016-06-07 | Jaunt Inc. | Virtual reality system including social graph |
TWI577493B (en) | 2014-12-26 | 2017-04-11 | 財團法人工業技術研究院 | Calibration method and automatic apparatus using the same |
US10511787B2 (en) | 2015-02-12 | 2019-12-17 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. | Light-field camera |
EP3274986A4 (en) * | 2015-03-21 | 2019-04-17 | Mine One GmbH | Virtual 3d methods, systems and software |
US10853625B2 (en) | 2015-03-21 | 2020-12-01 | Mine One Gmbh | Facial signature methods, systems and software |
US10510149B2 (en) | 2015-07-17 | 2019-12-17 | ams Sensors Singapore Pte. Ltd | Generating a distance map based on captured images of a scene |
WO2017023210A1 (en) * | 2015-08-06 | 2017-02-09 | Heptagon Micro Optics Pte. Ltd. | Generating a merged, fused three-dimensional point cloud based on captured images of a scene |
TWI744245B (en) | 2015-08-19 | 2021-11-01 | 新加坡商海特根微光學公司 | Generating a disparity map having reduced over-smoothing |
US9753126B2 (en) | 2015-12-18 | 2017-09-05 | Gerard Dirk Smits | Real time position sensing of objects |
US10721451B2 (en) * | 2016-03-23 | 2020-07-21 | Symbol Technologies, Llc | Arrangement for, and method of, loading freight into a shipping container |
TWI567693B (en) * | 2016-05-17 | 2017-01-21 | 緯創資通股份有限公司 | Method and system for generating depth information |
US20170364868A1 (en) * | 2016-06-17 | 2017-12-21 | Thumbtag India Private Limited | System of attendance and time tracking with reporting |
US10924638B2 (en) * | 2016-06-27 | 2021-02-16 | Intel Corporation | Compact, low cost VCSEL projector for high performance stereodepth camera |
CN116484029A (en) * | 2016-08-12 | 2023-07-25 | 派克赛斯有限责任公司 | System and method for automatically generating metadata for a media document |
CN108541322B (en) * | 2016-08-29 | 2021-09-14 | 北京清影机器视觉技术有限公司 | Method and device for processing three-dimensional vision measurement data |
US11032535B2 (en) | 2016-09-19 | 2021-06-08 | Verizon Patent And Licensing Inc. | Generating a three-dimensional preview of a three-dimensional video |
US10681341B2 (en) | 2016-09-19 | 2020-06-09 | Verizon Patent And Licensing Inc. | Using a sphere to reorient a location of a user in a three-dimensional virtual reality video |
US11032536B2 (en) | 2016-09-19 | 2021-06-08 | Verizon Patent And Licensing Inc. | Generating a three-dimensional preview from a two-dimensional selectable icon of a three-dimensional reality video |
WO2018125850A1 (en) * | 2016-12-27 | 2018-07-05 | Gerard Dirk Smits | Systems and methods for machine perception |
CN108470323B (en) * | 2018-03-13 | 2020-07-31 | 京东方科技集团股份有限公司 | Image splicing method, computer equipment and display device |
CN109493372B (en) * | 2018-10-24 | 2021-10-01 | 华侨大学 | Rapid global optimization registration method for product point cloud data with large data volume and few characteristics |
CN111179152B (en) * | 2018-11-12 | 2023-04-28 | 阿里巴巴集团控股有限公司 | Road identification recognition method and device, medium and terminal |
US10694167B1 (en) | 2018-12-12 | 2020-06-23 | Verizon Patent And Licensing Inc. | Camera array including camera modules |
US10963733B2 (en) * | 2019-06-17 | 2021-03-30 | Tianzhi Yang | Associating spatial point sets with candidate correspondences |
US11164325B2 (en) * | 2020-02-06 | 2021-11-02 | Tianzhi Yang | Generating and evaluating mappings between spatial point sets |
WO2021174227A1 (en) | 2020-02-27 | 2021-09-02 | Gerard Dirk Smits | High resolution scanning of remote objects with fast sweeping laser beams and signal recovery by twitchy pixel array |
CN112037279B (en) * | 2020-09-04 | 2021-11-16 | 贝壳找房(北京)科技有限公司 | Article position identification method and device, storage medium and electronic equipment |
CN113421207A (en) * | 2021-08-23 | 2021-09-21 | 深圳市信润富联数字科技有限公司 | Visual inspection method, apparatus, product and computer storage medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070147671A1 (en) * | 2005-12-22 | 2007-06-28 | Eastman Kodak Company | Analyzing radiological image using 3D stereo pairs |
US20070296809A1 (en) * | 2006-06-13 | 2007-12-27 | Billy Newbery | Digital stereo photographic system |
US20070296721A1 (en) * | 2004-11-08 | 2007-12-27 | Electronics And Telecommunications Research Institute | Apparatus and Method for Producting Multi-View Contents |
US20090003686A1 (en) * | 2005-01-07 | 2009-01-01 | Gesturetek, Inc. | Enhanced object reconstruction |
US20110225523A1 (en) * | 2008-11-24 | 2011-09-15 | Koninklijke Philips Electronics N.V. | Extending 2d graphics in a 3d gui |
US20110316980A1 (en) * | 2008-12-22 | 2011-12-29 | Nederlandse Organisatie voor toegepastnatuurweten schappelijk Onderzoek TNO | Method of estimating a motion of a multiple camera system, a multiple camera system and a computer program product |
Family Cites Families (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6028348A (en) | 1993-11-30 | 2000-02-22 | Texas Instruments Incorporated | Low thermal impedance integrated circuit |
US6985168B2 (en) | 1994-11-14 | 2006-01-10 | Reveo, Inc. | Intelligent method and system for producing and displaying stereoscopically-multiplexed images of three-dimensional objects for use in realistic stereoscopic viewing thereof in interactive virtual reality display environments |
US5703961A (en) | 1994-12-29 | 1997-12-30 | Worldscape L.L.C. | Image transformation and synthesis methods |
US5850352A (en) | 1995-03-31 | 1998-12-15 | The Regents Of The University Of California | Immersive video, including video hypermosaicing to generate from multiple video views of a scene a three-dimensional video mosaic from which diverse virtual video scene images are synthesized, including panoramic, scene interactive and stereoscopic images |
US5729471A (en) | 1995-03-31 | 1998-03-17 | The Regents Of The University Of California | Machine dynamic selection of one video camera/image of a scene from multiple video cameras/images of the scene in accordance with a particular perspective on the scene, an object in the scene, or an event in the scene |
US5675377A (en) | 1995-06-30 | 1997-10-07 | Telefonaktiebolaget Lm Ericsson | True three-dimensional imaging and display system |
US6084979A (en) | 1996-06-20 | 2000-07-04 | Carnegie Mellon University | Method for creating virtual reality |
US6002743A (en) | 1996-07-17 | 1999-12-14 | Telymonde; Timothy D. | Method and apparatus for image acquisition from a plurality of cameras |
EP2252071A3 (en) | 1997-12-05 | 2017-04-12 | Dynamic Digital Depth Research Pty. Ltd. | Improved image conversion and encoding techniques |
US6515659B1 (en) | 1998-05-27 | 2003-02-04 | In-Three, Inc. | Method and system for creating realistic smooth three-dimensional depth contours from two-dimensional images |
US6208348B1 (en) | 1998-05-27 | 2001-03-27 | In-Three, Inc. | System and method for dimensionalization processing of images in consideration of a pedetermined image projection format |
US7116323B2 (en) | 1998-05-27 | 2006-10-03 | In-Three, Inc. | Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional images |
US7116324B2 (en) | 1998-05-27 | 2006-10-03 | In-Three, Inc. | Method for minimizing visual artifacts converting two-dimensional motion pictures into three-dimensional motion pictures |
US7102633B2 (en) | 1998-05-27 | 2006-09-05 | In-Three, Inc. | Method for conforming objects to a common depth perspective for converting two-dimensional images into three-dimensional images |
US6268846B1 (en) | 1998-06-22 | 2001-07-31 | Adobe Systems Incorporated | 3D graphics based on images and morphing |
US20030026460A1 (en) | 1999-05-12 | 2003-02-06 | Conrad Gary W. | Method for producing a three-dimensional object which can be tactilely sensed, and the resultant object |
GB9913687D0 (en) | 1999-06-11 | 1999-08-11 | Canon Kk | Image processing apparatus |
US6791598B1 (en) | 2000-03-17 | 2004-09-14 | International Business Machines Corporation | Methods and apparatus for information capture and steroscopic display of panoramic images |
US6677982B1 (en) | 2000-10-11 | 2004-01-13 | Eastman Kodak Company | Method for three dimensional spatial panorama formation |
US7085409B2 (en) | 2000-10-18 | 2006-08-01 | Sarnoff Corporation | Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery |
US7298964B2 (en) | 2001-02-26 | 2007-11-20 | Matsushita Electric Industrial Co., Ltd. | Recording system, video camera device and video image recording method |
US6816629B2 (en) | 2001-09-07 | 2004-11-09 | Realty Mapping Llc | Method and system for 3-D content creation |
US7058239B2 (en) | 2001-10-29 | 2006-06-06 | Eyesee360, Inc. | System and method for panoramic imaging |
KR100433625B1 (en) | 2001-11-17 | 2004-06-02 | 학교법인 포항공과대학교 | Apparatus for reconstructing multiview image using stereo image and depth map |
US20030202120A1 (en) | 2002-04-05 | 2003-10-30 | Mack Newton Eliot | Virtual lighting system |
US7307654B2 (en) | 2002-10-31 | 2007-12-11 | Hewlett-Packard Development Company, L.P. | Image capture and viewing system and method for generating a synthesized image |
WO2006083297A2 (en) | 2004-06-10 | 2006-08-10 | Sarnoff Corporation | Method and apparatus for aligning video to three-dimensional point clouds |
US7292257B2 (en) | 2004-06-28 | 2007-11-06 | Microsoft Corporation | Interactive viewpoint video system and process |
CN101065785B (en) | 2004-08-30 | 2013-01-23 | 联邦科学和工业研究组织 | A method for automated 3D imaging |
WO2007032819A2 (en) | 2005-07-21 | 2007-03-22 | Regents Of The University Of Minnesota | Camera placement and virtual-scene construction |
US7856125B2 (en) | 2006-01-31 | 2010-12-21 | University Of Southern California | 3D face reconstruction from 2D images |
US7573475B2 (en) | 2006-06-01 | 2009-08-11 | Industrial Light & Magic | 2D to 3D image conversion |
SG138491A1 (en) * | 2006-06-21 | 2008-01-28 | Generic Power Pte Ltd | Method and apparatus for 3-dimensional vision and inspection of ball and like protrusions of electronic components |
CN101479765B (en) | 2006-06-23 | 2012-05-23 | 图象公司 | Methods and systems for converting 2d motion pictures for stereoscopic 3d exhibition |
US20080158345A1 (en) | 2006-09-11 | 2008-07-03 | 3Ality Digital Systems, Llc | 3d augmentation of traditional photography |
US20100118117A1 (en) | 2006-10-26 | 2010-05-13 | Seereal Technologies S.A. | 3d content generation system |
EP2084491A2 (en) * | 2006-11-21 | 2009-08-05 | Mantisvision Ltd. | 3d geometric modeling and 3d video content creation |
US8090194B2 (en) * | 2006-11-21 | 2012-01-03 | Mantis Vision Ltd. | 3D geometric modeling and motion capture using both single and dual imaging |
JP5366824B2 (en) | 2006-12-19 | 2013-12-11 | コーニンクレッカ フィリップス エヌ ヴェ | Method and system for converting 2D video to 3D video |
US8330801B2 (en) | 2006-12-22 | 2012-12-11 | Qualcomm Incorporated | Complexity-adaptive 2D-to-3D video sequence conversion |
JP4750859B2 (en) * | 2007-01-16 | 2011-08-17 | パナソニック株式会社 | Data processing apparatus, method, and recording medium |
GB2456802A (en) | 2008-01-24 | 2009-07-29 | Areograph Ltd | Image capture and motion picture generation using both motion camera and scene scanning imaging systems |
WO2009101798A1 (en) * | 2008-02-12 | 2009-08-20 | Panasonic Corporation | Compound eye imaging device, distance measurement device, parallax calculation method and distance measurement method |
AU2009201637B2 (en) * | 2009-04-24 | 2011-08-11 | Canon Kabushiki Kaisha | Processing multi-view digital images |
KR20100135032A (en) | 2009-06-16 | 2010-12-24 | 삼성전자주식회사 | Conversion device for two dimensional image to three dimensional image and method thereof |
US8553972B2 (en) | 2009-07-06 | 2013-10-08 | Samsung Electronics Co., Ltd. | Apparatus, method and computer-readable medium generating depth map |
US20110222757A1 (en) * | 2010-03-10 | 2011-09-15 | Gbo 3D Technology Pte. Ltd. | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
-
2011
- 2011-03-07 US US12/932,789 patent/US20110222757A1/en not_active Abandoned
-
2013
- 2013-11-25 US US14/089,300 patent/US8867827B2/en not_active Expired - Fee Related
-
2014
- 2014-09-10 US US14/481,949 patent/US20140375775A1/en not_active Abandoned
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070296721A1 (en) * | 2004-11-08 | 2007-12-27 | Electronics And Telecommunications Research Institute | Apparatus and Method for Producting Multi-View Contents |
US20090003686A1 (en) * | 2005-01-07 | 2009-01-01 | Gesturetek, Inc. | Enhanced object reconstruction |
US20070147671A1 (en) * | 2005-12-22 | 2007-06-28 | Eastman Kodak Company | Analyzing radiological image using 3D stereo pairs |
US20070296809A1 (en) * | 2006-06-13 | 2007-12-27 | Billy Newbery | Digital stereo photographic system |
US20110225523A1 (en) * | 2008-11-24 | 2011-09-15 | Koninklijke Philips Electronics N.V. | Extending 2d graphics in a 3d gui |
US20110316980A1 (en) * | 2008-12-22 | 2011-12-29 | Nederlandse Organisatie voor toegepastnatuurweten schappelijk Onderzoek TNO | Method of estimating a motion of a multiple camera system, a multiple camera system and a computer program product |
Non-Patent Citations (3)
Title |
---|
Gill et al., "Polygonal meshing for 3D stereo video sensor data", Canadian Conference on Computer and Robot Vision, 2008 * |
Izquierdo et al., "Image-based rendering and 3D modeling: A complete framework", Signal Processing: Image Communications 15 (2000) 817-858, 2000 * |
Matthies et al., "A photo-realistic 3-D mapping system for extreme nuclear environments: Chornobyl", Proc of the 1998 IEEE/RSJ, Intl. Conf on Intelligent Robots and Systems, 1998 * |
Cited By (228)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10027901B2 (en) | 2008-05-20 | 2018-07-17 | Fotonation Cayman Limited | Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras |
US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US10142560B2 (en) | 2008-05-20 | 2018-11-27 | Fotonation Limited | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US11412158B2 (en) | 2008-05-20 | 2022-08-09 | Fotonation Limited | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US11129562B2 (en) | 2008-05-22 | 2021-09-28 | The Trustees Of Dartmouth College | Surgical navigation with stereovision and associated methods |
US10568535B2 (en) | 2008-05-22 | 2020-02-25 | The Trustees Of Dartmouth College | Surgical navigation with stereovision and associated methods |
US10306120B2 (en) | 2009-11-20 | 2019-05-28 | Fotonation Limited | Capturing and processing of images captured by camera arrays incorporating cameras with telephoto and conventional lenses to generate depth maps |
US10455168B2 (en) | 2010-05-12 | 2019-10-22 | Fotonation Limited | Imager array interfaces |
US9936148B2 (en) | 2010-05-12 | 2018-04-03 | Fotonation Cayman Limited | Imager array interfaces |
US11875475B2 (en) | 2010-12-14 | 2024-01-16 | Adeia Imaging Llc | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US11423513B2 (en) | 2010-12-14 | 2022-08-23 | Fotonation Limited | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US10366472B2 (en) | 2010-12-14 | 2019-07-30 | Fotonation Limited | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US20130293681A1 (en) * | 2010-12-23 | 2013-11-07 | Fastree3D S.A. | 2d/3d real-time imager and corresponding imaging methods |
US9392259B2 (en) * | 2010-12-23 | 2016-07-12 | Fastree3D S.A. | 2D/3D real-time imager and corresponding imaging methods |
US10218889B2 (en) | 2011-05-11 | 2019-02-26 | Fotonation Limited | Systems and methods for transmitting and receiving array camera image data |
US10742861B2 (en) | 2011-05-11 | 2020-08-11 | Fotonation Limited | Systems and methods for transmitting and receiving array camera image data |
US8754929B1 (en) * | 2011-05-23 | 2014-06-17 | John Prince | Real time vergence control for 3D video capture and display |
US8497852B2 (en) * | 2011-09-09 | 2013-07-30 | Dreamworks Animation Llc | Minimal parallax coincident digital drawing and display surface |
US20130063363A1 (en) * | 2011-09-09 | 2013-03-14 | Dreamworks Animation Llc | Minimal parallax coincident digital drawing and display surface |
US10375302B2 (en) | 2011-09-19 | 2019-08-06 | Fotonation Limited | Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures |
US11729365B2 (en) | 2011-09-28 | 2023-08-15 | Adela Imaging LLC | Systems and methods for encoding image files containing depth maps stored as metadata |
US10430682B2 (en) | 2011-09-28 | 2019-10-01 | Fotonation Limited | Systems and methods for decoding image files containing depth maps stored as metadata |
US20180197035A1 (en) | 2011-09-28 | 2018-07-12 | Fotonation Cayman Limited | Systems and Methods for Encoding Image Files Containing Depth Maps Stored as Metadata |
US10984276B2 (en) | 2011-09-28 | 2021-04-20 | Fotonation Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
US9864921B2 (en) | 2011-09-28 | 2018-01-09 | Fotonation Cayman Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
US10019816B2 (en) | 2011-09-28 | 2018-07-10 | Fotonation Cayman Limited | Systems and methods for decoding image files containing depth maps stored as metadata |
US10275676B2 (en) | 2011-09-28 | 2019-04-30 | Fotonation Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
WO2013052089A1 (en) * | 2011-10-03 | 2013-04-11 | Multiple Interocular 3-D, L.L.C. | Stereoscopic three-dimensional camera rigs |
EP2766875A4 (en) * | 2011-10-13 | 2014-08-20 | Microsoft Corp | Generating free viewpoint video using stereo imaging |
EP2766875A1 (en) * | 2011-10-13 | 2014-08-20 | Microsoft Corporation | Generating free viewpoint video using stereo imaging |
US20130095920A1 (en) * | 2011-10-13 | 2013-04-18 | Microsoft Corporation | Generating free viewpoint video using stereo imaging |
US9661310B2 (en) * | 2011-11-28 | 2017-05-23 | ArcSoft Hanzhou Co., Ltd. | Image depth recovering method and stereo image fetching device thereof |
US20130135441A1 (en) * | 2011-11-28 | 2013-05-30 | Hui Deng | Image Depth Recovering Method and Stereo Image Fetching Device thereof |
US20130141433A1 (en) * | 2011-12-02 | 2013-06-06 | Per Astrand | Methods, Systems and Computer Program Products for Creating Three Dimensional Meshes from Two Dimensional Images |
US9529902B2 (en) | 2011-12-06 | 2016-12-27 | Honeywell International | Hand held bar code readers or mobile computers with cloud computing services |
US8485430B2 (en) | 2011-12-06 | 2013-07-16 | Honeywell International, Inc. | Hand held bar code readers or mobile computers with cloud computing services |
US9223860B2 (en) | 2011-12-06 | 2015-12-29 | Hand Held Products, Inc. | Hand held bar code readers or mobile computers with cloud computing services |
US8820630B2 (en) | 2011-12-06 | 2014-09-02 | Honeywell International, Inc. | Hand held bar code readers or mobile computers with cloud computing services |
GB2497517B (en) * | 2011-12-06 | 2016-05-25 | Toshiba Res Europe Ltd | A reconstruction system and method |
US11510600B2 (en) | 2012-01-04 | 2022-11-29 | The Trustees Of Dartmouth College | Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance |
US11857317B2 (en) | 2012-01-04 | 2024-01-02 | The Trustees Of Dartmouth College | Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance |
US9456200B2 (en) * | 2012-01-04 | 2016-09-27 | The Trustees Of Dartmouth College | Method and apparatus for calibration of stereo-optical three-dimensional surface-mapping system |
US20140362186A1 (en) * | 2012-01-04 | 2014-12-11 | The Trustees Of Dartmouth College | Method and apparatus for calibration of stereo-optical three-dimensional surface-mapping system |
US9336592B2 (en) | 2012-02-03 | 2016-05-10 | The Trustees Of Dartmouth College | Method and apparatus for determining tumor shift during surgery using a stereo-optical three-dimensional surface-mapping system |
US11210433B2 (en) | 2012-02-15 | 2021-12-28 | Xactware Solutions, Inc. | System and method for construction estimation using aerial images |
US9501700B2 (en) | 2012-02-15 | 2016-11-22 | Xactware Solutions, Inc. | System and method for construction estimation using aerial images |
US10503842B2 (en) | 2012-02-15 | 2019-12-10 | Xactware Solutions, Inc. | System and method for construction estimation using aerial images |
US11727163B2 (en) | 2012-02-15 | 2023-08-15 | Xactware Solutions, Inc. | System and method for construction estimation using aerial images |
US10311649B2 (en) | 2012-02-21 | 2019-06-04 | Fotonation Limited | Systems and method for performing depth based image editing |
CN102592284A (en) * | 2012-02-27 | 2012-07-18 | 上海交通大学 | Method for transforming part surface appearance three-dimensional high-density point cloud data into grayscale image |
US20130286004A1 (en) * | 2012-04-27 | 2013-10-31 | Daniel J. McCulloch | Displaying a collision between real and virtual objects |
US9183676B2 (en) * | 2012-04-27 | 2015-11-10 | Microsoft Technology Licensing, Llc | Displaying a collision between real and virtual objects |
US11721076B2 (en) * | 2012-05-09 | 2023-08-08 | Ncam Technologies Limited | System for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera |
US11182960B2 (en) * | 2012-05-09 | 2021-11-23 | Ncam Technologies Limited | System for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera |
US20220076501A1 (en) * | 2012-05-09 | 2022-03-10 | Ncam Technologies Limited | A system for mixing or compositing in real-time, computer generated 3d objects and a video feed from a film camera |
US11727231B2 (en) | 2012-05-15 | 2023-08-15 | Honeywell International Inc. | Encoded information reading terminal configured to pre-process images |
US10885291B2 (en) | 2012-05-15 | 2021-01-05 | Honeywell International Inc. | Encoded information reading terminal configured to pre-process images |
US9558386B2 (en) | 2012-05-15 | 2017-01-31 | Honeywell International, Inc. | Encoded information reading terminal configured to pre-process images |
US11301661B2 (en) | 2012-05-15 | 2022-04-12 | Honeywell International Inc. | Encoded information reading terminal configured to pre-process images |
US9398008B2 (en) | 2012-05-17 | 2016-07-19 | Honeywell International Inc. | Cloud-based system for reading of decodable indicia |
US9064254B2 (en) | 2012-05-17 | 2015-06-23 | Honeywell International Inc. | Cloud-based system for reading of decodable indicia |
US9665655B2 (en) | 2012-05-17 | 2017-05-30 | Honeywell International, Inc. | Cloud-based system for reading of decodable indicia |
CN113008265A (en) * | 2012-06-05 | 2021-06-22 | 苹果公司 | Virtual camera for 3D maps |
US11956609B2 (en) | 2012-06-05 | 2024-04-09 | Apple Inc. | Context-aware voice guidance |
CN102750730A (en) * | 2012-06-15 | 2012-10-24 | 北京理工大学 | Characteristic-maintained point cloud data compacting method |
US10334241B2 (en) | 2012-06-28 | 2019-06-25 | Fotonation Limited | Systems and methods for detecting defective camera arrays and optic arrays |
US11022725B2 (en) | 2012-06-30 | 2021-06-01 | Fotonation Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US10261219B2 (en) | 2012-06-30 | 2019-04-16 | Fotonation Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US9092683B2 (en) | 2012-07-10 | 2015-07-28 | Honeywell International Inc. | Cloud-based system for processing of decodable indicia |
US9755703B2 (en) | 2012-07-10 | 2017-09-05 | Honeywell International, Inc. | Cloud-based system for processing of decodable indicia |
US9858673B2 (en) | 2012-08-21 | 2018-01-02 | Fotonation Cayman Limited | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints |
US10380752B2 (en) | 2012-08-21 | 2019-08-13 | Fotonation Limited | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints |
US10462362B2 (en) | 2012-08-23 | 2019-10-29 | Fotonation Limited | Feature based high resolution motion estimation from low resolution images captured using an array source |
CN103679127A (en) * | 2012-09-24 | 2014-03-26 | 株式会社理光 | Method and device for detecting drivable area of road pavement |
US10390005B2 (en) | 2012-09-28 | 2019-08-20 | Fotonation Limited | Generating images from light fields utilizing virtual viewpoints |
US9286522B2 (en) | 2013-01-15 | 2016-03-15 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US8908041B2 (en) | 2013-01-15 | 2014-12-09 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US10764517B2 (en) | 2013-01-15 | 2020-09-01 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US10200638B2 (en) | 2013-01-15 | 2019-02-05 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US9531966B2 (en) | 2013-01-15 | 2016-12-27 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US9854185B2 (en) | 2013-01-15 | 2017-12-26 | Mobileye Vision Technologies Ltd. | Stereo assist with rolling shutters |
US9924156B2 (en) | 2013-01-31 | 2018-03-20 | Here Global B.V. | Stereo panoramic images |
US20140210940A1 (en) * | 2013-01-31 | 2014-07-31 | Navteq B.V. | Stereo panoramic images |
US9215448B2 (en) * | 2013-01-31 | 2015-12-15 | Here Global B.V. | Stereo panoramic images |
US11564639B2 (en) | 2013-02-13 | 2023-01-31 | The Trustees Of Dartmouth College | Method and apparatus for medical imaging using differencing of multiple fluorophores |
US11937951B2 (en) | 2013-02-13 | 2024-03-26 | The Trustees Of Dartmouth College | Method and apparatus for medical imaging using differencing of multiple fluorophores |
US10009538B2 (en) | 2013-02-21 | 2018-06-26 | Fotonation Cayman Limited | Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information |
US10205889B2 (en) | 2013-03-08 | 2019-02-12 | Digitarena Sa | Method of replacing objects in a video stream and computer program |
CN105191287A (en) * | 2013-03-08 | 2015-12-23 | 吉恩-鲁克·埃法蒂卡迪 | Method of replacing objects in a video stream and computer program |
US9917998B2 (en) | 2013-03-08 | 2018-03-13 | Fotonation Cayman Limited | Systems and methods for measuring scene information while capturing images using array cameras |
US10225543B2 (en) | 2013-03-10 | 2019-03-05 | Fotonation Limited | System and methods for calibration of an array camera |
US9986224B2 (en) | 2013-03-10 | 2018-05-29 | Fotonation Cayman Limited | System and methods for calibration of an array camera |
US11272161B2 (en) | 2013-03-10 | 2022-03-08 | Fotonation Limited | System and methods for calibration of an array camera |
US11570423B2 (en) | 2013-03-10 | 2023-01-31 | Adeia Imaging Llc | System and methods for calibration of an array camera |
US10958892B2 (en) | 2013-03-10 | 2021-03-23 | Fotonation Limited | System and methods for calibration of an array camera |
US10127682B2 (en) | 2013-03-13 | 2018-11-13 | Fotonation Limited | System and methods for calibration of an array camera |
US9888194B2 (en) | 2013-03-13 | 2018-02-06 | Fotonation Cayman Limited | Array camera architecture implementing quantum film image sensors |
US10091405B2 (en) | 2013-03-14 | 2018-10-02 | Fotonation Cayman Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US10547772B2 (en) | 2013-03-14 | 2020-01-28 | Fotonation Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US10542208B2 (en) | 2013-03-15 | 2020-01-21 | Fotonation Limited | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US10638099B2 (en) | 2013-03-15 | 2020-04-28 | Fotonation Limited | Extended color processing on pelican array cameras |
US10455218B2 (en) | 2013-03-15 | 2019-10-22 | Fotonation Limited | Systems and methods for estimating depth using stereo array cameras |
US10182216B2 (en) | 2013-03-15 | 2019-01-15 | Fotonation Limited | Extended color processing on pelican array cameras |
US10412368B2 (en) * | 2013-03-15 | 2019-09-10 | Uber Technologies, Inc. | Methods, systems, and apparatus for multi-sensory stereo vision for robotics |
US10674138B2 (en) | 2013-03-15 | 2020-06-02 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
US9955070B2 (en) | 2013-03-15 | 2018-04-24 | Fotonation Cayman Limited | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US20160057406A1 (en) * | 2013-04-09 | 2016-02-25 | Vit | Three-dimensional image acquisition system |
US10268885B2 (en) | 2013-04-15 | 2019-04-23 | Microsoft Technology Licensing, Llc | Extracting true color from a color and infrared sensor |
US20140307058A1 (en) * | 2013-04-15 | 2014-10-16 | Microsoft Corporation | Robust stereo depth system |
US10816331B2 (en) | 2013-04-15 | 2020-10-27 | Microsoft Technology Licensing, Llc | Super-resolving depth map by moving pattern projector |
WO2014172228A1 (en) * | 2013-04-15 | 2014-10-23 | Microsoft Corporation | Robust stereo depth system |
US10929658B2 (en) * | 2013-04-15 | 2021-02-23 | Microsoft Technology Licensing, Llc | Active stereo with adaptive support weights from a separate image |
US9928420B2 (en) * | 2013-04-15 | 2018-03-27 | Microsoft Technology Licensing, Llc | Depth imaging system based on stereo vision and infrared radiation |
US10928189B2 (en) | 2013-04-15 | 2021-02-23 | Microsoft Technology Licensing, Llc | Intensity-modulated light pattern for active stereo |
EP3541071A1 (en) * | 2013-08-02 | 2019-09-18 | Xactware Solutions Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US10540577B2 (en) | 2013-08-02 | 2020-01-21 | Xactware Solutions, Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US9679227B2 (en) * | 2013-08-02 | 2017-06-13 | Xactware Solutions, Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US10896353B2 (en) | 2013-08-02 | 2021-01-19 | Xactware Solutions, Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
WO2015017855A1 (en) | 2013-08-02 | 2015-02-05 | Xactware Solutions, Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US20220027622A1 (en) * | 2013-08-02 | 2022-01-27 | Xactware Solutions, Inc. | System and Method for Detecting Features in Aerial Images Using Disparity Mapping and Segmentation Techniques |
EP3028464A4 (en) * | 2013-08-02 | 2017-04-12 | Xactware Solutions Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US11144795B2 (en) | 2013-08-02 | 2021-10-12 | Xactware Solutions, Inc. | System and method for detecting features in aerial images using disparity mapping and segmentation techniques |
US9898856B2 (en) | 2013-09-27 | 2018-02-20 | Fotonation Cayman Limited | Systems and methods for depth-assisted perspective distortion correction |
US10540806B2 (en) | 2013-09-27 | 2020-01-21 | Fotonation Limited | Systems and methods for depth-assisted perspective distortion correction |
CN103595935A (en) * | 2013-10-15 | 2014-02-19 | 深圳市掌网立体时代视讯技术有限公司 | Dynamic compensation method and system for three-dimensional video image dead pixels |
US9924092B2 (en) | 2013-11-07 | 2018-03-20 | Fotonation Cayman Limited | Array cameras incorporating independently aligned lens stacks |
US11486698B2 (en) | 2013-11-18 | 2022-11-01 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US10119808B2 (en) | 2013-11-18 | 2018-11-06 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US10767981B2 (en) | 2013-11-18 | 2020-09-08 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US10708492B2 (en) | 2013-11-26 | 2020-07-07 | Fotonation Limited | Array camera configurations incorporating constituent array cameras and constituent cameras |
US9697604B2 (en) | 2014-01-28 | 2017-07-04 | Altek Semiconductor Corp. | Image capturing device and method for detecting image deformation thereof |
US10089740B2 (en) | 2014-03-07 | 2018-10-02 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
US10574905B2 (en) | 2014-03-07 | 2020-02-25 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
US10055816B2 (en) | 2014-03-20 | 2018-08-21 | Gopro, Inc. | Target-less auto-alignment of image sensors in a multi-camera system |
US10798365B2 (en) | 2014-03-20 | 2020-10-06 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US10389993B2 (en) | 2014-03-20 | 2019-08-20 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US9521318B2 (en) | 2014-03-20 | 2016-12-13 | Gopro, Inc. | Target-less auto-alignment of image sensors in a multi-camera system |
US11375173B2 (en) | 2014-03-20 | 2022-06-28 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US9197885B2 (en) | 2014-03-20 | 2015-11-24 | Gopro, Inc. | Target-less auto-alignment of image sensors in a multi-camera system |
WO2015142412A3 (en) * | 2014-03-20 | 2015-11-12 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US9792667B2 (en) | 2014-03-20 | 2017-10-17 | Gopro, Inc. | Target-less auto-alignment of image sensors in a multi-camera system |
US9325917B2 (en) | 2014-03-20 | 2016-04-26 | Gopro, Inc. | Auto-alignment of image sensors in a multi-camera system |
US9196039B2 (en) * | 2014-04-01 | 2015-11-24 | Gopro, Inc. | Image sensor read window adjustment for multi-camera array tolerance |
US9794498B2 (en) | 2014-04-01 | 2017-10-17 | Gopro, Inc. | Multi-camera array with housing |
US9473713B2 (en) | 2014-04-01 | 2016-10-18 | Gopro, Inc. | Image taping in a multi-camera array |
US10805559B2 (en) | 2014-04-01 | 2020-10-13 | Gopro, Inc. | Multi-camera array with shared spherical lens |
US20150279038A1 (en) * | 2014-04-01 | 2015-10-01 | Gopro, Inc. | Image Sensor Read Window Adjustment for Multi-Camera Array Tolerance |
US9832397B2 (en) | 2014-04-01 | 2017-11-28 | Gopro, Inc. | Image taping in a multi-camera array |
US20160042493A1 (en) * | 2014-04-01 | 2016-02-11 | Gopro, Inc. | Image Sensor Read Window Adjustment for Multi-Camera Array Tolerance |
US9681068B2 (en) * | 2014-04-01 | 2017-06-13 | Gopro, Inc. | Image sensor read window adjustment for multi-camera array tolerance |
US10200636B2 (en) | 2014-04-01 | 2019-02-05 | Gopro, Inc. | Multi-camera array with shared spherical lens |
US9734264B2 (en) * | 2014-05-05 | 2017-08-15 | Microsoft Technology Licensing, Llc | Fabricating three-dimensional objects with embossing |
US20150317412A1 (en) * | 2014-05-05 | 2015-11-05 | Microsoft Corporation | Fabricating three-dimensional objects with embossing |
US10600245B1 (en) * | 2014-05-28 | 2020-03-24 | Lucasfilm Entertainment Company Ltd. | Navigating a virtual environment of a media content item |
US11508125B1 (en) | 2014-05-28 | 2022-11-22 | Lucasfilm Entertainment Company Ltd. | Navigating a virtual environment of a media content item |
US10602200B2 (en) | 2014-05-28 | 2020-03-24 | Lucasfilm Entertainment Company Ltd. | Switching modes of a media content item |
CN104008189A (en) * | 2014-06-11 | 2014-08-27 | 义乌市勘测设计研究院 | Measurable street view establishing method with automatic information pushing function |
US20170214899A1 (en) * | 2014-07-23 | 2017-07-27 | Metaio Gmbh | Method and system for presenting at least part of an image of a real object in a view of a real environment, and method and system for selecting a subset of a plurality of images |
US10659750B2 (en) * | 2014-07-23 | 2020-05-19 | Apple Inc. | Method and system for presenting at least part of an image of a real object in a view of a real environment, and method and system for selecting a subset of a plurality of images |
WO2016022008A1 (en) * | 2014-08-08 | 2016-02-11 | Samsung Electronics Co., Ltd. | Method and apparatus for environmental profile generation |
US10469826B2 (en) | 2014-08-08 | 2019-11-05 | Samsung Electronics Co., Ltd. | Method and apparatus for environmental profile generation |
US9671625B2 (en) | 2014-08-20 | 2017-06-06 | David Kind, Inc. | System and method of providing custom-fitted and styled eyewear based on user-provided images and preferences |
US10222636B2 (en) | 2014-08-20 | 2019-03-05 | David Kind, Inc. | System and method of providing custom-fitted and styled eyewear based on user-provided images and preferences |
US9086582B1 (en) | 2014-08-20 | 2015-07-21 | David Kind, Inc. | System and method of providing custom-fitted and styled eyewear based on user-provided images and preferences |
US10250813B2 (en) * | 2014-09-03 | 2019-04-02 | Fuji Xerox Co., Ltd. | Methods and systems for sharing views |
US20160065858A1 (en) * | 2014-09-03 | 2016-03-03 | Fuji Xerox Co., Ltd. | Methods and systems for sharing views |
US10250871B2 (en) | 2014-09-29 | 2019-04-02 | Fotonation Limited | Systems and methods for dynamic calibration of array cameras |
US11546576B2 (en) | 2014-09-29 | 2023-01-03 | Adeia Imaging Llc | Systems and methods for dynamic calibration of array cameras |
EP3201688A4 (en) * | 2014-10-02 | 2018-02-28 | CJ CGV Co., Ltd. | Rig for multi-camera photographing |
JP2017535805A (en) * | 2014-10-02 | 2017-11-30 | シゼイ シジブイ カンパニー リミテッド | Multi-camera rig |
US11729510B2 (en) * | 2014-10-24 | 2023-08-15 | Bounce Imaging, Inc. | Imaging systems and methods |
US20160119541A1 (en) * | 2014-10-24 | 2016-04-28 | Bounce Imaging, Inc. | Imaging systems and methods |
US20200366841A1 (en) * | 2014-10-24 | 2020-11-19 | Bounce Imaging, Inc. | Imaging systems and methods |
US10771692B2 (en) * | 2014-10-24 | 2020-09-08 | Bounce Imaging, Inc. | Imaging systems and methods |
US10091418B2 (en) * | 2014-10-24 | 2018-10-02 | Bounce Imaging, Inc. | Imaging systems and methods |
US20160173757A1 (en) * | 2014-12-15 | 2016-06-16 | Samsung Electro-Mechanics Co., Ltd. | Camera module |
US9406343B1 (en) * | 2015-01-26 | 2016-08-02 | Bryan T. Evans | Method of tracking for animation insertions to video recordings |
US9563958B1 (en) | 2015-07-02 | 2017-02-07 | Yesvideo, Inc. | 2D/3D image scanning and compositing |
US9563957B1 (en) * | 2015-07-02 | 2017-02-07 | Yesvideo, Inc. | 2D/3D image scanning and compositing |
US10210644B1 (en) | 2015-07-02 | 2019-02-19 | Yesvideo, Inc. | Image capture using target area illumination |
US9786080B1 (en) | 2015-07-02 | 2017-10-10 | Yesvideo, Inc. | 2D/3D image scanning and compositing |
AT517656A1 (en) * | 2015-08-20 | 2017-03-15 | Ait Austrian Inst Of Tech G M B H | Photometric Stereomatching |
WO2017112905A1 (en) * | 2015-12-22 | 2017-06-29 | Google Inc. | Capture and render of virtual reality content employing a light field camera array |
US10244227B2 (en) | 2015-12-22 | 2019-03-26 | Google Llc | Capture and render of virtual reality content employing a light field camera array |
US20170359561A1 (en) * | 2016-06-08 | 2017-12-14 | Uber Technologies, Inc. | Disparity mapping for an autonomous vehicle |
US11238603B2 (en) * | 2016-06-08 | 2022-02-01 | Amazon Technologies, Inc. | Selectively paired imaging elements for stereo images |
US20170374360A1 (en) * | 2016-06-28 | 2017-12-28 | Magic Leap, Inc. | Camera calibration system, target, and process |
CN109416744A (en) * | 2016-06-28 | 2019-03-01 | 奇跃公司 | Improved camera calibration system, target and process |
US10547833B2 (en) * | 2016-06-28 | 2020-01-28 | Magic Leap, Inc. | Camera calibration system, target, and process |
US20190219902A1 (en) * | 2016-09-28 | 2019-07-18 | Jacek LIPIK | Scanner, specifically for scanning antique books, and a method of scanning |
US10788735B2 (en) * | 2016-09-28 | 2020-09-29 | Jacek LIPIK | Scanner, specifically for scanning antique books, and a method of scanning |
CN109952760A (en) * | 2016-12-30 | 2019-06-28 | 谷歌有限责任公司 | The splicing of multiple view scene flows |
WO2018125369A1 (en) * | 2016-12-30 | 2018-07-05 | Google Llc | Multi-view scene flow stitching |
EP3358846A1 (en) * | 2017-02-06 | 2018-08-08 | Robo-Team Home Ltd. | Method and device for stereoscopic vision |
CN108399633A (en) * | 2017-02-06 | 2018-08-14 | 罗伯团队家居有限公司 | Method and apparatus for stereoscopic vision |
US10567732B2 (en) | 2017-02-06 | 2020-02-18 | Robotemi Ltd | Method and device for stereoscopic vision |
CN110832547A (en) * | 2017-02-08 | 2020-02-21 | 约夫·舍菲 | System and method for generating stereo paired images of virtual objects |
CN108810571A (en) * | 2017-05-04 | 2018-11-13 | 汤姆逊许可公司 | The method and apparatus for coding and decoding two-dimensional points cloud |
US11671717B2 (en) * | 2017-10-20 | 2023-06-06 | Lucasfilm Entertainment Company Ltd. | Camera systems for motion capture |
US10967862B2 (en) | 2017-11-07 | 2021-04-06 | Uatc, Llc | Road anomaly detection for autonomous vehicle |
US11731627B2 (en) | 2017-11-07 | 2023-08-22 | Uatc, Llc | Road anomaly detection for autonomous vehicle |
CN111902072A (en) * | 2017-12-01 | 2020-11-06 | 索罗新公司 | System and method for ultrasound tissue screening |
US11012676B2 (en) | 2017-12-13 | 2021-05-18 | Google Llc | Methods, systems, and media for generating and rendering immersive video content |
US11589027B2 (en) | 2017-12-13 | 2023-02-21 | Google Llc | Methods, systems, and media for generating and rendering immersive video content |
US10542247B2 (en) | 2017-12-20 | 2020-01-21 | Wistron Corporation | 3D image capture method and system |
CN110728740A (en) * | 2018-07-13 | 2020-01-24 | 辉达公司 | Virtual photogrammetry |
US11625894B2 (en) | 2018-07-13 | 2023-04-11 | Nvidia Corporation | Virtual photogrammetry |
CN110060291A (en) * | 2019-04-04 | 2019-07-26 | 长春理工大学 | It is a kind of consider people because stereopsis in distance calculation method |
US11699273B2 (en) | 2019-09-17 | 2023-07-11 | Intrinsic Innovation Llc | Systems and methods for surface modeling using polarization cues |
US11270110B2 (en) | 2019-09-17 | 2022-03-08 | Boston Polarimetrics, Inc. | Systems and methods for surface modeling using polarization cues |
US11525906B2 (en) | 2019-10-07 | 2022-12-13 | Intrinsic Innovation Llc | Systems and methods for augmentation of sensor systems and imaging systems with polarization |
US11302012B2 (en) | 2019-11-30 | 2022-04-12 | Boston Polarimetrics, Inc. | Systems and methods for transparent object segmentation using polarization cues |
US11842495B2 (en) | 2019-11-30 | 2023-12-12 | Intrinsic Innovation Llc | Systems and methods for transparent object segmentation using polarization cues |
US11915368B2 (en) | 2019-12-04 | 2024-02-27 | Insurance Services Office, Inc. | Systems and methods for modeling structures using point clouds derived from stereoscopic image pairs |
US11094113B2 (en) | 2019-12-04 | 2021-08-17 | Geomni, Inc. | Systems and methods for modeling structures using point clouds derived from stereoscopic image pairs |
CN111340871A (en) * | 2019-12-31 | 2020-06-26 | 东软集团股份有限公司 | Depth map prediction method, depth map prediction device, storage medium and electronic equipment |
US11580667B2 (en) | 2020-01-29 | 2023-02-14 | Intrinsic Innovation Llc | Systems and methods for characterizing object pose detection and measurement systems |
US11797863B2 (en) | 2020-01-30 | 2023-10-24 | Intrinsic Innovation Llc | Systems and methods for synthesizing data for training statistical models on different imaging modalities including polarized images |
CN111735451A (en) * | 2020-04-16 | 2020-10-02 | 中国北方车辆研究所 | Point cloud matching high-precision positioning method based on multi-source prior information |
US11953700B2 (en) | 2020-05-27 | 2024-04-09 | Intrinsic Innovation Llc | Multi-aperture polarization optical systems using beam splitters |
US11836940B2 (en) * | 2020-06-15 | 2023-12-05 | Zebra Technologies Corporation | Three-dimensional sensor acuity recovery assistance |
US20210390724A1 (en) * | 2020-06-15 | 2021-12-16 | Zebra Technologies Corporation | Three-Dimensional Sensor Acuity Recovery Assistance |
US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
US11954886B2 (en) | 2021-04-15 | 2024-04-09 | Intrinsic Innovation Llc | Systems and methods for six-degree of freedom pose estimation of deformable objects |
US11683594B2 (en) | 2021-04-15 | 2023-06-20 | Intrinsic Innovation Llc | Systems and methods for camera exposure control |
WO2022237544A1 (en) * | 2021-05-11 | 2022-11-17 | 梅卡曼德(北京)机器人科技有限公司 | Trajectory generation method and apparatus, and electronic device and storage medium |
US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
CN113409450A (en) * | 2021-07-09 | 2021-09-17 | 浙江大学 | Three-dimensional reconstruction method for chickens containing RGBDT information |
Also Published As
Publication number | Publication date |
---|---|
US20140375775A1 (en) | 2014-12-25 |
US20140098199A1 (en) | 2014-04-10 |
US8867827B2 (en) | 2014-10-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8867827B2 (en) | Systems and methods for 2D image and spatial data capture for 3D stereo imaging | |
US10540806B2 (en) | Systems and methods for depth-assisted perspective distortion correction | |
US11869205B1 (en) | Techniques for determining a three-dimensional representation of a surface of an object from a set of images | |
CN108886611B (en) | Splicing method and device of panoramic stereo video system | |
US10609282B2 (en) | Wide-area image acquiring method and apparatus | |
JP6273163B2 (en) | Stereoscopic panorama | |
KR20180111798A (en) | Adaptive stitching of frames in the panorama frame creation process | |
US20100085423A1 (en) | Stereoscopic imaging | |
US10275898B1 (en) | Wedge-based light-field video capture | |
CN105530431A (en) | Reflective panoramic imaging system and method | |
WO2018235163A1 (en) | Calibration device, calibration chart, chart pattern generation device, and calibration method | |
KR101969082B1 (en) | Optimal Spherical Image Acquisition Method Using Multiple Cameras | |
US20120154518A1 (en) | System for capturing panoramic stereoscopic video | |
US20120154548A1 (en) | Left/right image generation for 360-degree stereoscopic video | |
CN112330736A (en) | Scene picture shooting method and device, electronic equipment and storage medium | |
KR20200129657A (en) | Method for gaining 3D model video sequence | |
US20120154519A1 (en) | Chassis assembly for 360-degree stereoscopic video capture | |
US20220137555A1 (en) | System and method for lightfield capture | |
JP7195801B2 (en) | Image processing device and its control method, distance detection device, imaging device, program | |
WO2005034527A1 (en) | Stereoscopic imaging | |
JP2020191624A (en) | Electronic apparatus and control method for the same | |
EP3977405A1 (en) | 3d model capture system | |
JP5555046B2 (en) | Image input device, image input method, and image input program | |
JP2022024688A (en) | Depth map generation device and program thereof, and depth map generation system | |
KR20150047604A (en) | Method for description of object points of the object space and connection for its implementation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GBO 3D TECHNOLOGY PTE LTD, SINGAPORE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YEATMAN, JR., HOYT H;ROBINSON, GARY;SIGNING DATES FROM 20110305 TO 20110307;REEL/FRAME:025962/0439 |
|
AS | Assignment |
Owner name: SHAPEQUEST, INC., ARIZONA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GBO 3D TECHNOLOGY PTE LTD;REEL/FRAME:031558/0066 Effective date: 20131105 |
|
STCB | Information on status: application discontinuation |
Free format text: EXPRESSLY ABANDONED -- DURING EXAMINATION |