US20050031169A1 - Birds eye view virtual imaging for real time composited wide field of view - Google Patents
Birds eye view virtual imaging for real time composited wide field of view Download PDFInfo
- Publication number
- US20050031169A1 US20050031169A1 US10/914,375 US91437504A US2005031169A1 US 20050031169 A1 US20050031169 A1 US 20050031169A1 US 91437504 A US91437504 A US 91437504A US 2005031169 A1 US2005031169 A1 US 2005031169A1
- Authority
- US
- United States
- Prior art keywords
- image
- images
- vehicle
- view
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/36—Input/output arrangements for on-board computers
- G01C21/3626—Details of the output of route guidance instructions
- G01C21/3647—Guidance involving output of stored or live camera images or video streams
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
Definitions
- the present invention relates to methods of acquiring multiple images for display in the form of a composite image from alternative vantage points, and more particularly to the use of such methods for steering transportation vehicles to real-time or providing situational awareness.
- the present invention recognizes certain limitations which inherently exist in an attempte to navigate a vehicle.
- vehicles provide a driver with a limited view of the driver's surroundings.
- large trucks and military vehicles such as a tanks position a driver high above a roadway with a somewhat limited viewing angle.
- potholes and other impediments are closely approached by the vehicle, they are no longer in a driver's field of view.
- an attempt was made to suggest a means of providing a driver both with obstacle positioning and coordinance together with a broader view of the vehicle's upcoming terrain.
- a first object of the instant invention is to display a virtual image to an individual steering or driving a plane, vessel or transportation vehicle in real time, or other visualization requirement when the image comprising a live image is suitably transformed to reflect a vantage point reflecting the position of the vehicle in reference to obstacles and hazards that are no longer in view.
- Yet another object of the present invention is to display virtual imaging that combines visual and non-visual imaging sensors in real-time.
- It is yet another object of the present invention is to combine live images acquired by multiple vehicles to form composite images reflecting a wider virtual field of view, the field of view optionally combining using previously acquired or generated images superimposed thereon.
- a further object of the invention is to superimpose reference information on the aforementioned composite images illustrating, for example, the relative position of the vehicle, hazards, targets and the desired path or roadway between such objects.
- One aspect of the invention is characterized in that images acquired at times t1 and t2 are optionally superimposed or composited by correlating the relative magnification such that pixels from the distant image are placed with the corresponding pixels of the live image.
- this superposition is preferably continuously updated to account for both forward movement and rotation (X, Y, Z) of the image frames.
- FIG. 1 the elevation of a vehicle, traveling on the road is intended to schematically illustrate the driver's actual viewpoint and the preferred viewpoint according to the teachings of the instant invention.
- FIG. 2 is an elevation showing the principal of acquiring and utilizing time sequence images corresponding to a vehicle's first position at time t 1 and second position at t 2 while traveling on the road depicted in FIG. 1 .
- FIG. 3 is schematic illustration of a vehicle is the actual position as display to the driver of a vehicle from the preferred viewpoint on an information display.
- FIG. 5 is a plain view illustrating the use of the instant invention for assisting a truck driver to backup a truck rig into a loading ramp.
- FIG. 6 is a plan view illustrating the use of the instant invention for the command and control of a variety of combat vehicle executing a mission was spreading out over the terrain
- FIG. 1 is an elevation view to illustrate the deficiencies of the vantage point of a driver in vehicle 100 .
- Objects having reference numerals 10 , 20 and 30 represent normally visible obstacles in or adjacent to the road 110 . More specifically, the driver has just passed by pothole 10 , which is no longer visible from the front window, whereas the farthest obstacles 30 are still within the field of view, being subtended by angle 121 to represent the driver's vantage point.
- object 20 is about to leave the driver's field of view as the vehicle progresses forward, it is very difficult to pass such road obstacles at a close distance, or drive between them as the spacing approaches the width of the vehicle.
- the disappearance at close range from the driver's field of vision is undesirable.
- the navigation vehicle depends on non-visual imaging system, for instance an infrared imaging system for use during night driving or fog
- the viewing angle of such imaging systems can be a further limitation. Under these and other conditions it would be preferable if the driver could simultaneously have a sufficiently distant view in front of the vehicle, while at the same time perceive the vehicles position with respect to close objects and hazards they are attempting or forced to avoid at a close distance.
- a display of the virtual viewing angle shows the driver the vehicle location with respect to road hazards 10 , 20 and 30 , which might be located either just in front or to either side of the vehicle.
- FIG. 2 illustrates the operative principles for a first embodiment of the instant invention, which includes acquiring and displaying the desirable image of virtual camera 130 of FIG. 1 .
- Vehicle 200 in FIG. 2 includes a video or digital imaging camera 240 that continually acquires at least a forward image as vehicle 200 progresses from the right side of the Figure at time t1 to the left side of the Figure at time t2.
- Camera 240 has an actual viewing area within the arc subtended by angle 223 , which may be the same, narrower or wider than the driver's actual field of view.
- Two or more images acquired by camera 240 between t1 and t 2 are used to generate a composite, for example, an actual image acquired at time t2 as if acquired from virtual camera position 130 , but overlaid in correspondence with image data acquired at time t1.
- the driver would be able to view and steer around or close to the edge of the road but avoid hazards 10 , 20 and 30 .
- FIG. 3 illustrates a display 300 of the driver's view of composite image corresponding to time t2, as would be seen from virtual camera position 130 .
- the display 300 has a first frame region 302 and a surrounding region 301 .
- the first or inner frame region 301 preferably is used to display the current, or live image accorded time t2, whereas the surrounding region 301 was acquired earlier, that is at time t1, or between time t2 and time t1, and thus includes pothole type road 10 , which is adjacent to vehicle, and out of the driver's current actual view.
- the image of the vehicle 310 is synthesized, as it is never in actual view.
- Display 300 may also include various indicia, such as a guideline 320 to follow to drive between other hazard objects 20 and 30 .
- each of the images acquired at time t1 and time t2 are generally transformed prior to display of the composite image representing the virtual viewpoint at time t2.
- the image at time t1 in the above example can be generated from another image source not on the vehicle, including an image database, and may in fact have been acquired at the reference viewpoint.
- Generating the Birds Eye View (BEV) image of FIG. 3 utilizes one or more images acquired at a different viewpoint, which transformed to reflect a different vantage point above vehicle.
- BEV Birds Eye View
- Aerial video is rapidly emerging as a low cost, widely used source of imagery for mapping, surveillance and monitoring applications.
- U.S. Pat. No. 5,590,037 which is incorporated herein by reference, discloses a method for converting forward-looking video or motion picture imagery into a down looking database suitable for use in an image generation system to generate real-time perspective images for simulation purposes.
- U.S. Pat. No. 5,649,032 which is incorporated herein by reference, discloses methods for automatically generating a mosaic from a plurality of input images.
- the inventor's of the '032 patent teach a mosaic construction system that sequentially executes an image alignment process and a mosaic composition process such that, from a sequence of images, the system automatically produces a mosaic for utilization by various applications.
- the invention is described as being capable of constructing both dynamic and static mosaics.
- a dynamic mosaic includes imagery that is time variant, e.g., the mosaic is updated with new content over time, while the content of a static mosaic is time invariant.
- U.S. Pat. No. 6,512,857 which is incorporated herein by reference, discloses a system and method for accurately mapping between camera coordinates and geo-coordinates, called geo-spatial registration.
- the method utilizes the imagery and terrain information contained in the geo-spatial database to precisely align the reference imagery with input imagery, such as dynamically generated video images or video mosaics, and thus achieve a high accuracy identification of locations within the scene.
- the geo-spatial reference database generally contains a substantial amount of reference imagery as well as scene annotation information and object identification information.
- a sensor such as a video camera
- the system recalls a reference image pertaining to the imaged scene.
- This reference image is aligned very accurately with the sensor's images using a parametric transformation. Thereafter, other information (annotation, sound, and the like) that is associated with the reference image can easily be overlaid upon or otherwise associated with the sensor imagery.
- Applications of geo-spatial registration include text/graphica/audio annotations of objects of interest in the current video using the stored annotations in the reference database to augment and add meaning to the current video
- Observera Inc. of 4451 Brookfield Corporate Drive, Suite 107, Chantilly, Va. 20151-1693 provides software that has a range of features allowing modification to serve a variety of applications.
- Samoff Corporation and Pyramid Vision Technologies both of 201 Washington Road, CN 5300 Princeton, N.J. 08543-5300, supply commercial software and hardware for performing Birds-Eye-View transformations to create fly by images.
- FIG. 4 illustrates one embodiment of operative principles for acquiring, transforming and aligning the image data used to generate the real-time display of FIG. 3 . Accordingly, the following description should not be construed as limiting the scope of the patent.
- FIG. 4 illustrates an alternative embodiment for generating the image display
- image 401 is acquired the time t 1 whereas image 400 and to is acquired a time t2, accordingly the pair of image is represented by bracket 410 correspond to live actual images recorded for the moving vehicle.
- the first image frame 402 is acquired in either video or digital format.
- a second or live image 402 is similarly acquired.
- Bracket 410 contains a digital representation of the actual images acquired at time t1 and t2 for further transformation and merger to form a composite image 420 .
- images 403 and 404 , in bracket 420 represent bird's eye view transformations of the corresponding adjacent images within bracket 410 .
- the rectangular image frames in bracket 410 distorted to trapezoidal shapes in generating the higher elevation or Birds-eye-view.
- One embodiment for generating such bird's eye view images optionally includes performing scaled transformation of the rectangular image frame to a trapezoid to simulate the loss of prospective as the birds eye view camera increases in azimuth angle from the actual viewpoint of the camera mounted on the vehicle.
- the trapezoid results from transforming each row of the x-axis gradually with increased compression starting from the upper edge of the picture frame of the actual view 410 , with increasing compression towards the bottom of the frame.
- a trapezoidal transformation decreases the divergence of the lines representing the highway traffic lanes in the images of bracket 410 .
- images 401 is preferably modified by digital processing to image 403 to correspond to the expected appearance from camera position 130 in FIG. 2
- alternative viewpoints are possible, including a position forward of the vehicle at the time t1 at which image 401 is recorded, provided images 403 and 404 are generated with substantially the same virtual camera position.
- the near image acquired at time t1 is appropriately scaled and overlapped with respect to the earlier acquired image at time t2.
- the scaling factor to convert image 403 to image 405 must be determined, as well as any x and y displacement for overlay of image 405 on image 404 .
- virtual features such as the image of the vehicle, frame separating the image regions, optional projection of any intended driving path, and the like, are preferably overlaid on the penultimate composite images to form the final composite image.
- the composite image is most preferably refreshed in real time to reflect the forward progress of the vehicle.
- image 405 is represented as de-magnified from image 403 to represent its scaling prior to merger over virtual image 404 to create display image 420 .
- equivalent operations can be performed on image 401 , such that detail in the real time image is more fully preserved.
- the displayed image 420 is optionally generated by merging de-magnified image 405 with image 404 , taking into account lateral translation and rotation of the actual cameras viewpoint between the acquisition of frames 401 at time t1 and 402 at time t2.
- the overlay can be determined by mapping the displacement of pixels from image frames 405 to 404 , such that a selected sub group of pixels in image 404 is replaced with image frame 405 pixels prior to display.
- a relative movement or rotation is optionally determined by first searching each image field to identify high contrast features, and then comparing the relative orientation of these features to generate the appropriate correction factors. That is, when the correction factors are applied the high contrast features must coincide to compose an accurate virtual image for display.
- Image 403 is scaled to generate image 405 by a de-magnification factor based on Z-axis displacement of the vehicle between image frames 401 and 402 .
- the factor used to generate image 405 from image 403 can be determined by several alternative methods. Thus, depending on the method of forming and generating the composite image 420 , it is desirable to know the absolute movement of the actual camera position between times t1 and t2, as this determines the scaling factor for converting image 403 to 405 by geometric calculations, or magnifying images 404 with respect to 403 .
- the Z-axis distance can determined by several methods, including but not limited to global satellite positioning, or calculating the change in position by integrating the speedometer output overtime to synchronize time period between t1 and t2.
- JPEG and MPEG data format transmits full images, or I frames, infrequently to conserve bandwidth, using a sequence of intervening frames (B and P type in MPEG) to communicate the changes to portions of the image between I frames.
- B and P type in MPEG a sequence of intervening frames
- the image is broken down into macro blocks, which is collections of pixels, and analyzed to identify macro blocks that change location between successive image frames, which are then used to reconstruct the full image for final display.
- B and P frames identify and track macro blocks that change location between I frames.
- the translation and rotation of image 404 with respect image 405 necessary for merging these images may be determined from the movement of macro block represented in the JPEG and MPEG formats by extracting an average macro block translation to represent the relative movement between consecutive I frames.
- the vector sum of the individual translation factors can be applied to align images 404 and 405 .
- the magnification factor is alternatively computed from the y-axis components of the macro block translation between I frames.
- the position of car 310 in FIG. 3 is synthetically generated based on the data set representing the actual car's dimensions, and the selection of the birds eye view position, that is, Z, Y position in FIG. 1 .
- Z, Y position in FIG. 1 .
- two or more parameters which may include, the azimuth angle, vertical height above the vehicle, horizontal distance behind the vehicle (at time t2), viewing angle and like parameters define the Z and Y positions necessary to determine the appropriate transformation factors used to generate the pair transformed images in bracket 415 from the corresponding actual images in bracket 410 in FIG. 4 .
- MPEG conversion can be limited to images 401 and 402 , rather than the entire video frame sequence thus simplifying the computational complexity.
- the entire bit stream representing each individual frame recorded by the video camera between images 401 and 402 is utilized.
- the aforementioned method of macro block tracking can be extended to determine the factor used to compute the magnification of image 403 for generating image 405 while simultaneously correcting for what has been initially described as discrete steps of image rotation and translations determination, i.e. the steps used to place image 405 in image 404 .
- the aforementioned method of macro block tracking can be extended to determine the factor used to compute the magnification of image 403 for generating image 405 while simultaneously correcting for what has been initially described as discrete steps of image rotation and translations determination, i.e. the steps used to place image 405 in image 404 .
- it may be possible in some instances to identify one or more macro blocks that correspond to distinct objects or edges of the vision field it is unlikely that the same macro blocks can be uniquely identified for each frame of the video source from t1 to t2. However, this is not necessarily, provided a refreshed or updated subset of macro blocks is used at each I-frame.
- the updated subset would correspond to the same x-y coordinate range of the macro blocks in the previous I frame, as updated to reflect the most recent preceding image.
- using the MPEG bit stream to trace the displacement of macro blocks at corners of image frame 402 includes correction for magnification, translation and rotation. Reverse tracking the relative positions of macro blocks that correspond to the corner regions of frame 402 define a relative position for edges of the frame acquired at time t1 frame with respect to the frame acquired at time t2.
- a linear scaling between the compression and distortion ratios necessary to re-map the frame corner also accomplishes translation and rotation. It is anticipated that either of frames 402 or 401 can be modified, either before or after the trapezoidal distortion, that represents the “bird-eye-view”.
- magnification factor can also be determined by computation from the integrated speedometer readings, alternatively larger gaps can also be accommodated by calculation based on the GPS coordinates recorded at the time of acquisition for images 401 and 402 .
- the translation and rotation factors are preferably acquired by selecting the corresponding macro blocks that represent such features.
- the previous calculation likewise carried out by starting with the last live frame 402 , may be carried out b averaging (before accumulating the sum thereof) a limited number of macro block translation factors depending on the area selected.
- the identical macro blocks used as the starting point for the reverse computation may only maintain the same identity between I Frames in the MPEG bit stream, the accuracy is the likely to be improved by selecting a newer subset of macro blocks that correspond to the same x-y coordinate range of the macro blocks in the previous I frame.
- the steps in executing the aforementioned method of macro block tracking include; 1) identifying the first set of nearest neighbor macro blocks corresponding to corners of frame 402 or selected portions of the live image, 2) record in the average translation to the previous I frame, 3) record the average x-y coordinate position corresponding to average translation to previous I frame, 4) identifying a second set of nearest neighbor macro blocks corresponding to average x-y coordinate positions, 5) recording the average translation to the next previous I frame, 6) compute the sum of the first average translation and each subsequent average translation for each of the corresponding four corners of frame 402 (or selected regions therein), 7) repeating the previous steps of identifying the second set of nearest neighbor's until the subsequent I frame corresponds to the closest I frame from image frame 401 , 8) linearly distorting one of image frame 401 to 402 to aligned the corresponding corners according to their respective translation factors, 9) either before or after merging the distorted and undistorted image frame from the previous step generating a second composite image by distorting a first composite image to correspond to
- FIG. 5 is a plain view illustrating the use of the instant invention for assisting a truck driver to backup a truck rig 600 into a loading ramp 610 .
- the driver views a display that provides a composite of a live and recorded image from camera 640 (with viewing angle 641 ) according to the teachings of the invention with virtual camera position 630 (having viewing angle 631 ) such that the drivers “sees” the corners or other obstacles 611 and 612 in close proximity as they back up the truck to the loading dock.
- FIG. 6 is a plan view illustrating the use of the instant invention for the command and control (optionally from vehicle 705 ) of a variety of combat vehicles 700 - 704 executing a mission was spreading out over the terrain.
- Alternative virtual camera positions 730 a and 730 b allow a wide-angle view of the battlefield from any vehicle, with the images being acquired from manned vehicles 700 - 705 or drones 706 .
- the image preferably shows the actual view from each vehicle, identifies “friend” and “foe” with additional icons thus avoiding friendly fire accidents.
- images described and combined need not be solely from visual sources, but include IR, NIR and other non-visual sources, and may combine visual images with non-visual or enhanced images in either monoscopic or stereoscopic views in the final composite images.
Abstract
A live image and a previously acquired or generated image are superimposed or composited to represented a virtual vantage point for flying, driving or navigating a plane, vehicle or vessel.
Description
- The present application is based upon provisional application Ser. No. 60/493,579 filed on Aug. 9, 2003.
- The present invention relates to methods of acquiring multiple images for display in the form of a composite image from alternative vantage points, and more particularly to the use of such methods for steering transportation vehicles to real-time or providing situational awareness.
- The present invention recognizes certain limitations which inherently exist in an attempte to navigate a vehicle. Often times, vehicles provide a driver with a limited view of the driver's surroundings. For example, large trucks and military vehicles such as a tanks position a driver high above a roadway with a somewhat limited viewing angle. By the time potholes and other impediments are closely approached by the vehicle, they are no longer in a driver's field of view. As such, an attempt was made to suggest a means of providing a driver both with obstacle positioning and coordinance together with a broader view of the vehicle's upcoming terrain. To the inventor's understanding, there has been no successful means suggested to date for providing such useful information to a vehicle operator.
- A first object of the instant invention is to display a virtual image to an individual steering or driving a plane, vessel or transportation vehicle in real time, or other visualization requirement when the image comprising a live image is suitably transformed to reflect a vantage point reflecting the position of the vehicle in reference to obstacles and hazards that are no longer in view.
- Yet another object of the present invention is to display virtual imaging that combines visual and non-visual imaging sensors in real-time.
- It is yet another object of the present invention is to combine live images acquired by multiple vehicles to form composite images reflecting a wider virtual field of view, the field of view optionally combining using previously acquired or generated images superimposed thereon.
- A further object of the invention is to superimpose reference information on the aforementioned composite images illustrating, for example, the relative position of the vehicle, hazards, targets and the desired path or roadway between such objects.
- One aspect of the invention is characterized in that images acquired at times t1 and t2 are optionally superimposed or composited by correlating the relative magnification such that pixels from the distant image are placed with the corresponding pixels of the live image. However, to the extent that the virtual viewpoint is intended to enable navigation around objects that are no longer visible to the live image, this superposition is preferably continuously updated to account for both forward movement and rotation (X, Y, Z) of the image frames.
- The above and other objects, effects, features, and advantages of the present invention will become more apparent from the following description of the embodiments thereof taken in conjunction with the accompanying drawings.
- In
FIG. 1 the elevation of a vehicle, traveling on the road is intended to schematically illustrate the driver's actual viewpoint and the preferred viewpoint according to the teachings of the instant invention. -
FIG. 2 is an elevation showing the principal of acquiring and utilizing time sequence images corresponding to a vehicle's first position at time t 1 and second position at t 2 while traveling on the road depicted inFIG. 1 . -
FIG. 3 is schematic illustration of a vehicle is the actual position as display to the driver of a vehicle from the preferred viewpoint on an information display. -
FIG. 4 a schematic diagramFIG. 4 a systematic diagram illustrating the principles underlying one embodiment of a method of image processing to transform the images to equivalent virtual view point above and behind the vehicle at the same magnification, including discrete steps in transforming, aligning and superimposed a real-time image with a corresponding synthetic image for display. -
FIG. 5 is a plain view illustrating the use of the instant invention for assisting a truck driver to backup a truck rig into a loading ramp. -
FIG. 6 is a plan view illustrating the use of the instant invention for the command and control of a variety of combat vehicle executing a mission was spreading out over the terrain -
FIG. 1 is an elevation view to illustrate the deficiencies of the vantage point of a driver in vehicle 100. Objects havingreference numerals road 110. More specifically, the driver has just passed bypothole 10, which is no longer visible from the front window, whereas thefarthest obstacles 30 are still within the field of view, being subtended by angle 121 to represent the driver's vantage point. However, asobject 20 is about to leave the driver's field of view as the vehicle progresses forward, it is very difficult to pass such road obstacles at a close distance, or drive between them as the spacing approaches the width of the vehicle. Thus, to the extent a road, bridge or rough terrain is extremely narrow with obstacles or borders that represent significant safety hazards, or the driver is required to navigate in close proximity to such hazards, the disappearance at close range from the driver's field of vision is undesirable. Further, to the extent the navigation vehicle depends on non-visual imaging system, for instance an infrared imaging system for use during night driving or fog, the viewing angle of such imaging systems, represented by reference numerals 122, can be a further limitation. Under these and other conditions it would be preferable if the driver could simultaneously have a sufficiently distant view in front of the vehicle, while at the same time perceive the vehicles position with respect to close objects and hazards they are attempting or forced to avoid at a close distance. - More specifically, in particularly hazardous situations it would be desirable if the driver could control the vehicle with a virtual viewpoint situated slightly behind a vehicle, as indicated by the
camera icon 130, having a virtual viewing area within the arc subtended by angle 131. Thus, a display of the virtual viewing angle shows the driver the vehicle location with respect toroad hazards - Accordingly,
FIG. 2 illustrates the operative principles for a first embodiment of the instant invention, which includes acquiring and displaying the desirable image ofvirtual camera 130 ofFIG. 1 .Vehicle 200 inFIG. 2 includes a video ordigital imaging camera 240 that continually acquires at least a forward image asvehicle 200 progresses from the right side of the Figure at time t1 to the left side of the Figure at time t2.Camera 240 has an actual viewing area within the arc subtended byangle 223, which may be the same, narrower or wider than the driver's actual field of view. Two or more images acquired bycamera 240 between t1 and t 2 are used to generate a composite, for example, an actual image acquired at time t2 as if acquired fromvirtual camera position 130, but overlaid in correspondence with image data acquired at time t1. Thus, the driver would be able to view and steer around or close to the edge of the road but avoidhazards - Accordingly,
FIG. 3 illustrates adisplay 300 of the driver's view of composite image corresponding to time t2, as would be seen fromvirtual camera position 130. Thedisplay 300 has afirst frame region 302 and a surrounding region 301. The first or inner frame region 301 preferably is used to display the current, or live image accorded time t2, whereas the surrounding region 301 was acquired earlier, that is at time t1, or between time t2 and time t1, and thus includespothole type road 10, which is adjacent to vehicle, and out of the driver's current actual view. The image of the vehicle 310 is synthesized, as it is never in actual view. In yet another embodiment,Display 300 may also include various indicia, such as a guideline 320 to follow to drive betweenother hazard objects - It should be appreciated that since it is very difficult to position a camera for recording images that correspond with the
virtual camera 130 position each of the images acquired at time t1 and time t2 are generally transformed prior to display of the composite image representing the virtual viewpoint at time t2. However, the image at time t1 in the above example can be generated from another image source not on the vehicle, including an image database, and may in fact have been acquired at the reference viewpoint. - Generating the Birds Eye View (BEV) image of
FIG. 3 utilizes one or more images acquired at a different viewpoint, which transformed to reflect a different vantage point above vehicle. Those of ordinary skill in the art of computer graphics will recognize that there are numerous schemes for performing such transformations. Aerial video is rapidly emerging as a low cost, widely used source of imagery for mapping, surveillance and monitoring applications. The disclosure of U.S. Pat. No. 5,259,037, which is incorporated herein by reference, disclosures a method of acquiring individual images from an aerial video that can be aligned with one another and merged to form an image mosaic. In surveillance applications, such a video map provides the basis for estimating motion of objects within a scene. U.S. Pat. No. 5,590,037, which is incorporated herein by reference, discloses a method for converting forward-looking video or motion picture imagery into a down looking database suitable for use in an image generation system to generate real-time perspective images for simulation purposes. - Further, U.S. Pat. No. 5,649,032, which is incorporated herein by reference, discloses methods for automatically generating a mosaic from a plurality of input images. The inventor's of the '032 patent teach a mosaic construction system that sequentially executes an image alignment process and a mosaic composition process such that, from a sequence of images, the system automatically produces a mosaic for utilization by various applications. The invention is described as being capable of constructing both dynamic and static mosaics. A dynamic mosaic includes imagery that is time variant, e.g., the mosaic is updated with new content over time, while the content of a static mosaic is time invariant.
- U.S. Pat. No. 6,512,857, which is incorporated herein by reference, discloses a system and method for accurately mapping between camera coordinates and geo-coordinates, called geo-spatial registration. The method utilizes the imagery and terrain information contained in the geo-spatial database to precisely align the reference imagery with input imagery, such as dynamically generated video images or video mosaics, and thus achieve a high accuracy identification of locations within the scene. The geo-spatial reference database generally contains a substantial amount of reference imagery as well as scene annotation information and object identification information. When a sensor, such as a video camera, images a scene contained in the geo-spatial database, the system recalls a reference image pertaining to the imaged scene. This reference image is aligned very accurately with the sensor's images using a parametric transformation. Thereafter, other information (annotation, sound, and the like) that is associated with the reference image can easily be overlaid upon or otherwise associated with the sensor imagery. Applications of geo-spatial registration include text/graphica/audio annotations of objects of interest in the current video using the stored annotations in the reference database to augment and add meaning to the current video
- Commercial software is available for performing the manipulations disclosed in
FIG. 4 , or alternative methods of combining adjacent images having some overlap, known as “mosaic tiling” may be deployed. For example, Observera Inc. of 4451 Brookfield Corporate Drive, Suite 107, Chantilly, Va. 20151-1693 provides software that has a range of features allowing modification to serve a variety of applications. In addition, Samoff Corporation and Pyramid Vision Technologies, both of 201 Washington Road, CN 5300 Princeton, N.J. 08543-5300, supply commercial software and hardware for performing Birds-Eye-View transformations to create fly by images. - The methods for generating the useful displayed image in
FIG. 3 can be deconstructed into a sequence of steps, although they need not all occur in a discrete manner, depending on the method of implementation. Thus,FIG. 4 illustrates one embodiment of operative principles for acquiring, transforming and aligning the image data used to generate the real-time display ofFIG. 3 . Accordingly, the following description should not be construed as limiting the scope of the patent. -
FIG. 4 illustrates an alternative embodiment for generating the image displayFIG. 3 image 401 is acquired the time t 1 whereas image 400 and to is acquired a time t2, accordingly the pair of image is represented bybracket 410 correspond to live actual images recorded for the moving vehicle. In process of generatingimage 420 for display, thefirst image frame 402 is acquired in either video or digital format. Then after movement or displacement of the camera on the vehicle, a second orlive image 402 is similarly acquired.Bracket 410 contains a digital representation of the actual images acquired at time t1 and t2 for further transformation and merger to form acomposite image 420. Although the Bird's Eye View can be created before or after merger of the images,images bracket 420 represent bird's eye view transformations of the corresponding adjacent images withinbracket 410. The rectangular image frames inbracket 410 distorted to trapezoidal shapes in generating the higher elevation or Birds-eye-view. One embodiment for generating such bird's eye view images optionally includes performing scaled transformation of the rectangular image frame to a trapezoid to simulate the loss of prospective as the birds eye view camera increases in azimuth angle from the actual viewpoint of the camera mounted on the vehicle. The trapezoid results from transforming each row of the x-axis gradually with increased compression starting from the upper edge of the picture frame of theactual view 410, with increasing compression towards the bottom of the frame. As shown inimage 403, a trapezoidal transformation decreases the divergence of the lines representing the highway traffic lanes in the images ofbracket 410. - Although
images 401 is preferably modified by digital processing to image 403 to correspond to the expected appearance fromcamera position 130 inFIG. 2 , alternative viewpoints are possible, including a position forward of the vehicle at the time t1 at whichimage 401 is recorded, providedimages - Once the transformed
image composite image 420, the scaling factor to convertimage 403 to image 405 must be determined, as well as any x and y displacement for overlay ofimage 405 onimage 404. - Further, virtual features, such as the image of the vehicle, frame separating the image regions, optional projection of any intended driving path, and the like, are preferably overlaid on the penultimate composite images to form the final composite image. Further, the composite image is most preferably refreshed in real time to reflect the forward progress of the vehicle. Thus,
image 405 is represented as de-magnified fromimage 403 to represent its scaling prior to merger overvirtual image 404 to createdisplay image 420. However, equivalent operations can be performed onimage 401, such that detail in the real time image is more fully preserved. - The displayed
image 420 is optionally generated by mergingde-magnified image 405 withimage 404, taking into account lateral translation and rotation of the actual cameras viewpoint between the acquisition offrames 401 at time t1 and 402 at time t2. The overlay can be determined by mapping the displacement of pixels from image frames 405 to 404, such that a selected sub group of pixels inimage 404 is replaced withimage frame 405 pixels prior to display. As the image acquired at time t1 and t2 are mapped to the same magnification, a relative movement or rotation is optionally determined by first searching each image field to identify high contrast features, and then comparing the relative orientation of these features to generate the appropriate correction factors. That is, when the correction factors are applied the high contrast features must coincide to compose an accurate virtual image for display. -
Image 403 is scaled to generateimage 405 by a de-magnification factor based on Z-axis displacement of the vehicle between image frames 401 and 402. The factor used to generateimage 405 fromimage 403 can be determined by several alternative methods. Thus, depending on the method of forming and generating thecomposite image 420, it is desirable to know the absolute movement of the actual camera position between times t1 and t2, as this determines the scaling factor for convertingimage 403 to 405 by geometric calculations, or magnifyingimages 404 with respect to 403. The Z-axis distance can determined by several methods, including but not limited to global satellite positioning, or calculating the change in position by integrating the speedometer output overtime to synchronize time period between t1 and t2. - To the extent that the initial images are readily acquired in digital format by converting an analog video feed into an JPEG or MPEG format data stream, the correction factors can be generated from selected parameters of the digital data streams. Briefly, JPEG and MPEG data format transmits full images, or I frames, infrequently to conserve bandwidth, using a sequence of intervening frames (B and P type in MPEG) to communicate the changes to portions of the image between I frames. In forming the MPEG/JPEG data stream the image is broken down into macro blocks, which is collections of pixels, and analyzed to identify macro blocks that change location between successive image frames, which are then used to reconstruct the full image for final display. In the MPEG format both B and P frames identify and track macro blocks that change location between I frames. Specifically, the translation and rotation of
image 404 withrespect image 405 necessary for merging these images may be determined from the movement of macro block represented in the JPEG and MPEG formats by extracting an average macro block translation to represent the relative movement between consecutive I frames. Thus, the vector sum of the individual translation factors can be applied to alignimages - It should be appreciated by one of ordinary skill in the art that the position of car 310 in
FIG. 3 is synthetically generated based on the data set representing the actual car's dimensions, and the selection of the birds eye view position, that is, Z, Y position inFIG. 1 . For example by specifying two or more parameters which may include, the azimuth angle, vertical height above the vehicle, horizontal distance behind the vehicle (at time t2), viewing angle and like parameters define the Z and Y positions necessary to determine the appropriate transformation factors used to generate the pair transformed images inbracket 415 from the corresponding actual images inbracket 410 inFIG. 4 . - In another alternative embodiment, MPEG conversion can be limited to
images images - Further, in yet another embodiment the aforementioned method of macro block tracking can be extended to determine the factor used to compute the magnification of
image 403 for generatingimage 405 while simultaneously correcting for what has been initially described as discrete steps of image rotation and translations determination, i.e. the steps used to placeimage 405 inimage 404. Although it may be possible in some instances to identify one or more macro blocks that correspond to distinct objects or edges of the vision field it is unlikely that the same macro blocks can be uniquely identified for each frame of the video source from t1 to t2. However, this is not necessarily, provided a refreshed or updated subset of macro blocks is used at each I-frame. The updated subset would correspond to the same x-y coordinate range of the macro blocks in the previous I frame, as updated to reflect the most recent preceding image. More specifically, using the MPEG bit stream to trace the displacement of macro blocks at corners ofimage frame 402 includes correction for magnification, translation and rotation. Reverse tracking the relative positions of macro blocks that correspond to the corner regions offrame 402 define a relative position for edges of the frame acquired at time t1 frame with respect to the frame acquired at time t2. Thus applying a linear scaling between the compression and distortion ratios necessary to re-map the frame corner also accomplishes translation and rotation. It is anticipated that either offrames images - Alternatively, if the driver is concerned about a particular image feature or region of the live image, the translation and rotation factors are preferably acquired by selecting the corresponding macro blocks that represent such features. The previous calculation, likewise carried out by starting with the last
live frame 402, may be carried out b averaging (before accumulating the sum thereof) a limited number of macro block translation factors depending on the area selected. Although the identical macro blocks used as the starting point for the reverse computation (from a particular region of the live image 402) may only maintain the same identity between I Frames in the MPEG bit stream, the accuracy is the likely to be improved by selecting a newer subset of macro blocks that correspond to the same x-y coordinate range of the macro blocks in the previous I frame. - Thus, the steps in executing the aforementioned method of macro block tracking include; 1) identifying the first set of nearest neighbor macro blocks corresponding to corners of frame 402 or selected portions of the live image, 2) record in the average translation to the previous I frame, 3) record the average x-y coordinate position corresponding to average translation to previous I frame, 4) identifying a second set of nearest neighbor macro blocks corresponding to average x-y coordinate positions, 5) recording the average translation to the next previous I frame, 6) compute the sum of the first average translation and each subsequent average translation for each of the corresponding four corners of frame 402 (or selected regions therein), 7) repeating the previous steps of identifying the second set of nearest neighbor's until the subsequent I frame corresponds to the closest I frame from image frame 401, 8) linearly distorting one of image frame 401 to 402 to aligned the corresponding corners according to their respective translation factors, 9) either before or after merging the distorted and undistorted image frame from the previous step generating a second composite image by distorting a first composite image to correspond with the position of the virtual camera, 10) calculating a second pixel subset corresponding to the profile of the vehicle as determined by the position of the virtual camera, 11) replacing selected pixels in the second composite image with the second subset of pixels to form a third composite image, and 12) displaying the third composite image.
-
FIG. 5 is a plain view illustrating the use of the instant invention for assisting a truck driver to backup atruck rig 600 into aloading ramp 610. The driver views a display that provides a composite of a live and recorded image from camera 640 (with viewing angle 641) according to the teachings of the invention with virtual camera position 630 (having viewing angle 631) such that the drivers “sees” the corners orother obstacles -
FIG. 6 is a plan view illustrating the use of the instant invention for the command and control (optionally from vehicle 705) of a variety of combat vehicles 700-704 executing a mission was spreading out over the terrain. Alternativevirtual camera positions 730 a and 730 b allow a wide-angle view of the battlefield from any vehicle, with the images being acquired from manned vehicles 700-705 or drones 706. The image preferably shows the actual view from each vehicle, identifies “friend” and “foe” with additional icons thus avoiding friendly fire accidents. - It should be appreciated that the images described and combined need not be solely from visual sources, but include IR, NIR and other non-visual sources, and may combine visual images with non-visual or enhanced images in either monoscopic or stereoscopic views in the final composite images.
- While the invention has been described in connection with a preferred embodiment, it is not intended to limit the scope of the invention to the particular form set forth, but on the contrary, it is intended to cover such alternatives, modifications, and equivalents as may be within the spirit and scope of the invention as defined by the appended claims. For example, it should be appreciated that the alternative methods of forming a composite image disclosed herein can be combined with any of the prior art methods of digital image processing provided the real-time images are either acquired in digital format or converted to digital format from an analog video recorder or camera.
Claims (2)
1. A method for generating a dynamic composite image by combining a live image and a previously acquired or generated input image, the method comprising the steps of aligning said live image with said previous image, wherein the alignment is performed before or after the input image and the live image are converted to represent a common virtual vantage point.
2. A method of illustrating relative position of a vehicle with respect to its surroundings comprising taking a first image at a first time and taking a second image at a second time, superimposing the first and second images and correcting the first and second images to account for different image sizes due ot the difference in the first and second times.
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/914,375 US20050031169A1 (en) | 2003-08-09 | 2004-08-09 | Birds eye view virtual imaging for real time composited wide field of view |
US12/259,227 US8295644B2 (en) | 2003-08-09 | 2008-10-27 | Birds eye view virtual imaging for real time composited wide field of view |
US13/657,338 US20130169675A1 (en) | 2003-08-09 | 2012-10-22 | Birds eye view virtual imaging for real time composited wide field of view |
US14/300,741 US20140285516A1 (en) | 2003-08-09 | 2014-06-10 | Birds eye view virtual imaging for real time composited wide field of view |
US15/273,635 US20170234692A1 (en) | 2003-08-09 | 2016-09-22 | Birds eye view virtual imaging for real time composited wide field of view |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US49357903P | 2003-08-09 | 2003-08-09 | |
US10/914,375 US20050031169A1 (en) | 2003-08-09 | 2004-08-09 | Birds eye view virtual imaging for real time composited wide field of view |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/259,227 Continuation US8295644B2 (en) | 2003-08-09 | 2008-10-27 | Birds eye view virtual imaging for real time composited wide field of view |
Publications (1)
Publication Number | Publication Date |
---|---|
US20050031169A1 true US20050031169A1 (en) | 2005-02-10 |
Family
ID=34119116
Family Applications (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/914,375 Abandoned US20050031169A1 (en) | 2003-08-09 | 2004-08-09 | Birds eye view virtual imaging for real time composited wide field of view |
US12/259,227 Active - Reinstated US8295644B2 (en) | 2003-08-09 | 2008-10-27 | Birds eye view virtual imaging for real time composited wide field of view |
US13/657,338 Abandoned US20130169675A1 (en) | 2003-08-09 | 2012-10-22 | Birds eye view virtual imaging for real time composited wide field of view |
US14/300,741 Abandoned US20140285516A1 (en) | 2003-08-09 | 2014-06-10 | Birds eye view virtual imaging for real time composited wide field of view |
US15/273,635 Abandoned US20170234692A1 (en) | 2003-08-09 | 2016-09-22 | Birds eye view virtual imaging for real time composited wide field of view |
Family Applications After (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/259,227 Active - Reinstated US8295644B2 (en) | 2003-08-09 | 2008-10-27 | Birds eye view virtual imaging for real time composited wide field of view |
US13/657,338 Abandoned US20130169675A1 (en) | 2003-08-09 | 2012-10-22 | Birds eye view virtual imaging for real time composited wide field of view |
US14/300,741 Abandoned US20140285516A1 (en) | 2003-08-09 | 2014-06-10 | Birds eye view virtual imaging for real time composited wide field of view |
US15/273,635 Abandoned US20170234692A1 (en) | 2003-08-09 | 2016-09-22 | Birds eye view virtual imaging for real time composited wide field of view |
Country Status (1)
Country | Link |
---|---|
US (5) | US20050031169A1 (en) |
Cited By (44)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070198146A1 (en) * | 2004-05-19 | 2007-08-23 | Honda Motor Co., Ltd. | Traffic lane marking line recognition system for vehicle |
US20070225913A1 (en) * | 2005-05-17 | 2007-09-27 | Tetsuo Ikeda | Traffic Lane Marking Line Recognition System for Vehicle |
US20070239357A1 (en) * | 2006-03-30 | 2007-10-11 | Aisin Aw Co., Ltd. | Driving support method and driving support device |
US20070253622A1 (en) * | 2004-05-19 | 2007-11-01 | Tetsuo Ikeda | Traffic Lane Marking Line Recognition System for Vehicle |
US20080013790A1 (en) * | 2006-07-13 | 2008-01-17 | Toru Ihara | Moving-state determining device |
US20080027627A1 (en) * | 2004-05-19 | 2008-01-31 | Honda Motor Co., Ltd. | Traffic Lane Marking Line Recognition System for Vehicle |
EP1921419A1 (en) * | 2005-09-02 | 2008-05-14 | Matsushita Electric Industrial Co., Ltd. | Image display device and image generation device |
US20080129756A1 (en) * | 2006-09-26 | 2008-06-05 | Hirotaka Iwano | Image generating apparatus and image generating method |
US20080266324A1 (en) * | 2007-04-30 | 2008-10-30 | Navteq North America, Llc | Street level video simulation display system and method |
US20090021609A1 (en) * | 2007-07-16 | 2009-01-22 | Trw Automotive U.S. Llc | Method and apparatus for distortion correction and image enhancing of a vehicle rear viewing system |
US20090073265A1 (en) * | 2006-04-13 | 2009-03-19 | Curtin University Of Technology | Virtual observer |
CN101489117A (en) * | 2005-06-07 | 2009-07-22 | 日产自动车株式会社 | Apparatus and method for displaying images |
US20090231431A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | Displayed view modification in a vehicle-to-vehicle network |
US20090231432A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | View selection in a vehicle-to-vehicle network |
US20090231158A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | Guided video feed selection in a vehicle-to-vehicle network |
US20090231433A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | Scene selection in a vehicle-to-vehicle network |
US20090248300A1 (en) * | 2008-03-31 | 2009-10-01 | Sony Ericsson Mobile Communications Ab | Methods and Apparatus for Viewing Previously-Recorded Multimedia Content from Original Perspective |
US20100017047A1 (en) * | 2005-06-02 | 2010-01-21 | The Boeing Company | Systems and methods for remote display of an enhanced image |
US20100054541A1 (en) * | 2008-08-26 | 2010-03-04 | National Taiwan University | Driving support system with plural dimension processing units |
US20100094541A1 (en) * | 2007-06-16 | 2010-04-15 | Bayerische Motoren Werke Aktiengesellschaft | Method for Assisting a Motor Vehicle Driver When Driving Through a Narrow Passage and/or for Maintaining a Safe Distance from a Vehicle in Front |
US20100165105A1 (en) * | 2006-08-18 | 2010-07-01 | Kazufumi Mizusawa | Vehicle-installed image processing apparatus and eye point conversion information generation method |
US20100225761A1 (en) * | 2008-08-07 | 2010-09-09 | Sanyo Electric Co., Ltd. | Maneuvering Assisting Apparatus |
US20100246901A1 (en) * | 2007-11-20 | 2010-09-30 | Sanyo Electric Co., Ltd. | Operation Support System, Vehicle, And Method For Estimating Three-Dimensional Object Area |
US20100295937A1 (en) * | 2009-05-20 | 2010-11-25 | International Business Machines Corporation | Transmitting a composite image |
US20100313146A1 (en) * | 2009-06-08 | 2010-12-09 | Battelle Energy Alliance, Llc | Methods and systems relating to an augmented virtuality environment |
WO2011047732A1 (en) * | 2009-10-22 | 2011-04-28 | Tele Atlas B.V. | Method for identifying moving foreground objects in an orthorectified photographic image |
US20110164790A1 (en) * | 2008-10-22 | 2011-07-07 | Kazuyuki Sakurai | Lane marking detection apparatus, lane marking detection method, and lane marking detection program |
US20120293613A1 (en) * | 2011-05-17 | 2012-11-22 | Occipital, Inc. | System and method for capturing and editing panoramic images |
CN103379314A (en) * | 2012-04-27 | 2013-10-30 | 能晶科技股份有限公司 | Aerial view acquisition system and aerial view acquisition method thereof |
EP2631696A3 (en) * | 2011-10-13 | 2015-02-11 | Aisin Seiki Kabushiki Kaisha | Image generator |
US20150106010A1 (en) * | 2013-10-15 | 2015-04-16 | Ford Global Technologies, Llc | Aerial data for vehicle navigation |
US9019347B2 (en) | 2011-10-13 | 2015-04-28 | Aisin Seiki Kabushiki Kaisha | Image generator |
US9230300B2 (en) | 2009-10-22 | 2016-01-05 | Tim Bekaert | Method for creating a mosaic image using masks |
US20160148421A1 (en) * | 2014-11-24 | 2016-05-26 | Caterpillar Inc. | Integrated Bird's Eye View with Situational Awareness |
US9558408B2 (en) | 2013-10-15 | 2017-01-31 | Ford Global Technologies, Llc | Traffic signal prediction |
US20180365501A1 (en) * | 2017-06-15 | 2018-12-20 | Blackberry Limited | Method & system for rear status detection |
US20190100144A1 (en) * | 2016-11-25 | 2019-04-04 | JVC Kenwood Corporation | Bird's-eye view video generation device, bird's-eye view video generation system, bird's-eye view video generation method, and non-transitory storage medium |
US20190329627A1 (en) * | 2011-11-20 | 2019-10-31 | Magna Electronics Inc. | Vehicular vision system with enhanced functionality |
US10922881B2 (en) * | 2018-11-02 | 2021-02-16 | Star Global Expert Solutions Joint Stock Company | Three dimensional/360 degree (3D/360°) real-time full information smart management integrated mapping system (SMIMS) and process of generating the same |
US11064184B2 (en) | 2017-08-25 | 2021-07-13 | Aurora Flight Sciences Corporation | Aerial vehicle imaging and targeting system |
US11074827B2 (en) * | 2017-08-25 | 2021-07-27 | Aurora Flight Sciences Corporation | Virtual reality system for aerial vehicle |
US11126204B2 (en) | 2017-08-25 | 2021-09-21 | Aurora Flight Sciences Corporation | Aerial vehicle interception system |
US11329937B2 (en) * | 2019-06-27 | 2022-05-10 | Rovi Guides, Inc. | Identifying non-focus area of content item and overlaying another content item onto the non-focus area |
US20220292846A1 (en) * | 2019-08-28 | 2022-09-15 | Toyota Motor Europe | Method and system for processing a plurality of images so as to detect lanes on a road |
Families Citing this family (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2010127351A1 (en) * | 2009-05-01 | 2010-11-04 | Aai Corporation | Method, apparatus, system and computer program product for automated collection and correlation for tactical information |
JP5251927B2 (en) * | 2010-06-21 | 2013-07-31 | 日産自動車株式会社 | Moving distance detection device and moving distance detection method |
JP5413516B2 (en) | 2010-08-19 | 2014-02-12 | 日産自動車株式会社 | Three-dimensional object detection apparatus and three-dimensional object detection method |
JP5389002B2 (en) * | 2010-12-07 | 2014-01-15 | 日立オートモティブシステムズ株式会社 | Driving environment recognition device |
DE102011084084A1 (en) | 2011-10-06 | 2013-04-11 | Robert Bosch Gmbh | Display method and display system for a vehicle |
JP2013131878A (en) * | 2011-12-21 | 2013-07-04 | Sony Corp | Image processing device, image processing method, and program |
JP5456123B1 (en) * | 2012-09-20 | 2014-03-26 | 株式会社小松製作所 | Work vehicle periphery monitoring system and work vehicle |
JP5629740B2 (en) * | 2012-09-21 | 2014-11-26 | 株式会社小松製作所 | Work vehicle periphery monitoring system and work vehicle |
US9842418B1 (en) * | 2013-09-07 | 2017-12-12 | Google Inc. | Generating compositions |
US9406114B2 (en) * | 2014-02-18 | 2016-08-02 | Empire Technology Development Llc | Composite image generation to remove obscuring objects |
US9531952B2 (en) * | 2015-03-27 | 2016-12-27 | Google Inc. | Expanding the field of view of photograph |
KR102375411B1 (en) * | 2015-05-11 | 2022-03-18 | 삼성전자주식회사 | Method and apparatus for providing around view of vehicle |
GB2556797B (en) * | 2015-11-06 | 2018-10-24 | Mitsubishi Electric Corp | Image processing apparatus, image processing method, and image processing program |
JP6665605B2 (en) * | 2016-03-15 | 2020-03-13 | 株式会社デンソー | Display control device and display control method |
JP6730613B2 (en) * | 2017-02-28 | 2020-07-29 | 株式会社Jvcケンウッド | Overhead video generation device, overhead video generation system, overhead video generation method and program |
US10650256B2 (en) * | 2017-04-18 | 2020-05-12 | nuTonomy Inc. | Automatically perceiving travel signals |
US10643084B2 (en) | 2017-04-18 | 2020-05-05 | nuTonomy Inc. | Automatically perceiving travel signals |
US10684137B2 (en) * | 2017-11-29 | 2020-06-16 | Deere & Company | Work site monitoring system and method |
WO2019130827A1 (en) * | 2017-12-25 | 2019-07-04 | キヤノン株式会社 | Image processing apparatus and control method therefor |
CN111886624A (en) * | 2018-03-28 | 2020-11-03 | 三菱电机株式会社 | Image processing apparatus, image processing method, and image processing program |
GB2589247B (en) | 2019-01-29 | 2023-07-05 | Motional Ad Llc | Traffic light estimation |
EP4087236A1 (en) * | 2021-05-03 | 2022-11-09 | Honeywell International Inc. | Video surveillance system with vantage point transformation |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030165255A1 (en) * | 2001-06-13 | 2003-09-04 | Hirohiko Yanagawa | Peripheral image processor of vehicle and recording medium |
Family Cites Families (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5420940A (en) * | 1992-06-16 | 1995-05-30 | Hughes Training, Inc. | CGSI pipeline performance improvement |
US5587929A (en) * | 1994-09-02 | 1996-12-24 | Caterpillar Inc. | System and method for tracking objects using a detection system |
JP3474022B2 (en) * | 1995-04-20 | 2003-12-08 | 株式会社日立製作所 | Map display device, map display method, arithmetic processing unit for map display device, and navigation system |
US6654014B2 (en) * | 1995-04-20 | 2003-11-25 | Yoshinori Endo | Bird's-eye view forming method, map display apparatus and navigation system |
KR100263366B1 (en) * | 1995-06-09 | 2000-08-01 | 하기와라 가즈토시 | Map display apparatus |
EP2259220A3 (en) * | 1998-07-31 | 2012-09-26 | Panasonic Corporation | Method and apparatus for displaying image |
JP2000161915A (en) * | 1998-11-26 | 2000-06-16 | Matsushita Electric Ind Co Ltd | On-vehicle single-camera stereoscopic vision system |
US6396535B1 (en) * | 1999-02-16 | 2002-05-28 | Mitsubishi Electric Research Laboratories, Inc. | Situation awareness system |
JP2000341501A (en) * | 1999-03-23 | 2000-12-08 | Minolta Co Ltd | Device and method for processing image and recording medium with image processing program stored therein |
WO2000064175A1 (en) * | 1999-04-16 | 2000-10-26 | Matsushita Electric Industrial Co., Ltd. | Image processing device and monitoring system |
EP1050866B1 (en) * | 1999-04-28 | 2003-07-09 | Matsushita Electric Industrial Co., Ltd. | Parking assistance device and method |
US6704621B1 (en) * | 1999-11-26 | 2004-03-09 | Gideon P. Stein | System and method for estimating ego-motion of a moving vehicle using successive images recorded along the vehicle's path of motion |
JP2001315603A (en) * | 2000-05-09 | 2001-11-13 | Matsushita Electric Ind Co Ltd | Drive supporting device |
JP3599639B2 (en) * | 2000-05-26 | 2004-12-08 | 松下電器産業株式会社 | Image processing device |
JP3269056B2 (en) * | 2000-07-04 | 2002-03-25 | 松下電器産業株式会社 | Monitoring system |
JP3773433B2 (en) * | 2000-10-11 | 2006-05-10 | シャープ株式会社 | Ambient monitoring device for moving objects |
JP3297040B1 (en) * | 2001-04-24 | 2002-07-02 | 松下電器産業株式会社 | Image composing and displaying method of vehicle-mounted camera and apparatus therefor |
JP2002334322A (en) * | 2001-05-10 | 2002-11-22 | Sharp Corp | System, method and program for perspective projection image generation, and storage medium stored with perspective projection image generating program |
JP2003066944A (en) * | 2001-08-29 | 2003-03-05 | Denso Corp | Map image processor and program |
JP3695377B2 (en) * | 2001-10-01 | 2005-09-14 | 日産自動車株式会社 | Image composition apparatus and image composition method |
KR100866450B1 (en) * | 2001-10-15 | 2008-10-31 | 파나소닉 주식회사 | Automobile surrounding observation device and method for adjusting the same |
US7145519B2 (en) * | 2002-04-18 | 2006-12-05 | Nissan Motor Co., Ltd. | Image display apparatus, method, and program for automotive vehicle |
-
2004
- 2004-08-09 US US10/914,375 patent/US20050031169A1/en not_active Abandoned
-
2008
- 2008-10-27 US US12/259,227 patent/US8295644B2/en active Active - Reinstated
-
2012
- 2012-10-22 US US13/657,338 patent/US20130169675A1/en not_active Abandoned
-
2014
- 2014-06-10 US US14/300,741 patent/US20140285516A1/en not_active Abandoned
-
2016
- 2016-09-22 US US15/273,635 patent/US20170234692A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030165255A1 (en) * | 2001-06-13 | 2003-09-04 | Hirohiko Yanagawa | Peripheral image processor of vehicle and recording medium |
Cited By (79)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070198146A1 (en) * | 2004-05-19 | 2007-08-23 | Honda Motor Co., Ltd. | Traffic lane marking line recognition system for vehicle |
US20070253622A1 (en) * | 2004-05-19 | 2007-11-01 | Tetsuo Ikeda | Traffic Lane Marking Line Recognition System for Vehicle |
US7421095B2 (en) * | 2004-05-19 | 2008-09-02 | Honda Motor Co., Ltd. | Traffic lane marking line recognition system for vehicle |
US20080027627A1 (en) * | 2004-05-19 | 2008-01-31 | Honda Motor Co., Ltd. | Traffic Lane Marking Line Recognition System for Vehicle |
US7421094B2 (en) * | 2004-05-19 | 2008-09-02 | Honda Motor Co., Ltd. | Traffic lane marking line recognition system for vehicle |
US7415133B2 (en) * | 2004-05-19 | 2008-08-19 | Honda Motor Co., Ltd. | Traffic lane marking line recognition system for vehicle |
US7415134B2 (en) * | 2005-05-17 | 2008-08-19 | Honda Motor Co., Ltd. | Traffic lane marking line recognition system for vehicle |
US20070225913A1 (en) * | 2005-05-17 | 2007-09-27 | Tetsuo Ikeda | Traffic Lane Marking Line Recognition System for Vehicle |
US20110187563A1 (en) * | 2005-06-02 | 2011-08-04 | The Boeing Company | Methods for remote display of an enhanced image |
US7925391B2 (en) * | 2005-06-02 | 2011-04-12 | The Boeing Company | Systems and methods for remote display of an enhanced image |
US8874284B2 (en) | 2005-06-02 | 2014-10-28 | The Boeing Company | Methods for remote display of an enhanced image |
US20100017047A1 (en) * | 2005-06-02 | 2010-01-21 | The Boeing Company | Systems and methods for remote display of an enhanced image |
CN101489117A (en) * | 2005-06-07 | 2009-07-22 | 日产自动车株式会社 | Apparatus and method for displaying images |
EP1921419A1 (en) * | 2005-09-02 | 2008-05-14 | Matsushita Electric Industrial Co., Ltd. | Image display device and image generation device |
EP1921419A4 (en) * | 2005-09-02 | 2011-09-07 | Panasonic Corp | Image display device and image generation device |
US20070239357A1 (en) * | 2006-03-30 | 2007-10-11 | Aisin Aw Co., Ltd. | Driving support method and driving support device |
US20090073265A1 (en) * | 2006-04-13 | 2009-03-19 | Curtin University Of Technology | Virtual observer |
US9420234B2 (en) * | 2006-04-13 | 2016-08-16 | Virtual Observer Pty Ltd | Virtual observer |
US20080013790A1 (en) * | 2006-07-13 | 2008-01-17 | Toru Ihara | Moving-state determining device |
US7965871B2 (en) | 2006-07-13 | 2011-06-21 | Mitsubishi Fuso Truck And Bus Corporation | Moving-state determining device |
US20100165105A1 (en) * | 2006-08-18 | 2010-07-01 | Kazufumi Mizusawa | Vehicle-installed image processing apparatus and eye point conversion information generation method |
US20080129756A1 (en) * | 2006-09-26 | 2008-06-05 | Hirotaka Iwano | Image generating apparatus and image generating method |
US8368687B2 (en) * | 2006-09-26 | 2013-02-05 | Clarion Co., Ltd. | Image generating apparatus and image generating method |
US20080266324A1 (en) * | 2007-04-30 | 2008-10-30 | Navteq North America, Llc | Street level video simulation display system and method |
US9240029B2 (en) * | 2007-04-30 | 2016-01-19 | Here Global B.V. | Street level video simulation display system and method |
US20100094541A1 (en) * | 2007-06-16 | 2010-04-15 | Bayerische Motoren Werke Aktiengesellschaft | Method for Assisting a Motor Vehicle Driver When Driving Through a Narrow Passage and/or for Maintaining a Safe Distance from a Vehicle in Front |
US10214143B2 (en) * | 2007-06-16 | 2019-02-26 | Bayerische Motoren Werke Aktiengesellschaft | Method for assisting a motor vehicle driver when driving through a narrow passage and/or for maintaining a safe distance from a vehicle in front |
US8233045B2 (en) * | 2007-07-16 | 2012-07-31 | Trw Automotive U.S. Llc | Method and apparatus for distortion correction and image enhancing of a vehicle rear viewing system |
US20090021609A1 (en) * | 2007-07-16 | 2009-01-22 | Trw Automotive U.S. Llc | Method and apparatus for distortion correction and image enhancing of a vehicle rear viewing system |
DE102008031784B4 (en) * | 2007-07-16 | 2016-05-12 | Trw Automotive U.S. Llc | Method and apparatus for distortion correction and image enhancement of a vehicle rearview system |
US20100246901A1 (en) * | 2007-11-20 | 2010-09-30 | Sanyo Electric Co., Ltd. | Operation Support System, Vehicle, And Method For Estimating Three-Dimensional Object Area |
US8345098B2 (en) * | 2008-03-17 | 2013-01-01 | International Business Machines Corporation | Displayed view modification in a vehicle-to-vehicle network |
US10671259B2 (en) | 2008-03-17 | 2020-06-02 | International Business Machines Corporation | Guided video feed selection in a vehicle-to-vehicle network |
US20090231431A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | Displayed view modification in a vehicle-to-vehicle network |
US20090231432A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | View selection in a vehicle-to-vehicle network |
US20090231158A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | Guided video feed selection in a vehicle-to-vehicle network |
US20090231433A1 (en) * | 2008-03-17 | 2009-09-17 | International Business Machines Corporation | Scene selection in a vehicle-to-vehicle network |
US9123241B2 (en) | 2008-03-17 | 2015-09-01 | International Business Machines Corporation | Guided video feed selection in a vehicle-to-vehicle network |
US9043483B2 (en) | 2008-03-17 | 2015-05-26 | International Business Machines Corporation | View selection in a vehicle-to-vehicle network |
US8400507B2 (en) | 2008-03-17 | 2013-03-19 | International Business Machines Corporation | Scene selection in a vehicle-to-vehicle network |
US20090248300A1 (en) * | 2008-03-31 | 2009-10-01 | Sony Ericsson Mobile Communications Ab | Methods and Apparatus for Viewing Previously-Recorded Multimedia Content from Original Perspective |
US20100225761A1 (en) * | 2008-08-07 | 2010-09-09 | Sanyo Electric Co., Ltd. | Maneuvering Assisting Apparatus |
US8213683B2 (en) * | 2008-08-26 | 2012-07-03 | National Taiwan University | Driving support system with plural dimension processing units |
US20100054541A1 (en) * | 2008-08-26 | 2010-03-04 | National Taiwan University | Driving support system with plural dimension processing units |
US20110164790A1 (en) * | 2008-10-22 | 2011-07-07 | Kazuyuki Sakurai | Lane marking detection apparatus, lane marking detection method, and lane marking detection program |
US8594380B2 (en) * | 2008-10-22 | 2013-11-26 | Nec Corporation | Lane marking detection apparatus, lane marking detection method, and lane marking detection program |
US8416300B2 (en) * | 2009-05-20 | 2013-04-09 | International Business Machines Corporation | Traffic system for enhancing driver visibility |
US9706176B2 (en) | 2009-05-20 | 2017-07-11 | International Business Machines Corporation | Traffic system for enhancing driver visibility |
US8817099B2 (en) | 2009-05-20 | 2014-08-26 | International Business Machines Corporation | Traffic system for enhancing driver visibility |
US20100295937A1 (en) * | 2009-05-20 | 2010-11-25 | International Business Machines Corporation | Transmitting a composite image |
US8732592B2 (en) * | 2009-06-08 | 2014-05-20 | Battelle Energy Alliance, Llc | Methods and systems relating to an augmented virtuality environment |
US20100313146A1 (en) * | 2009-06-08 | 2010-12-09 | Battelle Energy Alliance, Llc | Methods and systems relating to an augmented virtuality environment |
US9230300B2 (en) | 2009-10-22 | 2016-01-05 | Tim Bekaert | Method for creating a mosaic image using masks |
WO2011047732A1 (en) * | 2009-10-22 | 2011-04-28 | Tele Atlas B.V. | Method for identifying moving foreground objects in an orthorectified photographic image |
US20120293613A1 (en) * | 2011-05-17 | 2012-11-22 | Occipital, Inc. | System and method for capturing and editing panoramic images |
US9019347B2 (en) | 2011-10-13 | 2015-04-28 | Aisin Seiki Kabushiki Kaisha | Image generator |
EP2631696A3 (en) * | 2011-10-13 | 2015-02-11 | Aisin Seiki Kabushiki Kaisha | Image generator |
US20190329627A1 (en) * | 2011-11-20 | 2019-10-31 | Magna Electronics Inc. | Vehicular vision system with enhanced functionality |
US11794553B2 (en) | 2011-11-20 | 2023-10-24 | Magna Electronics Inc. | Vehicular vision system with enhanced functionality |
US11267313B2 (en) | 2011-11-20 | 2022-03-08 | Magna Electronics Inc. | Vehicular vision system with enhanced functionality |
US10787056B2 (en) * | 2011-11-20 | 2020-09-29 | Magna Electronics Inc. | Vehicular vision system with enhanced functionality |
US20130286210A1 (en) * | 2012-04-27 | 2013-10-31 | Altek Autotronics Corp. | Bird-View Image Capture System and Bird-View Image Capture Method Thereof |
CN103379314A (en) * | 2012-04-27 | 2013-10-30 | 能晶科技股份有限公司 | Aerial view acquisition system and aerial view acquisition method thereof |
US20150106010A1 (en) * | 2013-10-15 | 2015-04-16 | Ford Global Technologies, Llc | Aerial data for vehicle navigation |
US9558408B2 (en) | 2013-10-15 | 2017-01-31 | Ford Global Technologies, Llc | Traffic signal prediction |
US20160148421A1 (en) * | 2014-11-24 | 2016-05-26 | Caterpillar Inc. | Integrated Bird's Eye View with Situational Awareness |
US20190100144A1 (en) * | 2016-11-25 | 2019-04-04 | JVC Kenwood Corporation | Bird's-eye view video generation device, bird's-eye view video generation system, bird's-eye view video generation method, and non-transitory storage medium |
US10710505B2 (en) * | 2016-11-25 | 2020-07-14 | JVC Kenwood Corporation | Bird's-eye view video generation device, bird's-eye view video generation system, bird's-eye view video generation method, and non-transitory storage medium |
US20180365501A1 (en) * | 2017-06-15 | 2018-12-20 | Blackberry Limited | Method & system for rear status detection |
US10949680B2 (en) * | 2017-06-15 | 2021-03-16 | Blackberry Limited | Method and system for rear status detection |
US10339392B2 (en) * | 2017-06-15 | 2019-07-02 | Blackberry Limited | Method and system for rear status detection |
US11064184B2 (en) | 2017-08-25 | 2021-07-13 | Aurora Flight Sciences Corporation | Aerial vehicle imaging and targeting system |
US11074827B2 (en) * | 2017-08-25 | 2021-07-27 | Aurora Flight Sciences Corporation | Virtual reality system for aerial vehicle |
US11126204B2 (en) | 2017-08-25 | 2021-09-21 | Aurora Flight Sciences Corporation | Aerial vehicle interception system |
US10922881B2 (en) * | 2018-11-02 | 2021-02-16 | Star Global Expert Solutions Joint Stock Company | Three dimensional/360 degree (3D/360°) real-time full information smart management integrated mapping system (SMIMS) and process of generating the same |
US11329937B2 (en) * | 2019-06-27 | 2022-05-10 | Rovi Guides, Inc. | Identifying non-focus area of content item and overlaying another content item onto the non-focus area |
US11683285B2 (en) | 2019-06-27 | 2023-06-20 | Rovi Guides, Inc. | Enhancing a social media post with content that is relevant to the audience of the post |
US20220292846A1 (en) * | 2019-08-28 | 2022-09-15 | Toyota Motor Europe | Method and system for processing a plurality of images so as to detect lanes on a road |
US11900696B2 (en) * | 2019-08-28 | 2024-02-13 | Toyota Motor Europe | Method and system for processing a plurality of images so as to detect lanes on a road |
Also Published As
Publication number | Publication date |
---|---|
US20130169675A1 (en) | 2013-07-04 |
US20140285516A1 (en) | 2014-09-25 |
US20170234692A1 (en) | 2017-08-17 |
US20090092334A1 (en) | 2009-04-09 |
US8295644B2 (en) | 2012-10-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8295644B2 (en) | Birds eye view virtual imaging for real time composited wide field of view | |
US11893682B1 (en) | Method for rendering 2D and 3D data within a 3D virtual environment | |
US8036827B2 (en) | Cognitive change detection system | |
JP5208203B2 (en) | Blind spot display device | |
JP3286306B2 (en) | Image generation device and image generation method | |
EP2423871B1 (en) | Apparatus and method for generating an overview image of a plurality of images using an accuracy information | |
EP2423873B1 (en) | Apparatus and Method for Generating an Overview Image of a Plurality of Images Using a Reference Plane | |
CN111046743B (en) | Barrier information labeling method and device, electronic equipment and storage medium | |
Varga et al. | Super-sensor for 360-degree environment perception: Point cloud segmentation using image features | |
US11288785B2 (en) | Virtual overlay system and method for occluded objects | |
US20080309763A1 (en) | Driving Support System And Vehicle | |
JP2002135765A (en) | Camera calibration instruction device and camera calibration device | |
EP2491527B1 (en) | Method for creating a mosaic image using masks | |
WO2006035755A1 (en) | Method for displaying movable-body navigation information and device for displaying movable-body navigation information | |
US10863111B2 (en) | Method and system for generating a composed top-view image of a road | |
JP4272966B2 (en) | 3DCG synthesizer | |
US8860810B2 (en) | Method and device for extending a visibility area | |
US11380111B2 (en) | Image colorization for vehicular camera images | |
CN110428361A (en) | A kind of multiplex image acquisition method based on artificial intelligence | |
Noda et al. | Road image update using in-vehicle camera images and aerial image | |
Solbrig et al. | Online annotation of airborne surveillance and reconnaissance videos | |
US20230386059A1 (en) | Warping depth features for depth estimation | |
US20220390252A1 (en) | Use of predefined (pre-built) graphical representations of roads for autonomous driving of vehicles and display of route planning | |
Bansal et al. | Vision-based perception for autonomous urban navigation | |
Dumortier et al. | Real-time vehicle motion estimation using texture learning and monocular vision |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: DOUBLESHOT, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SHULMAN, ALAN;SNYDER, DONALD R., III;REEL/FRAME:019862/0704;SIGNING DATES FROM 20070910 TO 20070921 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |