US20150121222A1 - Method and System for Visualization Enhancement for Situational Awareness - Google Patents
Method and System for Visualization Enhancement for Situational Awareness Download PDFInfo
- Publication number
- US20150121222A1 US20150121222A1 US14/587,295 US201414587295A US2015121222A1 US 20150121222 A1 US20150121222 A1 US 20150121222A1 US 201414587295 A US201414587295 A US 201414587295A US 2015121222 A1 US2015121222 A1 US 2015121222A1
- Authority
- US
- United States
- Prior art keywords
- mission
- map
- search
- image
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G06F17/30061—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04842—Selection of displayed objects or displayed text elements
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/36—Input/output arrangements for on-board computers
- G01C21/3667—Display of a road map
- G01C21/367—Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/44—Browsing; Visualisation therefor
- G06F16/444—Spatial browsing, e.g. 2D maps, 3D or virtual spaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/0485—Scrolling or panning
- G06F3/04855—Interaction with scrollbars
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/102—Programmed access in sequence to addressed parts of tracks of operating record carriers
- G11B27/105—Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
Definitions
- This invention relates to robotic mapping systems.
- the invention relates to a robotic mapping system providing situational awareness based on previously recorded mapping data.
- Unmanned Ground Vehicles are useful for taking people out of harm's way. Although there have been many advances in using these vehicles, one of the key areas that is currently lacking is the poor situational awareness provided to operators.
- a robotic mapping system that provides a realistic, high-resolution three-dimensional graphics database that clearly depicts and geo-locates the interior and exterior of a building, prior to soldiers entering the structure, provides a significant benefit to ground forces.
- a widely-used visualization tool consists of viewing a video stream from a platform's camera.
- the user cannot easily change perspective, zoom, follow the image frames in an order other than that with which they were captured, or gain a sense of depth for objects within the image.
- video imposes high bandwidth requirements.
- a strong visualization tool creates a very realistic representation of the environment that the user can navigate in any way desired.
- the present invention is an after-action and real time mission review tool that displays camera, ladar and navigation sensor data.
- the present invention allows the user to pan, tilt, and zoom through the images from the front and back cameras on the robot, while simultaneously viewing time/date information, along with any available navigation information (e.g., the latitude and longitude of the robot at that time instance).
- time/date information e.g., the latitude and longitude of the robot at that time instance.
- a visual representation of the path the robot traversed when the user clicks on the path, the image is automatically changed to the image corresponding to that position.
- aerial images of the area are available, the path can be plotted on the geo-referenced image.
- the tool geo-registers images and maps so that they can be retrieved based on location and time.
- the system allows the user to compare information from multiple data collections by “locking” the images as the operator pan, tilt and zooms.
- FIG. 1 illustrates the VESA concept: visualization tools
- FIGS. 2 a and 2 b show the three separate images from the cameras as well as the 360 degree omni-directional image
- FIG. 3 is a screen shot of the display of the computer interfaced for executing the method of the present invention
- FIG. 4 illustrates the navigational map component of the interface display used by the present invention
- FIG. 5 illustrates the status display component of the interface display used by the present invention
- FIG. 6 illustrates the camera image and timeline scroll bar component of the interface display used by the present invention
- FIG. 7 illustrates a touch screen embodiment of the interface display used by the present invention
- FIG. 8 is a screen shot of the display of an alternative embodiment of the computer interfaced for executing the method of the present invention.
- FIG. 9 is a screen shot illustrating the zoomable map feature of the display of the present invention.
- FIG. 10 is a screen shot of the event file interface of the present invention.
- FIG. 11 is a screen shot of the event adding feature of the present invention.
- FIGS. 12 and 13 are screen shots illustrating the display of an additional 3D image of the display of the present invention.
- FIG. 14 is a screen shot of the map viewer of the present invention.
- FIG. 15 is a screen shot of the results of a query illustrating navigation paths within the bounding box of the map viewer, along with a series of images from three different points on the timeline;
- FIG. 16 illustrates a real-time application interface
- FIG. 17 illustrates the situation where a first packet of an image was dropped, but the rest of the image is viewable
- FIG. 18 illustrates the situation where a live view is paused when the user drags the slide bar
- FIG. 19 illustrates where the user clicks on the barrel as the first triangulation point
- FIG. 20 illustrates where the user clicks on the barrel again in the subsequent frame as the second triangulation point
- FIGS. 20-22 illustrate where, in the next frame, the barrel is now automatically centered on the screen
- FIG. 23 illustrates an image comparison between the two missions where a person has left the scene and one of the barrels has been moved and another robot has entered the scene.
- VESA is defined as Visualization Enhancement for Situational Awareness.
- UGV is defined as is an unmanned ground vehicle that operates while in contact with the ground and without an onboard human presence. UGVs can be used for many applications where it may be inconvenient, dangerous, or impossible to have a human operator present. Generally, the vehicle will have a set of sensors to observe the environment, and will either autonomously make decisions about its behavior or pass the information to a human operator at a different location who will control the vehicle through teleoperation.
- LADAR also known as LIDAR
- LIDAR is an optical remote sensing technology that can measure the distance to, or other properties of a target by illuminating the target with light, often using pulses from a laser.
- LIDAR technology has application in geomatics, archaeology, geography, geology, geomorphology, seismology, forestry, remote sensing and atmospheric physics, as well as in airborne laser swath mapping (ALSM), laser altimetry and LIDAR contour mapping.
- ALAM airborne laser swath mapping
- LADAR Laser Detection and Ranging
- the term LADAR Laser Detection and Ranging
- laser radar is sometimes used, even though LIDAR does not employ microwaves or radio waves and therefore is not radar in the strict sense of the word.
- DAQ (also referred to as “buf” or “blackbox”) are files that record the navigation information of the UGV. These files are time and date stamped, making it simple to determine which files correspond to a mission.
- DBS is a network-accessible storage layer that will use the resources of many machines to provide transactional read-write access for fixed-size blocks of binary data.
- GUI graphical user interface
- a GUI represents the information and actions available to a user through graphical icons and visual indicators such as secondary notation, as opposed to text-based interfaces, typed command labels or text navigation. The actions are usually performed through direct manipulation of the graphical elements.
- VESA Visualization Enhancement for Situational Awareness toolkit
- the goal of the VESA toolkit project was to combine autonomy with accurate mapping and visualization capabilities for use on vehicles or robotic systems.
- the inventors worked on creating a visualization widget to display imagery, as well as mapping results, from a small robot which has traveled through an unknown environment.
- VESA is designed to work with a variety of sensors; while cameras form the base of the visualization tool, if a LADAR is present on the robot, then that information will be able to be displayed as well.
- the soldiers are then able to analyze the information prior to entering the structure. Because of VESA's capabilities, each soldier knows the basic structural layout, therefore greatly enhancing the team's situational awareness.
- the inventors have developed hardware and software for a sensor system that is specifically designed and implemented on a variety of platforms. These platforms were used to collect data during a first phase of VESA.
- the machine components of the present invention consist of the VESA concept: visualization tools as shown in FIG. 1 .
- the present camera system consists of three machine vision cameras 101 , 102 , and 103 with fisheye lenses. Each camera has a 150 degree field-of-view, and is mounted so that a user has access to a full 360 degree field-of-view around the vehicle. More cameras with a smaller field of view or fewer cameras with a larger field of view may be used.
- the inventors choose to use multiple cameras, as opposed to a single forward-looking lens/camera configuration, to significantly increase the area can be modeled and texture mapped.
- FIGS. 2 a and 2 b show the three separate images from the cameras 201 , 202 , and 203 , as well as the 360 degree omni-directional image 204 .
- the inventors developed the present invention Situational Awareness Toolkit.
- the main goal of this toolkit is to improve the situational awareness of a soldier by providing an intuitive and easy-to-use way to “walk through” and interact with the sensor and navigation data of the robot during and post mission.
- the method of the present invention is enabled by software executed on a computer machine.
- the interface 300 for a user interacting with the software running and/or executing the method of the present invention is shown in FIG. 3 .
- FIGS. 3 and 4 on the left side of the window 301 is the navigation map 400 as shown individually in FIG. 4
- the navigation map 400 displays the traveled path 401 in a first color, if available.
- the current position of the vehicle is indicated by a triangle 402 in a second color, where the front of the vehicle is designated by the point of the triangle.
- the field of view (FOV) of the camera is also indicated on the triangle by the two adjacent lines 403 and 404 , shown in a third color if available.
- FOV field of view
- the FOV lines 403 and 404 are centered towards the back of the vehicle, the camera image being displayed is taken from the rear camera of the vehicle. If the FOV lines 403 and 404 are spread far apart (indicating a wide FOV), the image is zoomed far out. Vice-versa, if the FOV lines 403 and 404 are close together, the image is zoomed in to a smaller section of the environment. If aerial imagery 405 is available, it can be displayed as well and the navigation map 400 superimposed or laid over top of the aerial imagery 405 using GPS to align the aerial imagery 405 with the navigation map 400 .
- a status display 500 is located in the bottom left corner of the window as shown in FIG. 3 and individually in FIG. 5 .
- the information displayed includes the date and time of the mission 501 , along with the GMT time in seconds 502 . Also displayed, if available, are the position of the robot in UTM and lat/long position 503 , and vehicle speed (in miles per hour and meters per second) 504 .
- a directional compass 505 located towards the right of the display info shows the vehicle's current cardinal direction of travel. Listed below the compass is the cardinal direction of the vehicle in degrees 506 .
- a de-warped image taken from the camera is located on the right side of the window 600 as shown in FIG. 3 and individually in FIG. 6 .
- the image displayed 601 is the image taken from the camera at a given point in time. Overlaid on the top of the image display 600 is the cardinal direction of the displayed camera image 602 .
- the direction 602 changes with camera panning within the still frame image.
- Below the camera image 601 is a horizontal scroll bar 603 used to progress through the timeline of the mission.
- Each slider increment 604 is equivalent to moving one second in time.
- the scroll bar 603 also allows a user to jump to any point from the beginning to the end of the trial.
- the left and right keyboard keys allow for quickly incrementing or decrementing of the slider 604 .
- the ‘Front’ and ‘Back’ buttons 605 and 606 next to the scroll bar 603 allow the user to toggle between the front camera and the rear camera of the vehicle.
- Beneath the scroll bar 603 is a set of playback controls 607 that allow the user to play 608 , pause 609 , rewind 610 , and fast forward 611 through the images. Repeatedly pressing the rewind 610 or fast-forward 611 buttons increases the speed.
- buttons can be displayed that replicate some of the features usually accessed with mouse commands.
- the touch screen button interface 700 is shown in FIG. 7 .
- the “+”and “ ⁇ ” buttons 701 and 702 are used to zoom in and out, respectively, while the “Reset” button 703 will re-center the navigation map. These buttons can be toggled on and off.
- the data required to run the present invention consists of image files (.jpg) from the vehicle's cameras and daq (referred to as “buf” or “blackbox”) files that record the navigation information. These files are time and date stamped, making it simple to determine which files correspond to the current mission.
- the goal is to put the all the image and daq files from a single mission into a folder on the computer on which a user will be running the software executing the method of the present invention.
- Each mission should have its own separate folder to ensure that the data loads correctly.
- the play/rewind/fast-forward controls will automatically cycle through the images.
- the user By clicking and dragging the camera image, the user can rotate the camera to further inspect the image.
- the scroll button on the mouse can be used to zoom in or out of the image when the cursor is on the image.
- clicking the left and right mouse buttons simultaneously and dragging on the image a user can also zoom in or out of the image.
- buttons 701 and 702 In Touch Screen Mode, a user can use the “+” and “ ⁇ ” buttons 701 and 702 to zoom in and out instead of using a mouse. Using the “Reset” Button 703 will reset the navigation map 400 .
- FIG. 8 an alternative user interface is shown with alternative, additional, and/or improved features are illustrated.
- the interface 800 of the alternative embodiment is for a user interacting with the software running and/or executing the method of the present invention is shown in FIG. 8 .
- the navigation map 801 displays the traveled path 802 of the vehicle as created by two-dimensional LADAR in a first color, if available.
- the LADAR maps 815 are now semi-transparent when compared to the embodiment of FIG. 3 , allowing the user to view the aerial imagery underneath the map.
- the current position of the vehicle is indicated by a triangle 803 in a second color, where the front of the vehicle is designated by the point of the triangle.
- the field of view (FOV) of the camera is also indicated on the triangle by the two adjacent lines 804 and 805 , shown in a third color if available.
- the camera image being displayed is taken from the rear camera of the vehicle. If the FOV lines 804 and 805 are spread far apart (indicating a wide FOV), the image is zoomed far out. Vice-versa, if the FOV lines 804 and 805 are close together, the image is zoomed in to a smaller section of the environment. If geo-referenced aerial imagery 806 is available, it can be overlaid beneath the path of the robot for better situational awareness. If aerial imagery 806 is available, it can be displayed as well and the navigation map 801 superimposed or laid over top of the aerial imagery 806 using GPS to align the aerial imagery 806 with the navigation map 801 .
- a status display 807 is located in the bottom left corner of the window as shown in FIG. 8
- the information displayed includes the date and time of the mission 808 , along with the GMT time in seconds 809 . Also displayed, if available, are the position of the robot in UTM and lat/long position 810 , and vehicle speed (in miles per hour and meters per second) 811 .
- a directional compass 812 located towards the right of the display info shows the vehicle's current cardinal direction of travel. Listed below the compass is the cardinal direction of the vehicle in degrees 813 .
- a de-warped image taken from the camera 814 is located on the right side of the window 800 as shown in FIG. 8 .
- the image displayed 814 is the image taken from the camera at a given point in time. Overlaid on the top of the image display 814 is the cardinal direction of the displayed camera image 815 . The direction 815 changes with camera panning within the still frame image.
- Below the camera image 814 is a horizontal scroll bar 816 used to progress through the timeline of the mission. Each slider increment 817 is equivalent to moving one second in time.
- the scroll bar 816 also allows a user to jump to any point from the beginning to the end of the trial.
- the left and right keyboard keys allow for quickly incrementing or decrementing of the slider 817 .
- buttons 818 and 819 next to the scroll bar 816 allow the user to toggle between the front camera and the rear camera of the vehicle.
- Beneath the scroll bar 816 is a set of playback controls 820 that allow the user to play 821 , pause 822 , rewind 823 , and fast forward 824 through the images. Repeatedly pressing the rewind 823 or fast-forward 824 buttons increases the speed.
- the map area 801 where the robot path 802 , LADAR 815 , and aerial imagery 806 are displayed can now be panned and zoomed to get a detailed view of the area.
- the play/rewind/fast-forward controls will automatically cycle through the images.
- the user By clicking and dragging the camera image, the user can rotate the camera to further inspect the image.
- the scroll button on the mouse can be used to zoom in or out of the image when the cursor is on the image.
- clicking the left and right mouse buttons simultaneously and dragging on the image a user can also zoom in or out of the image.
- a user can use the “+” and “ ⁇ ” buttons 827 and 825 to zoom in and out instead of using a mouse.
- Using the “Reset” Button 826 will reset the navigation map 800 .
- an event view mode is displayed that allows the user to tag events in the video.
- the event file format is generic so that it can be created using the present invention, or can be written by different processes on a robot to record any notable events automatically (e.g., emergency stops, periods of time with no plans, high speed or curvature events).
- Each timeline 1001 is zoom-able using the buttons 1002 and 1003 at the bottom of the screen 1000 .
- the user can drag on the timeline 1001 to display different portions 1004 of the timeline 1001 .
- an overview bar 1005 displays, as denoted by a first colored line 1006 , what portion of the overall timeline 1001 is being displayed and the current image location, a denoted by second colored line or bar 1007 .
- the second colored line or bar 1007 can be dragged to quickly access other portions of the timelines 1001 .
- Events are noted by a colored line or bar 1008 on the timeline 1001 .
- Each event can be selected and the comment 1009 from that event is displayed beneath the timelines 1001 .
- the (x,y) location 1010 of that comment 1009 is plotted as a colored circle 1010 on the map 1011 .
- the current image position is represented by a colored line 1012 on the timelines 1001 . That line 1012 can be dragged to move forward or backward in the images.
- each timeline 1001 can be moved up or down for easily comparing events, and can also be closed if no longer needed.
- the present invention only displayed the 360 image viewer 1013 ; when displaying 3D maps, a separate program was launched.
- the downside of this was that the two programs had separate controls, so the area being seen on the image would not correspond directly to the 3D data without a lot of manual work by the operator.
- the present invention includes a built-in 3D viewer.
- a 3D map 1201 is displayed to the right of the main image 1202 , and the viewpoints are locked to each other so that zooming and panning on the image will affect the 3D view and vice versa.
- the image 1202 and 3D map scenes 1201 are adjusted to match the pan/tilt/zoom commands of the operator.
- This map viewer is extremely flexible, allowing many different types of objects and textures to be displayed. This allows the present invention to be able to load in many different kinds of maps, including maps based on the Kinect, LADAR scanners, stereo cameras, etc., depending upon what sensors/mapping technologies are available.
- the goal of the new database/map server system is to give users an easy way to view and compare VESA data from multiple vehicles and/or from multiple data collections. Desired functionality includes being able to search for all data collected in a certain area, taken during a specified time range, or a combination of both. Future work may involve adding a “tag” based search which allows the user to annotate the image data and search by keywords.
- the map viewer UI can be seen in FIG. 14 .
- the map display 1400 can be zoomed and panned using mouse actions. Users can select the time ranges to search using the calendar inputs 1401 below the map display 1400 . Status information 1402 about can be also be seen to the right of the date/time selection 1403 .
- a timeline 1404 is featured at the bottom of the UI map display 1400 . When a search is conducted, events 1405 will appear on the timeline 1404 . These events 1405 can be selected, and pertinent information is displayed above the timeline 1404 .
- the search results section 1503 summarizes the collections found in the bounding box 1501 .
- the colored lines 1504 on the timeline 1505 represent the start time of each collection in the search results, where each collection is a ten minute segment of data.
- the collection information is displayed in the search results section 1503 .
- a download images button 1507 requests and downloads the imagery from the database, and subsequently displays 360 degree imagery. Multiple images 1508 , 1509 , 1510 from different parts of the timeline can be displayed simultaneously.
- the location of the vehicle for each selected image is represented by an icon on the map 1511 .
- the inventors have developed a search server and related tools to maintain an efficiently searchable index of collected data.
- the inventors use a combination of spatial and temporal indexes at varying scales to answer search queries that may include time, distance or producer identity constraints.
- the server uses its query optimizer to determine the best search strategy for the constraints given.
- the presence and breadth of certain parameters inform the decision as to which type of index the server should employ.
- Time/ID search through the global linear indices The server uses one set of global indices to answer queries such as “Where was this vehicle (all vehicles) between March 2 and April 5?” This set is made up of B+-trees that map time and vehicle ID to a matching collection unit. Queries that include a spatial constraint as well as vehicle ID are also typically processed using this strategy, as the rate limit on data generated by a particular vehicle means that searching through the spatial index would be slower, unless the range of time were very large. When this strategy is used for such spatial queries, the result list is filtered by distance before being delivered to the client.
- Data are associated with cells at certain levels of the hierarchy, e.g., those at levels where cell area is 1.5, 6, 24, or 100 km 2 (each cell makes up four cells in the next lower level).
- the server performs a recursive search within this hierarchy, identifying the largest cells that are entirely within the search radius. Cells that are only partially covered by the search area are subdivided until either a completely covered cell is found or the process reaches the lowest level of grid cells.
- the server takes its list of matching cells and processes each one (possibly in parallel). Any cell that has been populated will appear in the root-level tree that maps cell IDs to a set of local indices. When the particular cell is completely enclosed within the search area, the server looks in the cell's local temporal index (a B+-tree mapping (time, vehicle ID) to collection unit) for collection units that match the time constraint (which may be (0, ⁇ )). Otherwise, the server will look in either of two indices that The inventors plan to implement in the future: a spatial tree for queries that match only part of the cell but do not specify a time range, or a local spatiotemporal index for queries that specify both a part of the cell and a time range. At present, there is no cell-local spatial index, and all of the cell's entries that match the time constraint are tested for distance to the search center.
- a spatial tree for queries that match only part of the cell but do not specify a time range
- a local spatiotemporal index for queries that specify
- the server returns a result set in which it lists each collection or 10-minute unit it found (depending on the result format the client requested). When asked for collections, it lists each collection's important data, including vehicle ID and start time, and when the client requests 10-minute units, the server lists each unit's start time as well as its sequential number within the collection and all of the data that would be returned for the corresponding collection. Start time and vehicle ID make up the unique key with which the client may identify a collection and fetch its raw data. If the client requires, the server will also include a summary of the unit or collection, consisting of a list of time-stamped points at a specified resolution.
- DBS is a network-accessible storage layer that will use the resources of many machines to provide transactional read-write access for fixed-size blocks of binary data.
- the service is designed as a platform that may be used to build large, shared data structures of any type. It is especially appropriate for hierarchical structures such as B-trees that depend on quick access to relatively large blocks (several KB) under read-heavy workloads.
- the inventors have focused on developing the client's interface to this system as well as implementing basic storage features in order to enable map viewer client usage, and The inventors plan to extend DBS with the goals of scalability and support for ACID guarantees.
- the present invention is embodied and run as an ANDROID application from using post-processed data (written to the phone's sdcard after the mission is completed) or by a real-time, live interface.
- the ANDROID application streams the images as they come in, automatically refreshing the view on the screen 1600 as shown in FIG. 16 .
- the user can choose to continue watching the live stream, panning, and zooming on the imagery as desired, or can pause the live streaming and review older images as shown in FIG. 19 .
- the images are JPEG compressed and sent as UDP messages.
- There is a set of configurable parameters e.g., image size, scale, quality, encoding, and frame rate
- One of the parameters, tile size allows that image to be sent as a series of “tiles” 1701 , 1702 , 1703 , and 1704 that can be reconstructed on the phone to form a full image.
- the main advantage to using a tiled image is that if a packet or tile is dropped, like tile 1701 in FIG. 17 , the rest of the image or tiles 1702 , 1703 , and 1704 are still viewable as shown in FIG. 17 This way, even though the information is incomplete, the soldier can still have access to some of the imagery, giving them enhanced awareness of the environment. The soldier can later request that any dropped packets be re-transmitted if and when communications quality has improved.
- the bandwidth used to transfer the images will depend on how these parameters are set. Tiling images uses higher bandwidth, but has the added advantage of being more robust and resilient to dropped packets. Using a higher quality image and/or increasing the frame rate would also increase the bandwidth.
- the images shown in the FIGS. 16-18 were sent as 1600 ⁇ 800 pixel images, split into four 400 ⁇ 400 pixel tiles at 50% quality. These images were ⁇ 40-55 KB each. As each image is sent to the phone, it is saved locally on the phone's sdcard.
- a slide bar 1600 at the bottom of the screen 1601 lets the user know where the image he or she is currently looking at is located with respect to the last image received as shown in FIG. 18 . For example, if the user was looking at the first image out of 10, the slide bar 1600 would be all the way to the left position 1602 . If the user was looking at the fifth image out of ten, the slide bar 1600 would be in a centered position 1603 . Clicking or dragging the slide bar 1600 pauses the live stream. The user can then review the imagery as desired, using either the forwards and backwards buttons 1604 and 1605 or the slide bar 1600 . To return to the live stream, one can press the button on the right side of the screen 1606 or drag the slide bar 1600 all the way to the right position 1607 .
- An additional functionality to the live-streaming android application allows a user to track a static object of interest (OOI) through multiple frames. This can be useful, for example, if the user identifies a suspicious package near the robot. Using the tracking feature, the user can go forwards and backwards in the video stream and the camera image will stay centered (by panning and tilting the virtual camera) around the desired object. This allows the user to automatically view the OOI from multiple angles, increasing his or her situational awareness while also reducing the workload of the operator. Since the OOI is always centered on the screen, the soldier does not need to spend time and focus to search for the OOI is subsequent imagery.
- OOI static object of interest
- This functionality is activated by clicking on the OOI in two different frames 1900 and 2000 as illustrated by FIGS. 19 and 20 .
- the object can be triangulated to its global location.
- the virtual camera is then adjusted to this new focus point 2100 and 2200 as illustrated by FIGS. 21 and 22 .
- Still another feature of the present invention is an image comparison tool that allows the user compare images from two separate missions.
- the path of the robots from both missions are displayed on the map in two different colors. Clicking the map on an overlapping location (i.e., an area that was seen in both missions) or navigating to that location using the buttons on the app brings up the two images 2301 and 2302 side by side.
- the navigation information from the time the images were taken is used to align the orientation of the images 2301 and 2302 so that the images 2301 and 2302 shown on the screen are showing the same area of the environment as shown in FIG. 23 .
- the images 2301 and 2302 are locked so that moving a finger on the screen changes the pan/tilt/zoom of both images.
Abstract
Description
- This application claims priority from and is a continuation of U.S. patent application Ser. No. 13/604,842, entitled “Method and System for Visualization Enhancement for Situational Awareness”, filed on Sep. 6, 2012. The benefit under 35 USC §119e of the United States provisional application is hereby claimed, and the aforementioned application is hereby incorporated herein by reference.
- Not Applicable
- Not Applicable
- This invention relates to robotic mapping systems. In particular, the invention relates to a robotic mapping system providing situational awareness based on previously recorded mapping data.
- Unmanned Ground Vehicles (UGVs) are useful for taking people out of harm's way. Although there have been many advances in using these vehicles, one of the key areas that is currently lacking is the poor situational awareness provided to operators.
- Military operations in urban areas often involve action in unknown environments such as buildings, tunnels, streets, and alleys. Having access to advanced intelligence in the area of operation increases the preparedness the soldiers when the operation begins. However, conducting “sweeps” of unknown areas can results in casualties and fatalities for warfighters. Additionally, a soldiers' ability to successfully complete a mission is frequently impaired by a lack of information about a structure and/or the locations of potentially hostile individuals within that structure. A robotic mapping system that provides a realistic, high-resolution three-dimensional graphics database that clearly depicts and geo-locates the interior and exterior of a building, prior to soldiers entering the structure, provides a significant benefit to ground forces.
- When a robot maps the building, tools are needed in order translate the raw data to an easy-to understand visualization display for the operator. This output must represent the environment, within which a small UGV traveled, in an intuitive and easily-understood way. A widely-used visualization tool consists of viewing a video stream from a platform's camera. However, with a video stream, the user cannot easily change perspective, zoom, follow the image frames in an order other than that with which they were captured, or gain a sense of depth for objects within the image. Moreover, video imposes high bandwidth requirements. A strong visualization tool creates a very realistic representation of the environment that the user can navigate in any way desired.
- The present invention is an after-action and real time mission review tool that displays camera, ladar and navigation sensor data. The present invention allows the user to pan, tilt, and zoom through the images from the front and back cameras on the robot, while simultaneously viewing time/date information, along with any available navigation information (e.g., the latitude and longitude of the robot at that time instance). Also displayed is a visual representation of the path the robot traversed; when the user clicks on the path, the image is automatically changed to the image corresponding to that position. If aerial images of the area are available, the path can be plotted on the geo-referenced image. The tool geo-registers images and maps so that they can be retrieved based on location and time. The system allows the user to compare information from multiple data collections by “locking” the images as the operator pan, tilt and zooms.
- The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present invention and, together with the description, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention.
-
FIG. 1 illustrates the VESA concept: visualization tools; -
FIGS. 2 a and 2 b show the three separate images from the cameras as well as the 360 degree omni-directional image; -
FIG. 3 is a screen shot of the display of the computer interfaced for executing the method of the present invention; -
FIG. 4 illustrates the navigational map component of the interface display used by the present invention; -
FIG. 5 illustrates the status display component of the interface display used by the present invention; -
FIG. 6 illustrates the camera image and timeline scroll bar component of the interface display used by the present invention; -
FIG. 7 illustrates a touch screen embodiment of the interface display used by the present invention; -
FIG. 8 is a screen shot of the display of an alternative embodiment of the computer interfaced for executing the method of the present invention; -
FIG. 9 is a screen shot illustrating the zoomable map feature of the display of the present invention; -
FIG. 10 is a screen shot of the event file interface of the present invention; -
FIG. 11 is a screen shot of the event adding feature of the present invention; -
FIGS. 12 and 13 are screen shots illustrating the display of an additional 3D image of the display of the present invention; -
FIG. 14 is a screen shot of the map viewer of the present invention; -
FIG. 15 is a screen shot of the results of a query illustrating navigation paths within the bounding box of the map viewer, along with a series of images from three different points on the timeline; -
FIG. 16 illustrates a real-time application interface; -
FIG. 17 illustrates the situation where a first packet of an image was dropped, but the rest of the image is viewable; -
FIG. 18 illustrates the situation where a live view is paused when the user drags the slide bar; -
FIG. 19 illustrates where the user clicks on the barrel as the first triangulation point; -
FIG. 20 illustrates where the user clicks on the barrel again in the subsequent frame as the second triangulation point; -
FIGS. 20-22 illustrate where, in the next frame, the barrel is now automatically centered on the screen; and -
FIG. 23 illustrates an image comparison between the two missions where a person has left the scene and one of the barrels has been moved and another robot has entered the scene. - VESA is defined as Visualization Enhancement for Situational Awareness.
- UGV is defined as is an unmanned ground vehicle that operates while in contact with the ground and without an onboard human presence. UGVs can be used for many applications where it may be inconvenient, dangerous, or impossible to have a human operator present. Generally, the vehicle will have a set of sensors to observe the environment, and will either autonomously make decisions about its behavior or pass the information to a human operator at a different location who will control the vehicle through teleoperation.
- LADAR (also known as LIDAR) is an optical remote sensing technology that can measure the distance to, or other properties of a target by illuminating the target with light, often using pulses from a laser. LIDAR technology has application in geomatics, archaeology, geography, geology, geomorphology, seismology, forestry, remote sensing and atmospheric physics, as well as in airborne laser swath mapping (ALSM), laser altimetry and LIDAR contour mapping. The acronym LADAR (Laser Detection and Ranging) is often used in military contexts. The term “laser radar” is sometimes used, even though LIDAR does not employ microwaves or radio waves and therefore is not radar in the strict sense of the word.
- DAQ (also referred to as “buf” or “blackbox”) are files that record the navigation information of the UGV. These files are time and date stamped, making it simple to determine which files correspond to a mission.
- DBS is a network-accessible storage layer that will use the resources of many machines to provide transactional read-write access for fixed-size blocks of binary data.
- In computing, a graphical user interface (GUI, commonly pronounced gooey) is a type of user interface that allows users to interact with electronic devices using images rather than text commands. GUIs can be used in computers, hand-held devices such as MP3 players, portable media players or gaming devices, household appliances and office equipment. A GUI represents the information and actions available to a user through graphical icons and visual indicators such as secondary notation, as opposed to text-based interfaces, typed command labels or text navigation. The actions are usually performed through direct manipulation of the graphical elements.
- In the following detailed description of the invention of exemplary embodiments of the invention, reference is made to the accompanying drawings (where like numbers represent like elements), which form a part hereof, and in which is shown by way of illustration specific exemplary embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, but other embodiments may be utilized and logical, mechanical, electrical, and other changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.
- In the following description, numerous specific details are set forth to provide a thorough understanding of the invention. However, it is understood that the invention may be practiced without these specific details. In other instances, well-known structures and techniques known to one of ordinary skill in the art have not been shown in detail in order not to obscure the invention. Referring to the figures, it is possible to see the various major elements constituting the apparatus of the present invention.
- During an initial phase of development, the inventors began work on a Visualization Enhancement for Situational Awareness toolkit referred to as VESA. The goal of the VESA toolkit project was to combine autonomy with accurate mapping and visualization capabilities for use on vehicles or robotic systems. The inventors worked on creating a visualization widget to display imagery, as well as mapping results, from a small robot which has traveled through an unknown environment. VESA is designed to work with a variety of sensors; while cameras form the base of the visualization tool, if a LADAR is present on the robot, then that information will be able to be displayed as well. The soldiers are then able to analyze the information prior to entering the structure. Because of VESA's capabilities, each soldier knows the basic structural layout, therefore greatly enhancing the team's situational awareness.
- The inventors have developed hardware and software for a sensor system that is specifically designed and implemented on a variety of platforms. These platforms were used to collect data during a first phase of VESA.
- The machine components of the present invention consist of the VESA concept: visualization tools as shown in
FIG. 1 . The present camera system consists of threemachine vision cameras FIGS. 2 a and 2 b show the three separate images from thecameras directional image 204. - During a first phase, the inventors developed the present invention Situational Awareness Toolkit. The main goal of this toolkit is to improve the situational awareness of a soldier by providing an intuitive and easy-to-use way to “walk through” and interact with the sensor and navigation data of the robot during and post mission.
- The method of the present invention is enabled by software executed on a computer machine. The
interface 300 for a user interacting with the software running and/or executing the method of the present invention is shown inFIG. 3 . Referring toFIGS. 3 and 4 , on the left side of thewindow 301 is thenavigation map 400 as shown individually inFIG. 4 Thenavigation map 400 displays the traveledpath 401 in a first color, if available. The current position of the vehicle is indicated by atriangle 402 in a second color, where the front of the vehicle is designated by the point of the triangle. The field of view (FOV) of the camera is also indicated on the triangle by the twoadjacent lines - If the
FOV lines FOV lines FOV lines aerial imagery 405 is available, it can be displayed as well and thenavigation map 400 superimposed or laid over top of theaerial imagery 405 using GPS to align theaerial imagery 405 with thenavigation map 400. - A
status display 500 is located in the bottom left corner of the window as shown inFIG. 3 and individually inFIG. 5 . The information displayed includes the date and time of the mission 501, along with the GMT time inseconds 502. Also displayed, if available, are the position of the robot in UTM and lat/long position 503, and vehicle speed (in miles per hour and meters per second) 504. Adirectional compass 505 located towards the right of the display info shows the vehicle's current cardinal direction of travel. Listed below the compass is the cardinal direction of the vehicle indegrees 506. - A de-warped image taken from the camera is located on the right side of the
window 600 as shown inFIG. 3 and individually inFIG. 6 . The image displayed 601 is the image taken from the camera at a given point in time. Overlaid on the top of theimage display 600 is the cardinal direction of the displayedcamera image 602. Thedirection 602 changes with camera panning within the still frame image. Below thecamera image 601 is ahorizontal scroll bar 603 used to progress through the timeline of the mission. Eachslider increment 604 is equivalent to moving one second in time. Thescroll bar 603 also allows a user to jump to any point from the beginning to the end of the trial. The left and right keyboard keys allow for quickly incrementing or decrementing of theslider 604. The ‘Front’ and ‘Back’buttons scroll bar 603 allow the user to toggle between the front camera and the rear camera of the vehicle. - Beneath the
scroll bar 603 is a set of playback controls 607 that allow the user to play 608,pause 609, rewind 610, andfast forward 611 through the images. Repeatedly pressing therewind 610 or fast-forward 611 buttons increases the speed. - To facilitate ease-of-use with touch screen devices, buttons can be displayed that replicate some of the features usually accessed with mouse commands. The touch
screen button interface 700 is shown inFIG. 7 . The “+”and “−”buttons button 703 will re-center the navigation map. These buttons can be toggled on and off. - There are two main tasks involved in running the present invention: getting data from the vehicle and opening/viewing the data. The data required to run the present invention consists of image files (.jpg) from the vehicle's cameras and daq (referred to as “buf” or “blackbox”) files that record the navigation information. These files are time and date stamped, making it simple to determine which files correspond to the current mission. When getting data from the robot, the goal is to put the all the image and daq files from a single mission into a folder on the computer on which a user will be running the software executing the method of the present invention. Each mission should have its own separate folder to ensure that the data loads correctly. There are two ways to get data from the vehicle—getting the files locally or remotely over the network.
- There are two ways to explore the images of the path mapped by the vehicle: in space by clicking on any point of the
navigation map 400 or in time by using thehorizontal scroll bar 603 or playback controls 607 below thecamera image 601. In each case, the point selected on either thenavigation map 400 or thescroll bar 603 will display the correspondingcamera image 601 at the vehicle's respective time and position. Thescroll bar 603 can be incremented or decremented by using the left and right arrow keys on the keyboard of the computer running the software executing the method of the present invention. - Alternatively, the play/rewind/fast-forward controls will automatically cycle through the images. By clicking and dragging the camera image, the user can rotate the camera to further inspect the image. The scroll button on the mouse can be used to zoom in or out of the image when the cursor is on the image. By clicking the left and right mouse buttons simultaneously and dragging on the image, a user can also zoom in or out of the image.
- In Touch Screen Mode, a user can use the “+” and “−”
buttons Button 703 will reset thenavigation map 400. - Now referring to
FIG. 8 , an alternative user interface is shown with alternative, additional, and/or improved features are illustrated. Theinterface 800 of the alternative embodiment is for a user interacting with the software running and/or executing the method of the present invention is shown inFIG. 8 . Referring toFIG. 8 , on the left side of thewindow 800 is thenavigation map 801. Thenavigation map 801 displays the traveledpath 802 of the vehicle as created by two-dimensional LADAR in a first color, if available. The LADAR maps 815 are now semi-transparent when compared to the embodiment ofFIG. 3 , allowing the user to view the aerial imagery underneath the map. The current position of the vehicle is indicated by atriangle 803 in a second color, where the front of the vehicle is designated by the point of the triangle. The field of view (FOV) of the camera is also indicated on the triangle by the twoadjacent lines - If the
FOV lines FOV lines FOV lines aerial imagery 806 is available, it can be overlaid beneath the path of the robot for better situational awareness. Ifaerial imagery 806 is available, it can be displayed as well and thenavigation map 801 superimposed or laid over top of theaerial imagery 806 using GPS to align theaerial imagery 806 with thenavigation map 801. - A
status display 807 is located in the bottom left corner of the window as shown inFIG. 8 The information displayed includes the date and time of the mission 808, along with the GMT time inseconds 809. Also displayed, if available, are the position of the robot in UTM and lat/long position 810, and vehicle speed (in miles per hour and meters per second) 811. A directional compass 812 located towards the right of the display info shows the vehicle's current cardinal direction of travel. Listed below the compass is the cardinal direction of the vehicle in degrees 813. - A de-warped image taken from the
camera 814 is located on the right side of thewindow 800 as shown inFIG. 8 . The image displayed 814 is the image taken from the camera at a given point in time. Overlaid on the top of theimage display 814 is the cardinal direction of the displayedcamera image 815. Thedirection 815 changes with camera panning within the still frame image. Below thecamera image 814 is ahorizontal scroll bar 816 used to progress through the timeline of the mission. Eachslider increment 817 is equivalent to moving one second in time. Thescroll bar 816 also allows a user to jump to any point from the beginning to the end of the trial. The left and right keyboard keys allow for quickly incrementing or decrementing of theslider 817. The ‘Front’ and ‘Back’buttons scroll bar 816 allow the user to toggle between the front camera and the rear camera of the vehicle. Beneath thescroll bar 816 is a set of playback controls 820 that allow the user to play 821,pause 822, rewind 823, andfast forward 824 through the images. Repeatedly pressing therewind 823 or fast-forward 824 buttons increases the speed. - Referring to
FIG. 9 , themap area 801 where therobot path 802,LADAR 815, andaerial imagery 806 are displayed can now be panned and zoomed to get a detailed view of the area. - There are two ways to explore the images of the path mapped by the vehicle: in space by clicking on any point of the
navigation map 800 or in time by using thehorizontal scroll bar 816 or playback controls 817 below thecamera image 814. In each case, the point selected on either thenavigation map 800 or thescroll bar 816 will display the correspondingcamera image 814 at the vehicle's respective time and position. Thescroll bar 816 can be incremented or decremented by using the left and right arrow keys on the keyboard of the computer running the software executing the method of the present invention. - Alternatively, the play/rewind/fast-forward controls will automatically cycle through the images. By clicking and dragging the camera image, the user can rotate the camera to further inspect the image. The scroll button on the mouse can be used to zoom in or out of the image when the cursor is on the image. By clicking the left and right mouse buttons simultaneously and dragging on the image, a user can also zoom in or out of the image.
- In Touch Screen Mode, a user can use the “+” and “−”
buttons Button 826 will reset thenavigation map 800. - In still another alternative embodiment shown in
FIG. 10 , an event view mode is displayed that allows the user to tag events in the video. The event file format is generic so that it can be created using the present invention, or can be written by different processes on a robot to record any notable events automatically (e.g., emergency stops, periods of time with no plans, high speed or curvature events). - Multiple event files can be loaded and are each displayed on
separate timelines 1001. Eachtimeline 1001 is zoom-able using thebuttons timeline 1001 to displaydifferent portions 1004 of thetimeline 1001. Below thetimelines 1001, an overview bar 1005 displays, as denoted by a firstcolored line 1006, what portion of theoverall timeline 1001 is being displayed and the current image location, a denoted by second colored line orbar 1007. The second colored line orbar 1007 can be dragged to quickly access other portions of thetimelines 1001. - Events are noted by a colored line or
bar 1008 on thetimeline 1001. Each event can be selected and thecomment 1009 from that event is displayed beneath thetimelines 1001. The (x,y)location 1010 of thatcomment 1009 is plotted as acolored circle 1010 on themap 1011. - The current image position is represented by a
colored line 1012 on thetimelines 1001. Thatline 1012 can be dragged to move forward or backward in the images. - Now referring to
FIG. 11 , on eachtimeline 1001, anevent 1113 can be added, edited, or deleted. There is also a save option to save the edits to the event file(s). Eachtimeline 1001 can be moved up or down for easily comparing events, and can also be closed if no longer needed. - Initially, the present invention only displayed the 360
image viewer 1013; when displaying 3D maps, a separate program was launched. The downside of this was that the two programs had separate controls, so the area being seen on the image would not correspond directly to the 3D data without a lot of manual work by the operator. - In still another embodiment, the present invention includes a built-in 3D viewer. As shown in
FIG. 12 , a3D map 1201 is displayed to the right of themain image 1202, and the viewpoints are locked to each other so that zooming and panning on the image will affect the 3D view and vice versa. As shown inFIG. 13 , theimage 1202 and3D map scenes 1201 are adjusted to match the pan/tilt/zoom commands of the operator. This map viewer is extremely flexible, allowing many different types of objects and textures to be displayed. This allows the present invention to be able to load in many different kinds of maps, including maps based on the Kinect, LADAR scanners, stereo cameras, etc., depending upon what sensors/mapping technologies are available. - The goal of the new database/map server system is to give users an easy way to view and compare VESA data from multiple vehicles and/or from multiple data collections. Desired functionality includes being able to search for all data collected in a certain area, taken during a specified time range, or a combination of both. Future work may involve adding a “tag” based search which allows the user to annotate the image data and search by keywords.
- One important part of the interface is the ability to easily view maps of the area in which the user wishes to search. The inventors do not want to rely on having an internet connection, so the inventors began work on our own map server. The map data the inventors decided to use an open source map data source that contain street level maps of the entire world at 19 different zoom levels. The map viewer UI can be seen in
FIG. 14 . Themap display 1400 can be zoomed and panned using mouse actions. Users can select the time ranges to search using thecalendar inputs 1401 below themap display 1400.Status information 1402 about can be also be seen to the right of the date/time selection 1403. Atimeline 1404 is featured at the bottom of theUI map display 1400. When a search is conducted, events 1405 will appear on thetimeline 1404. These events 1405 can be selected, and pertinent information is displayed above thetimeline 1404. - Now referring to
FIG. 15 , to query in a specific location, users can drag abounding box 1501 over amap 1502. The resultingnavigation paths 1503 will be drawn on thedisplay map 1502. Thesearch results section 1503 summarizes the collections found in thebounding box 1501. Thecolored lines 1504 on thetimeline 1505 represent the start time of each collection in the search results, where each collection is a ten minute segment of data. When anevent 1506 on thetimeline 1505 is selected, the collection information is displayed in thesearch results section 1503. Adownload images button 1507 requests and downloads the imagery from the database, and subsequently displays 360 degree imagery.Multiple images map 1511. - The inventors have developed a search server and related tools to maintain an efficiently searchable index of collected data. The inventors use a combination of spatial and temporal indexes at varying scales to answer search queries that may include time, distance or producer identity constraints.
- When a user sends such a query to a search server, the server uses its query optimizer to determine the best search strategy for the constraints given. The presence and breadth of certain parameters inform the decision as to which type of index the server should employ.
- Time/ID search through the global linear indices: The server uses one set of global indices to answer queries such as “Where was this vehicle (all vehicles) between March 2 and April 5?” This set is made up of B+-trees that map time and vehicle ID to a matching collection unit. Queries that include a spatial constraint as well as vehicle ID are also typically processed using this strategy, as the rate limit on data generated by a particular vehicle means that searching through the spatial index would be slower, unless the range of time were very large. When this strategy is used for such spatial queries, the result list is filtered by distance before being delivered to the client.
- Spatial search through indices in a hierarchy of grids: For queries such as “What did this vehicle (all vehicles) see within 10 km of point 39N, 77W last week (at any time)?” the server looks at a grid hierarchy that splits the surface of the planet into distinct cells.
- Data are associated with cells at certain levels of the hierarchy, e.g., those at levels where cell area is 1.5, 6, 24, or 100 km2 (each cell makes up four cells in the next lower level). The server performs a recursive search within this hierarchy, identifying the largest cells that are entirely within the search radius. Cells that are only partially covered by the search area are subdivided until either a completely covered cell is found or the process reaches the lowest level of grid cells.
- The server takes its list of matching cells and processes each one (possibly in parallel). Any cell that has been populated will appear in the root-level tree that maps cell IDs to a set of local indices. When the particular cell is completely enclosed within the search area, the server looks in the cell's local temporal index (a B+-tree mapping (time, vehicle ID) to collection unit) for collection units that match the time constraint (which may be (0, ∞)). Otherwise, the server will look in either of two indices that The inventors plan to implement in the future: a spatial tree for queries that match only part of the cell but do not specify a time range, or a local spatiotemporal index for queries that specify both a part of the cell and a time range. At present, there is no cell-local spatial index, and all of the cell's entries that match the time constraint are tested for distance to the search center.
- The server returns a result set in which it lists each collection or 10-minute unit it found (depending on the result format the client requested). When asked for collections, it lists each collection's important data, including vehicle ID and start time, and when the client requests 10-minute units, the server lists each unit's start time as well as its sequential number within the collection and all of the data that would be returned for the corresponding collection. Start time and vehicle ID make up the unique key with which the client may identify a collection and fetch its raw data. If the client requires, the server will also include a summary of the unit or collection, consisting of a list of time-stamped points at a specified resolution.
- To support the database, The inventors have developed an underlying storage system called DBS. DBS is a network-accessible storage layer that will use the resources of many machines to provide transactional read-write access for fixed-size blocks of binary data. The service is designed as a platform that may be used to build large, shared data structures of any type. It is especially appropriate for hierarchical structures such as B-trees that depend on quick access to relatively large blocks (several KB) under read-heavy workloads. The inventors have focused on developing the client's interface to this system as well as implementing basic storage features in order to enable map viewer client usage, and The inventors plan to extend DBS with the goals of scalability and support for ACID guarantees.
- In an alternative embodiment, the present invention is embodied and run as an ANDROID application from using post-processed data (written to the phone's sdcard after the mission is completed) or by a real-time, live interface. The ANDROID application streams the images as they come in, automatically refreshing the view on the
screen 1600 as shown inFIG. 16 . The user can choose to continue watching the live stream, panning, and zooming on the imagery as desired, or can pause the live streaming and review older images as shown inFIG. 19 . - The images are JPEG compressed and sent as UDP messages. There is a set of configurable parameters (e.g., image size, scale, quality, encoding, and frame rate) that set the desired image properties. One of the parameters, tile size, allows that image to be sent as a series of “tiles” 1701, 1702, 1703, and 1704 that can be reconstructed on the phone to form a full image. The main advantage to using a tiled image is that if a packet or tile is dropped, like
tile 1701 inFIG. 17 , the rest of the image ortiles FIG. 17 This way, even though the information is incomplete, the soldier can still have access to some of the imagery, giving them enhanced awareness of the environment. The soldier can later request that any dropped packets be re-transmitted if and when communications quality has improved. - The bandwidth used to transfer the images will depend on how these parameters are set. Tiling images uses higher bandwidth, but has the added advantage of being more robust and resilient to dropped packets. Using a higher quality image and/or increasing the frame rate would also increase the bandwidth. The images shown in the
FIGS. 16-18 were sent as 1600×800 pixel images, split into four 400×400 pixel tiles at 50% quality. These images were ˜40-55 KB each. As each image is sent to the phone, it is saved locally on the phone's sdcard. - The user interface has been enhanced to accommodate the new form-factor of the screen in this embodiment. As shown in
FIGS. 16 and 18 , aslide bar 1600 at the bottom of thescreen 1601 lets the user know where the image he or she is currently looking at is located with respect to the last image received as shown inFIG. 18 . For example, if the user was looking at the first image out of 10, theslide bar 1600 would be all the way to theleft position 1602. If the user was looking at the fifth image out of ten, theslide bar 1600 would be in acentered position 1603. Clicking or dragging theslide bar 1600 pauses the live stream. The user can then review the imagery as desired, using either the forwards andbackwards buttons slide bar 1600. To return to the live stream, one can press the button on the right side of thescreen 1606 or drag theslide bar 1600 all the way to theright position 1607. - An additional functionality to the live-streaming android application allows a user to track a static object of interest (OOI) through multiple frames. This can be useful, for example, if the user identifies a suspicious package near the robot. Using the tracking feature, the user can go forwards and backwards in the video stream and the camera image will stay centered (by panning and tilting the virtual camera) around the desired object. This allows the user to automatically view the OOI from multiple angles, increasing his or her situational awareness while also reducing the workload of the operator. Since the OOI is always centered on the screen, the soldier does not need to spend time and focus to search for the OOI is subsequent imagery.
- This functionality is activated by clicking on the OOI in two
different frames FIGS. 19 and 20 . By using the navigation information of the robot in those frames, along with the positions in the image clicked by the user, the object can be triangulated to its global location. The virtual camera is then adjusted to thisnew focus point FIGS. 21 and 22 . - Still another feature of the present invention is an image comparison tool that allows the user compare images from two separate missions. There are two different modalities: two missions can be compared after-action, or a previous mission can be compared to the live mission. The path of the robots from both missions are displayed on the map in two different colors. Clicking the map on an overlapping location (i.e., an area that was seen in both missions) or navigating to that location using the buttons on the app brings up the two
images images images FIG. 23 . Theimages - Thus, it is appreciated that the optimum dimensional relationships for the parts of the invention, to include variation in size, materials, shape, form, function, and manner of operation, assembly and use, are deemed readily apparent and obvious to one of ordinary skill in the art, and all equivalent relationships to those illustrated in the drawings and described in the above description are intended to be encompassed by the present invention.
- Furthermore, other areas of art may benefit from this method and adjustments to the design are anticipated. Thus, the scope of the invention should be determined by the appended claims and their legal equivalents, rather than by the examples given.
Claims (22)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/587,295 US20150121222A1 (en) | 2012-09-06 | 2014-12-31 | Method and System for Visualization Enhancement for Situational Awareness |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/604,842 US8954853B2 (en) | 2012-09-06 | 2012-09-06 | Method and system for visualization enhancement for situational awareness |
US14/587,295 US20150121222A1 (en) | 2012-09-06 | 2014-12-31 | Method and System for Visualization Enhancement for Situational Awareness |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/604,842 Continuation US8954853B2 (en) | 2012-09-06 | 2012-09-06 | Method and system for visualization enhancement for situational awareness |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150121222A1 true US20150121222A1 (en) | 2015-04-30 |
Family
ID=50189257
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/604,842 Active 2033-03-13 US8954853B2 (en) | 2012-09-06 | 2012-09-06 | Method and system for visualization enhancement for situational awareness |
US14/587,295 Abandoned US20150121222A1 (en) | 2012-09-06 | 2014-12-31 | Method and System for Visualization Enhancement for Situational Awareness |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/604,842 Active 2033-03-13 US8954853B2 (en) | 2012-09-06 | 2012-09-06 | Method and system for visualization enhancement for situational awareness |
Country Status (1)
Country | Link |
---|---|
US (2) | US8954853B2 (en) |
Cited By (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10217443B2 (en) | 2016-01-28 | 2019-02-26 | Samsung Electronics Co., Ltd. | Method for displaying image and electronic device thereof |
US10467353B2 (en) | 2017-02-22 | 2019-11-05 | Middle Chart, LLC | Building model with capture of as built features and experiential data |
US10620084B2 (en) | 2017-02-22 | 2020-04-14 | Middle Chart, LLC | System for hierarchical actions based upon monitored building conditions |
US10628617B1 (en) | 2017-02-22 | 2020-04-21 | Middle Chart, LLC | Method and apparatus for wireless determination of position and orientation of a smart device |
US10671767B2 (en) | 2017-02-22 | 2020-06-02 | Middle Chart, LLC | Smart construction with automated detection of adverse structure conditions and remediation |
US10733334B2 (en) | 2017-02-22 | 2020-08-04 | Middle Chart, LLC | Building vital conditions monitoring |
US10740502B2 (en) | 2017-02-22 | 2020-08-11 | Middle Chart, LLC | Method and apparatus for position based query with augmented reality headgear |
US10740503B1 (en) | 2019-01-17 | 2020-08-11 | Middle Chart, LLC | Spatial self-verifying array of nodes |
US10762251B2 (en) | 2017-02-22 | 2020-09-01 | Middle Chart, LLC | System for conducting a service call with orienteering |
US10824774B2 (en) | 2019-01-17 | 2020-11-03 | Middle Chart, LLC | Methods and apparatus for healthcare facility optimization |
US10831943B2 (en) | 2017-02-22 | 2020-11-10 | Middle Chart, LLC | Orienteering system for responding to an emergency in a structure |
US10831945B2 (en) | 2017-02-22 | 2020-11-10 | Middle Chart, LLC | Apparatus for operation of connected infrastructure |
US10872179B2 (en) | 2017-02-22 | 2020-12-22 | Middle Chart, LLC | Method and apparatus for automated site augmentation |
US10902160B2 (en) | 2017-02-22 | 2021-01-26 | Middle Chart, LLC | Cold storage environmental control and product tracking |
US10949579B2 (en) | 2017-02-22 | 2021-03-16 | Middle Chart, LLC | Method and apparatus for enhanced position and orientation determination |
US10984146B2 (en) | 2017-02-22 | 2021-04-20 | Middle Chart, LLC | Tracking safety conditions of an area |
US11054335B2 (en) | 2017-02-22 | 2021-07-06 | Middle Chart, LLC | Method and apparatus for augmented virtual models and orienteering |
US20210357637A1 (en) * | 2018-08-16 | 2021-11-18 | Beijing 7Invensun Technology Co., Ltd. | Method and Apparatus for Controlling Image Acquisition Device |
US11194938B2 (en) | 2020-01-28 | 2021-12-07 | Middle Chart, LLC | Methods and apparatus for persistent location based digital content |
US11436389B2 (en) | 2017-02-22 | 2022-09-06 | Middle Chart, LLC | Artificial intelligence based exchange of geospatial related digital content |
US11468209B2 (en) | 2017-02-22 | 2022-10-11 | Middle Chart, LLC | Method and apparatus for display of digital content associated with a location in a wireless communications area |
US11475177B2 (en) | 2017-02-22 | 2022-10-18 | Middle Chart, LLC | Method and apparatus for improved position and orientation based information display |
US11481527B2 (en) | 2017-02-22 | 2022-10-25 | Middle Chart, LLC | Apparatus for displaying information about an item of equipment in a direction of interest |
US11507714B2 (en) | 2020-01-28 | 2022-11-22 | Middle Chart, LLC | Methods and apparatus for secure persistent location based digital content |
US11625510B2 (en) | 2017-02-22 | 2023-04-11 | Middle Chart, LLC | Method and apparatus for presentation of digital content |
US11640486B2 (en) | 2021-03-01 | 2023-05-02 | Middle Chart, LLC | Architectural drawing based exchange of geospatial related digital content |
US11900023B2 (en) | 2017-02-22 | 2024-02-13 | Middle Chart, LLC | Agent supportable device for pointing towards an item of interest |
US11900021B2 (en) | 2017-02-22 | 2024-02-13 | Middle Chart, LLC | Provision of digital content via a wearable eye covering |
Families Citing this family (44)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10034144B2 (en) * | 2013-02-22 | 2018-07-24 | International Business Machines Corporation | Application and situation-aware community sensing |
US9734161B2 (en) * | 2013-03-15 | 2017-08-15 | The Florida International University Board Of Trustees | Streaming representation of moving objects and shapes in a geographic information service |
US9749541B2 (en) * | 2013-04-16 | 2017-08-29 | Tout Inc. | Method and apparatus for displaying and recording images using multiple image capturing devices integrated into a single mobile device |
US20150128045A1 (en) * | 2013-11-05 | 2015-05-07 | Honeywell International Inc. | E-map based intuitive video searching system and method for surveillance systems |
JP2015128206A (en) * | 2013-12-27 | 2015-07-09 | ソニー株式会社 | Editing apparatus, reproduction apparatus, editing method, reproducing method and program |
US10230925B2 (en) | 2014-06-13 | 2019-03-12 | Urthecast Corp. | Systems and methods for processing and providing terrestrial and/or space-based earth observation video |
WO2016065519A1 (en) * | 2014-10-27 | 2016-05-06 | SZ DJI Technology Co., Ltd. | Uav flight display |
CN113628500A (en) | 2014-09-30 | 2021-11-09 | 深圳市大疆创新科技有限公司 | System and method for supporting analog mobility |
US9773422B2 (en) | 2014-12-23 | 2017-09-26 | Intel Corporation | Community drone monitoring and information exchange |
JP6506032B2 (en) * | 2015-01-28 | 2019-04-24 | 株式会社トプコン | Survey data processing apparatus, survey data processing method and program |
US10871561B2 (en) | 2015-03-25 | 2020-12-22 | Urthecast Corp. | Apparatus and methods for synthetic aperture radar with digital beamforming |
US10083532B2 (en) * | 2015-04-13 | 2018-09-25 | International Business Machines Corporation | Sychronized display of street view map and video stream |
CN108432049B (en) | 2015-06-16 | 2020-12-29 | 阿卜杜拉阿齐兹国王科技城 | Efficient planar phased array antenna assembly |
US9910761B1 (en) | 2015-06-28 | 2018-03-06 | X Development Llc | Visually debugging robotic processes |
DE102015118540B4 (en) * | 2015-10-29 | 2021-12-02 | Geomar Helmholtz-Zentrum Für Ozeanforschung Kiel - Stiftung Des Öffentlichen Rechts | Diving robot image / video data visualization system |
WO2017091747A1 (en) | 2015-11-25 | 2017-06-01 | Urthecast Corp. | Synthetic aperture radar imaging apparatus and methods |
WO2017120617A1 (en) * | 2016-01-06 | 2017-07-13 | Russell David Wayne | System and method for single lens 3d imagers for situational awareness in autonomous platforms |
CN105677159B (en) * | 2016-01-14 | 2019-01-18 | 深圳市至壹科技开发有限公司 | Image display method and video display devices |
US20180060333A1 (en) * | 2016-08-23 | 2018-03-01 | Google Inc. | System and method for placement of virtual characters in an augmented/virtual reality environment |
CA3064735C (en) | 2017-05-23 | 2022-06-21 | Urthecast Corp. | Synthetic aperture radar imaging apparatus and methods |
EP3646054A4 (en) | 2017-05-23 | 2020-10-28 | King Abdulaziz City for Science and Technology | Synthetic aperture radar imaging apparatus and methods for moving targets |
US11222076B2 (en) * | 2017-05-31 | 2022-01-11 | Microsoft Technology Licensing, Llc | Data set state visualization comparison lock |
WO2019226194A2 (en) | 2017-11-22 | 2019-11-28 | Urthecast Corp. | Synthetic aperture radar apparatus and methods |
US10554953B2 (en) * | 2017-12-17 | 2020-02-04 | Google Llc | Distortion of video for seek in 360 degree video |
US10896218B2 (en) * | 2017-12-22 | 2021-01-19 | Oracle International Corporation | Computerized geo-referencing for images |
JP7098752B2 (en) * | 2018-05-07 | 2022-07-11 | アップル インコーポレイテッド | User interface for viewing live video feeds and recorded videos |
WO2019239211A2 (en) * | 2018-06-11 | 2019-12-19 | Insurance Services Office, Inc. | System and method for generating simulated scenes from open map data for machine learning |
JP7298116B2 (en) * | 2018-08-03 | 2023-06-27 | ソニーグループ株式会社 | Information processing device, information processing method, program |
US10659698B2 (en) * | 2018-09-19 | 2020-05-19 | Canon Kabushiki Kaisha | Method to configure a virtual camera path |
EP3875230A4 (en) * | 2018-11-01 | 2022-07-27 | Canon Kabushiki Kaisha | External input device, robot system, control method for robot system, control program, and recording medium |
KR20200061279A (en) * | 2018-11-23 | 2020-06-02 | 삼성전자주식회사 | Electronic apparatus and control method thereof |
EP3863277A4 (en) * | 2018-11-23 | 2022-05-04 | Samsung Electronics Co., Ltd. | Electronic device and control method thereof |
KR102569032B1 (en) | 2019-01-22 | 2023-08-23 | 삼성전자주식회사 | Electronic device and method for providing content thereof |
WO2020221186A1 (en) * | 2019-04-30 | 2020-11-05 | 广州虎牙信息科技有限公司 | Virtual image control method, apparatus, electronic device and storage medium |
US10904029B2 (en) | 2019-05-31 | 2021-01-26 | Apple Inc. | User interfaces for managing controllable external devices |
US11363071B2 (en) | 2019-05-31 | 2022-06-14 | Apple Inc. | User interfaces for managing a local network |
CN110648551B (en) * | 2019-09-20 | 2021-11-23 | 河海大学 | Underground garage parking space positioning and navigation method based on mixed beacon stream |
US11079913B1 (en) | 2020-05-11 | 2021-08-03 | Apple Inc. | User interface for status indicators |
US11589010B2 (en) | 2020-06-03 | 2023-02-21 | Apple Inc. | Camera and visitor user interfaces |
US11657614B2 (en) | 2020-06-03 | 2023-05-23 | Apple Inc. | Camera and visitor user interfaces |
WO2022051112A1 (en) | 2020-09-05 | 2022-03-10 | Apple Inc. | User interfaces for managing audio for media items |
US20220168909A1 (en) * | 2020-11-30 | 2022-06-02 | X Development Llc | Fusing a Static Large Field of View and High Fidelity Moveable Sensors for a Robot Platform |
CN112817023B (en) * | 2021-01-06 | 2024-03-26 | 西安空间无线电技术研究所 | Non-supported low-rail navigation enhancement system and method based on star-based enhancement service |
USD1013705S1 (en) * | 2021-07-20 | 2024-02-06 | Splunk Inc. | Display screen or portion thereof having a graphical user interface with a time slider for a map |
Citations (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5631838A (en) * | 1993-08-02 | 1997-05-20 | Aisin Seiki Kabushiki Kaisha | Apparatus for verifying GPS satellite data |
US5745866A (en) * | 1996-01-09 | 1998-04-28 | Trimble Navigation Limited | Cardinal-up graphic map display system |
US6591300B1 (en) * | 1999-06-30 | 2003-07-08 | Lucent Technologies Inc. | Integrated management application |
US6741790B1 (en) * | 1997-05-29 | 2004-05-25 | Red Hen Systems, Inc. | GPS video mapping system |
US20040164963A1 (en) * | 2003-01-16 | 2004-08-26 | Takatoshi Ono | Trackball device and vehicle incorporating the same |
US20040181747A1 (en) * | 2001-11-19 | 2004-09-16 | Hull Jonathan J. | Multimedia print driver dialog interfaces |
US20060142971A1 (en) * | 2004-12-08 | 2006-06-29 | David Reich | All surface data for use in substrate inspection |
US20060171453A1 (en) * | 2005-01-04 | 2006-08-03 | Rohlfing Thomas R | Video surveillance system |
US20070027583A1 (en) * | 2003-07-07 | 2007-02-01 | Sensomatix Ltd. | Traffic information system |
US20070061076A1 (en) * | 2005-01-06 | 2007-03-15 | Alan Shulman | Navigation and inspection system |
US20080189031A1 (en) * | 2007-02-06 | 2008-08-07 | Meadow William D | Methods and apparatus for presenting a continuum of image data |
US20080303901A1 (en) * | 2007-06-08 | 2008-12-11 | Variyath Girish S | Tracking an object |
US20090009533A1 (en) * | 2005-01-21 | 2009-01-08 | Matsushita Electric Industrial Co., Ltd | Display controller |
US20090015674A1 (en) * | 2006-04-28 | 2009-01-15 | Kevin Alley | Optical imaging system for unmanned aerial vehicle |
US20090074265A1 (en) * | 2007-09-17 | 2009-03-19 | Capsovision Inc. | Imaging review and navigation workstation system |
US7526718B2 (en) * | 2003-04-30 | 2009-04-28 | Hewlett-Packard Development Company, L.P. | Apparatus and method for recording “path-enhanced” multimedia |
US20090134968A1 (en) * | 2007-11-28 | 2009-05-28 | Fuji Xerox Co., Ltd. | Segmenting time based on the geographic distribution of activity in sensor data |
US20090240818A1 (en) * | 2008-03-18 | 2009-09-24 | Nortel Networks Limited | Method and Apparatus for Reconstructing a Communication Session |
US20090248610A1 (en) * | 2008-03-28 | 2009-10-01 | Borkur Sigurbjornsson | Extending media annotations using collective knowledge |
US20100115456A1 (en) * | 2008-11-03 | 2010-05-06 | Thomas Wm Lucas | Virtual cubic display template for search engine |
US20100146436A1 (en) * | 2008-02-01 | 2010-06-10 | Gabriel Jakobson | Displaying content associated with electronic mapping systems |
US20100205203A1 (en) * | 2009-02-09 | 2010-08-12 | Vitamin D, Inc. | Systems and methods for video analysis |
US20100305778A1 (en) * | 2009-05-27 | 2010-12-02 | Honeywell International Inc. | Adaptive user interface for semi-automatic operation |
US20110010674A1 (en) * | 2009-07-13 | 2011-01-13 | Raytheon Company | Displaying situational information based on geospatial data |
US7912596B2 (en) * | 2007-05-30 | 2011-03-22 | Honeywell International Inc. | Vehicle trajectory visualization system |
US20110288695A1 (en) * | 2010-05-18 | 2011-11-24 | Clearpath Robotics, Inc. | Control interface for unmanned vehicles |
US20130163956A1 (en) * | 2011-12-21 | 2013-06-27 | Pelco, Inc. | Method and System for Displaying a Timeline |
US8571302B2 (en) * | 2008-08-12 | 2013-10-29 | Samsung Electronics Co., Ltd | Method and apparatus to build 3-dimensional grid map and method and apparatus to control automatic traveling apparatus using the same |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3566451B2 (en) * | 1996-04-24 | 2004-09-15 | キヤノン株式会社 | Camera control system and camera control system control method |
IL159537A0 (en) * | 2001-06-28 | 2004-06-01 | Omnivee Inc | Method and apparatus for control and processing of video images |
US6833843B2 (en) * | 2001-12-03 | 2004-12-21 | Tempest Microsystems | Panoramic imaging and display system with canonical magnifier |
JP4649091B2 (en) * | 2002-01-30 | 2011-03-09 | 株式会社エヌ・ティ・ティ・ドコモ | Communication terminal, server device, relay device, broadcast communication system, broadcast communication method, and program |
AU2003241385A1 (en) * | 2002-05-03 | 2003-11-17 | Pixearth, Corporation | A system to navigate within images spatially referenced to a computed space |
US6934002B1 (en) * | 2002-11-01 | 2005-08-23 | Mark Setteducati | System for interactive display of a magic show |
US8395664B2 (en) * | 2006-09-13 | 2013-03-12 | Smartvue Corp. | Wireless surveillance system and method for 3-D visualization and user-controlled analytics of captured data |
US7796154B2 (en) * | 2005-03-07 | 2010-09-14 | International Business Machines Corporation | Automatic multiscale image acquisition from a steerable camera |
US7777648B2 (en) * | 2005-04-21 | 2010-08-17 | Microsoft Corporation | Mode information displayed in a mapping application |
US20060282776A1 (en) * | 2005-06-10 | 2006-12-14 | Farmer Larry C | Multimedia and performance analysis tool |
US20080028323A1 (en) * | 2006-07-27 | 2008-01-31 | Joshua Rosen | Method for Initiating and Launching Collaboration Sessions |
US20090271719A1 (en) * | 2007-04-27 | 2009-10-29 | Lpa Systems, Inc. | System and method for analysis and display of geo-referenced imagery |
US8290303B2 (en) * | 2007-10-11 | 2012-10-16 | General Electric Company | Enhanced system and method for volume based registration |
US8386942B2 (en) * | 2008-04-14 | 2013-02-26 | Disney Enterprises, Inc. | System and method for providing digital multimedia presentations |
JP2010092455A (en) * | 2008-09-10 | 2010-04-22 | Ricoh Co Ltd | System for managing file, and program |
US8732592B2 (en) * | 2009-06-08 | 2014-05-20 | Battelle Energy Alliance, Llc | Methods and systems relating to an augmented virtuality environment |
US9558013B2 (en) * | 2011-04-12 | 2017-01-31 | Citrix Systems, Inc. | Responsive scroller controls in server-hosted applications |
US10191641B2 (en) * | 2011-12-29 | 2019-01-29 | Apple Inc. | Device, method, and graphical user interface for navigation of information in a map-based interface |
-
2012
- 2012-09-06 US US13/604,842 patent/US8954853B2/en active Active
-
2014
- 2014-12-31 US US14/587,295 patent/US20150121222A1/en not_active Abandoned
Patent Citations (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5631838A (en) * | 1993-08-02 | 1997-05-20 | Aisin Seiki Kabushiki Kaisha | Apparatus for verifying GPS satellite data |
US5745866A (en) * | 1996-01-09 | 1998-04-28 | Trimble Navigation Limited | Cardinal-up graphic map display system |
US6741790B1 (en) * | 1997-05-29 | 2004-05-25 | Red Hen Systems, Inc. | GPS video mapping system |
US6591300B1 (en) * | 1999-06-30 | 2003-07-08 | Lucent Technologies Inc. | Integrated management application |
US20040181747A1 (en) * | 2001-11-19 | 2004-09-16 | Hull Jonathan J. | Multimedia print driver dialog interfaces |
US20040164963A1 (en) * | 2003-01-16 | 2004-08-26 | Takatoshi Ono | Trackball device and vehicle incorporating the same |
US7526718B2 (en) * | 2003-04-30 | 2009-04-28 | Hewlett-Packard Development Company, L.P. | Apparatus and method for recording “path-enhanced” multimedia |
US20070027583A1 (en) * | 2003-07-07 | 2007-02-01 | Sensomatix Ltd. | Traffic information system |
US20060142971A1 (en) * | 2004-12-08 | 2006-06-29 | David Reich | All surface data for use in substrate inspection |
US20060171453A1 (en) * | 2005-01-04 | 2006-08-03 | Rohlfing Thomas R | Video surveillance system |
US20070061076A1 (en) * | 2005-01-06 | 2007-03-15 | Alan Shulman | Navigation and inspection system |
US20090009533A1 (en) * | 2005-01-21 | 2009-01-08 | Matsushita Electric Industrial Co., Ltd | Display controller |
US20090015674A1 (en) * | 2006-04-28 | 2009-01-15 | Kevin Alley | Optical imaging system for unmanned aerial vehicle |
US20080189031A1 (en) * | 2007-02-06 | 2008-08-07 | Meadow William D | Methods and apparatus for presenting a continuum of image data |
US7912596B2 (en) * | 2007-05-30 | 2011-03-22 | Honeywell International Inc. | Vehicle trajectory visualization system |
US20080303901A1 (en) * | 2007-06-08 | 2008-12-11 | Variyath Girish S | Tracking an object |
US20090074265A1 (en) * | 2007-09-17 | 2009-03-19 | Capsovision Inc. | Imaging review and navigation workstation system |
US20090134968A1 (en) * | 2007-11-28 | 2009-05-28 | Fuji Xerox Co., Ltd. | Segmenting time based on the geographic distribution of activity in sensor data |
US20100146436A1 (en) * | 2008-02-01 | 2010-06-10 | Gabriel Jakobson | Displaying content associated with electronic mapping systems |
US20090240818A1 (en) * | 2008-03-18 | 2009-09-24 | Nortel Networks Limited | Method and Apparatus for Reconstructing a Communication Session |
US20090248610A1 (en) * | 2008-03-28 | 2009-10-01 | Borkur Sigurbjornsson | Extending media annotations using collective knowledge |
US8571302B2 (en) * | 2008-08-12 | 2013-10-29 | Samsung Electronics Co., Ltd | Method and apparatus to build 3-dimensional grid map and method and apparatus to control automatic traveling apparatus using the same |
US20100115456A1 (en) * | 2008-11-03 | 2010-05-06 | Thomas Wm Lucas | Virtual cubic display template for search engine |
US20100205203A1 (en) * | 2009-02-09 | 2010-08-12 | Vitamin D, Inc. | Systems and methods for video analysis |
US20100305778A1 (en) * | 2009-05-27 | 2010-12-02 | Honeywell International Inc. | Adaptive user interface for semi-automatic operation |
US20110010674A1 (en) * | 2009-07-13 | 2011-01-13 | Raytheon Company | Displaying situational information based on geospatial data |
US20110288695A1 (en) * | 2010-05-18 | 2011-11-24 | Clearpath Robotics, Inc. | Control interface for unmanned vehicles |
US20130163956A1 (en) * | 2011-12-21 | 2013-06-27 | Pelco, Inc. | Method and System for Displaying a Timeline |
Cited By (58)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10217443B2 (en) | 2016-01-28 | 2019-02-26 | Samsung Electronics Co., Ltd. | Method for displaying image and electronic device thereof |
US10410608B2 (en) | 2016-01-28 | 2019-09-10 | Samsung Electronics Co., Ltd. | Method for displaying image and electronic device thereof |
US11087039B2 (en) | 2017-02-22 | 2021-08-10 | Middle Chart, LLC | Headset apparatus for display of location and direction based content |
US11481527B2 (en) | 2017-02-22 | 2022-10-25 | Middle Chart, LLC | Apparatus for displaying information about an item of equipment in a direction of interest |
US10628617B1 (en) | 2017-02-22 | 2020-04-21 | Middle Chart, LLC | Method and apparatus for wireless determination of position and orientation of a smart device |
US10671767B2 (en) | 2017-02-22 | 2020-06-02 | Middle Chart, LLC | Smart construction with automated detection of adverse structure conditions and remediation |
US10726167B2 (en) | 2017-02-22 | 2020-07-28 | Middle Chart, LLC | Method and apparatus for determining a direction of interest |
US10733334B2 (en) | 2017-02-22 | 2020-08-04 | Middle Chart, LLC | Building vital conditions monitoring |
US10740502B2 (en) | 2017-02-22 | 2020-08-11 | Middle Chart, LLC | Method and apparatus for position based query with augmented reality headgear |
US10760991B2 (en) | 2017-02-22 | 2020-09-01 | Middle Chart, LLC | Hierarchical actions based upon monitored building conditions |
US10762251B2 (en) | 2017-02-22 | 2020-09-01 | Middle Chart, LLC | System for conducting a service call with orienteering |
US10831943B2 (en) | 2017-02-22 | 2020-11-10 | Middle Chart, LLC | Orienteering system for responding to an emergency in a structure |
US10831945B2 (en) | 2017-02-22 | 2020-11-10 | Middle Chart, LLC | Apparatus for operation of connected infrastructure |
US10866157B2 (en) | 2017-02-22 | 2020-12-15 | Middle Chart, LLC | Monitoring a condition within a structure |
US10872179B2 (en) | 2017-02-22 | 2020-12-22 | Middle Chart, LLC | Method and apparatus for automated site augmentation |
US10902160B2 (en) | 2017-02-22 | 2021-01-26 | Middle Chart, LLC | Cold storage environmental control and product tracking |
US10949579B2 (en) | 2017-02-22 | 2021-03-16 | Middle Chart, LLC | Method and apparatus for enhanced position and orientation determination |
US10984146B2 (en) | 2017-02-22 | 2021-04-20 | Middle Chart, LLC | Tracking safety conditions of an area |
US11900022B2 (en) | 2017-02-22 | 2024-02-13 | Middle Chart, LLC | Apparatus for determining a position relative to a reference transceiver |
US10983026B2 (en) | 2017-02-22 | 2021-04-20 | Middle Chart, LLC | Methods of updating data in a virtual model of a structure |
US10984148B2 (en) | 2017-02-22 | 2021-04-20 | Middle Chart, LLC | Methods for generating a user interface based upon orientation of a smart device |
US10984147B2 (en) | 2017-02-22 | 2021-04-20 | Middle Chart, LLC | Conducting a service call in a structure |
US11010501B2 (en) | 2017-02-22 | 2021-05-18 | Middle Chart, LLC | Monitoring users and conditions in a structure |
US11900021B2 (en) | 2017-02-22 | 2024-02-13 | Middle Chart, LLC | Provision of digital content via a wearable eye covering |
US10467353B2 (en) | 2017-02-22 | 2019-11-05 | Middle Chart, LLC | Building model with capture of as built features and experiential data |
US11893317B2 (en) | 2017-02-22 | 2024-02-06 | Middle Chart, LLC | Method and apparatus for associating digital content with wireless transmission nodes in a wireless communication area |
US11054335B2 (en) | 2017-02-22 | 2021-07-06 | Middle Chart, LLC | Method and apparatus for augmented virtual models and orienteering |
US11080439B2 (en) | 2017-02-22 | 2021-08-03 | Middle Chart, LLC | Method and apparatus for interacting with a tag in a cold storage area |
US11900023B2 (en) | 2017-02-22 | 2024-02-13 | Middle Chart, LLC | Agent supportable device for pointing towards an item of interest |
US11625510B2 (en) | 2017-02-22 | 2023-04-11 | Middle Chart, LLC | Method and apparatus for presentation of digital content |
US11610032B2 (en) | 2017-02-22 | 2023-03-21 | Middle Chart, LLC | Headset apparatus for display of location and direction based content |
US11106837B2 (en) | 2017-02-22 | 2021-08-31 | Middle Chart, LLC | Method and apparatus for enhanced position and orientation based information display |
US11120172B2 (en) | 2017-02-22 | 2021-09-14 | Middle Chart, LLC | Apparatus for determining an item of equipment in a direction of interest |
US11610033B2 (en) | 2017-02-22 | 2023-03-21 | Middle Chart, LLC | Method and apparatus for augmented reality display of digital content associated with a location |
US11188686B2 (en) | 2017-02-22 | 2021-11-30 | Middle Chart, LLC | Method and apparatus for holographic display based upon position and direction |
US11514207B2 (en) | 2017-02-22 | 2022-11-29 | Middle Chart, LLC | Tracking safety conditions of an area |
US11100260B2 (en) | 2017-02-22 | 2021-08-24 | Middle Chart, LLC | Method and apparatus for interacting with a tag in a wireless communication area |
US11429761B2 (en) | 2017-02-22 | 2022-08-30 | Middle Chart, LLC | Method and apparatus for interacting with a node in a storage area |
US11436389B2 (en) | 2017-02-22 | 2022-09-06 | Middle Chart, LLC | Artificial intelligence based exchange of geospatial related digital content |
US10620084B2 (en) | 2017-02-22 | 2020-04-14 | Middle Chart, LLC | System for hierarchical actions based upon monitored building conditions |
US11468209B2 (en) | 2017-02-22 | 2022-10-11 | Middle Chart, LLC | Method and apparatus for display of digital content associated with a location in a wireless communications area |
US11475177B2 (en) | 2017-02-22 | 2022-10-18 | Middle Chart, LLC | Method and apparatus for improved position and orientation based information display |
US20210357637A1 (en) * | 2018-08-16 | 2021-11-18 | Beijing 7Invensun Technology Co., Ltd. | Method and Apparatus for Controlling Image Acquisition Device |
US11853471B2 (en) * | 2018-08-16 | 2023-12-26 | Beijing 7Invensun Technology Co., Ltd. | Method and apparatus for controlling image acquisition device |
US11593536B2 (en) | 2019-01-17 | 2023-02-28 | Middle Chart, LLC | Methods and apparatus for communicating geolocated data |
US11042672B2 (en) | 2019-01-17 | 2021-06-22 | Middle Chart, LLC | Methods and apparatus for healthcare procedure tracking |
US10740503B1 (en) | 2019-01-17 | 2020-08-11 | Middle Chart, LLC | Spatial self-verifying array of nodes |
US11361122B2 (en) | 2019-01-17 | 2022-06-14 | Middle Chart, LLC | Methods of communicating geolocated data based upon a self-verifying array of nodes |
US10824774B2 (en) | 2019-01-17 | 2020-11-03 | Middle Chart, LLC | Methods and apparatus for healthcare facility optimization |
US10943034B2 (en) | 2019-01-17 | 2021-03-09 | Middle Chart, LLC | Method of wireless determination of a position of a node |
US11436388B2 (en) | 2019-01-17 | 2022-09-06 | Middle Chart, LLC | Methods and apparatus for procedure tracking |
US11861269B2 (en) | 2019-01-17 | 2024-01-02 | Middle Chart, LLC | Methods of determining location with self-verifying array of nodes |
US11636236B2 (en) | 2019-01-17 | 2023-04-25 | Middle Chart, LLC | Methods and apparatus for procedure tracking |
US11100261B2 (en) | 2019-01-17 | 2021-08-24 | Middle Chart, LLC | Method of wireless geolocated information communication in self-verifying arrays |
US11507714B2 (en) | 2020-01-28 | 2022-11-22 | Middle Chart, LLC | Methods and apparatus for secure persistent location based digital content |
US11194938B2 (en) | 2020-01-28 | 2021-12-07 | Middle Chart, LLC | Methods and apparatus for persistent location based digital content |
US11640486B2 (en) | 2021-03-01 | 2023-05-02 | Middle Chart, LLC | Architectural drawing based exchange of geospatial related digital content |
US11809787B2 (en) | 2021-03-01 | 2023-11-07 | Middle Chart, LLC | Architectural drawing aspect based exchange of geospatial related digital content |
Also Published As
Publication number | Publication date |
---|---|
US8954853B2 (en) | 2015-02-10 |
US20140068439A1 (en) | 2014-03-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8954853B2 (en) | Method and system for visualization enhancement for situational awareness | |
US11860923B2 (en) | Providing a thumbnail image that follows a main image | |
RU2580064C2 (en) | Adjustable and progressive mobile device street view | |
US20180322197A1 (en) | Video data creation and management system | |
US11272160B2 (en) | Tracking a point of interest in a panoramic video | |
US8543917B2 (en) | Method and apparatus for presenting a first-person world view of content | |
US9699375B2 (en) | Method and apparatus for determining camera location information and/or camera pose information according to a global coordinate system | |
DE112013002803B4 (en) | Method, system and device for providing a three-dimensional transition animation for a change in a map view | |
US8331611B2 (en) | Overlay information over video | |
US20130021475A1 (en) | Systems and methods for sensor control | |
US10084994B2 (en) | Live streaming video over 3D | |
US9990750B1 (en) | Interactive geo-referenced source imagery viewing system and method | |
US11636138B1 (en) | Temporal layers for presenting personalization markers on imagery | |
EP2681651A1 (en) | Method and apparatus for providing an active search user interface element | |
MX2010012826A (en) | 3d content aggregation built into devices. | |
US20120075345A1 (en) | Method, terminal and computer-readable recording medium for performing visual search based on movement or position of terminal | |
JP5987357B2 (en) | Server, electronic device system and program | |
EP3108349A1 (en) | Portals for visual interfaces | |
US20220189075A1 (en) | Augmented Reality Display Of Commercial And Residential Features During In-Person Real Estate Showings/Open Houses and Vacation Rental Stays | |
Zhang et al. | Annotating and navigating tourist videos | |
Mulloni et al. | Enhancing handheld navigation systems with augmented reality | |
WO2023173409A1 (en) | Display method and apparatus for information, comparison method and apparatus for models, and unmanned aerial vehicle system | |
AU2018203909A1 (en) | A User Interface | |
Lynch et al. | A multimedia approach to visualize and interact with large scale mobile lidar data | |
CN117435796A (en) | Method and device for displaying interest points in map, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: ROBOTIC RESEARCH, LLC, MARYLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LACAZE, ALBERTO DANIEL;MURPHY, KARL NICHOLAS;SCHNEIDER, ANNE RACHEL;AND OTHERS;SIGNING DATES FROM 20211028 TO 20211115;REEL/FRAME:058118/0293 |
|
AS | Assignment |
Owner name: ROBOTIC RESEARCH OPCO, LLC, MARYLAND Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME AND UPDATE ASSIGNEE TITLE PREVIOUSLY RECORDED AT REEL: 058118 FRAME: 0293. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:LACAZE, ALBERTO DANIEL;MURPHY, KARL NICHOLAS;SCHNEIDER, ANNE RACHEL;AND OTHERS;SIGNING DATES FROM 20211028 TO 20211115;REEL/FRAME:060330/0356 |