Method and system for visualization enhancement for situational awareness

• US 8,954,853 B2
• Filed: 09/06/2012
• Issued: 02/10/2015
• Est. Priority Date: 09/06/2012
• Status: Active Grant

• Current Assignee: Robotic Research OPCO LLC
• Original Assignee: Robotic Research LLC
• Inventors: Lacaze, Alberto Daniel, Murphy, Karl Nicholas, Schneider, Anne Rachel, Wilhelm, Raymond Paul III
• Primary Examiner(s): Tan, Alvin
• Assistant Examiner(s): Wan, Jasmine
• Application Number:
  US13/604,842
• Publication Number:
  US 20140068439A1
• Time in Patent Office: 887 Days
• Field of Search: 715/720, 715/716, 715/719, 715/201, 715/849, 701/211, 348/211
• US Class Current: 715/720
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G11B 27/105 of operating discs

Abstract

An after-action, mission review tool that displays camera and navigation sensor data allowing a user to pan, tilt, and zoom through the images from front and back cameras on an vehicle, while simultaneously viewing time/date information, along with any available navigation information such as the latitude and longitude of the vehicle at that time instance. Also displayed is a visual representation of the path the vehicle traversed; when the user clicks on the path, the image is automatically changed to the image corresponding to that position. If aerial images of the area are available, the path can be plotted on the geo-referenced image.

Claims

1. A program storage device readable by a computer machine, tangibly embodying program instructions executable by the computer machine to perform method steps for visualization enhancement for situational awareness, said method steps comprising:

providing a graphical user interface enabling a user to interact with the software program running on the computer machine running;

the computer machine providing a database of vehicle mission recordings;

selecting one or more mission recordings;

displaying a GUI window comprising;

a navigation map associated with the mission recording wherein,

the navigation map displays the traveled path traveled of an Unmanned Ground Vehicle (UGV) in first color, if available

the current position of the vehicle is indicated by a triangle in a second color,

the front of the vehicle is designated by the point of the triangle,

the field of view (FOV) of the camera is also indicated on the triangle by the two adjacent lines, shown in a third color when;

a status display; and

taking one or more camera images from the mission recording;

overlaying on the top of the image display is the cardinal direction of the displayed camera image;

changing the direction with camera panning within the still frame image;

progressing through the timeline of a mission using a horizontal scroll bar slider wherein each slider increment is equivalent to moving one second in time;

allowing a user to jump to any point from the beginning to the end of the mission;

quickly incrementing or decrementing of the slider;

providing ‘

Front’ and

‘

Back’

buttons next to the scroll bar;

allowing the user to toggle between the front camera and the rear camera of the vehicle;

providing a set of playback controls that allow the user to play, pause, rewind, and fast forward through the images,

repeatedly pressing the rewind or fast-forward buttons increases the speed;

tagging events in the mission recoding;

loading multiple mission files;

displaying each loaded mission file on separate timelines;

each timeline is zoom-able using buttons at the bottom of the screen;

the user can drag on the timeline to display different portions of the timeline;

below the timelines, an overview bar displays, as denoted by a first colored line, what portion of the overall timeline is being displayed and the current image location, a denoted by second colored line or bar;

the second colored line or bar can be dragged to quickly access other portions of the timelines;

events are noted by a colored line or bar on the timeline;

each event can be selected and the comment from that event is displayed beneath the timelines;

the (x,y) location of that comment is plotted as a colored circle on the map;

the current image position is represented by a colored line on the timelines; and

dragging the current image position colored line to move forward or backward in the images.

2. The method of claim 1, wherein

if the FOV lines are centered towards the back of the vehicle, the camera image being displayed is taken from the rear camera of the vehicle;

if the FOV lines are spread far apart (indicating a wide FOV), the image is zoomed far out;

if the FOV lines are close together, the image is zoomed in to a smaller section of the environment.

3. The method of claim 1, wherein an aerial imagery 405 is displayed as well and the navigation map superimposed or laid over top of the aerial imagery using GPS to align the aerial imagery with the navigation map.

4. The method of claim 1, wherein the information displayed the status display includes the date and time of the mission, along with a GMT time in seconds, the position of the UGV in Universal Transverse Mercator (UTM) and lat/long position, and vehicle speed.

5. The method of claim 1, wherein listed below a compass is a cardinal direction of the vehicle in degrees.

6. The method of claim 1, further comprising the steps of

Providing a touch screen mode;

providing “

+”

and “

−

”

display buttons used to zoom in and out, respectively; and

providing a “

Reset”

display button to re-center the navigation map.

7. The method of claim 1, further comprising the steps of

receiving data from a UGV mission;

storing data from the UGV in a database of UGV mission recordings

opening/viewing the data.

8. The method of claim 7, wherein

the data required to run the present invention consists of image files (.jpg) from the vehicle'"'"'s cameras and daq files that record the navigation information;

these files are time and date stamped;

placing all the image and daq files from a single mission into a folder on the computer on which a user will be executing the program,

each mission has its own separate folder to ensure that the data loads correctly; and

retrieving the files from the vehicle locally or remotely over a computer network.

9. The method of claim 1, further comprising the steps of

exploring the images of the path mapped by the vehicle in space by clicking on any point of the navigation map;

and

exploring the images of the path mapped by the vehicle in time by using the horizontal scroll bar or playback controls below the camera image;

selecting a point on either the navigation map or the scroll bar to display the corresponding camera image at the vehicle'"'"'s respective time and position;

clicking and dragging the camera image to rotate the camera to further inspect the image;

using a scroll button on a mouse to zoom in or out of the image when the cursor is on the image; and

clicking the left and right mouse buttons simultaneously and dragging on the image, a user can also zoom in or out of the image.

10. The method of claim 1, wherein

providing LADAR maps as semi-transparent allowing the user to view the aerial imagery underneath the map;

and

panning and zooming in and out on the navigation map, LADAR map, and aerial imagery get a detailed view of the area.

11. The method of claim 1, wherein events are tagged automatically in the recoding by the UGV.

12. The method of claim 1, further comprising the steps of

saving, adding, editing, or deleting an event;

rearranging the order of time lines by moving them up or down in order for easier comparison; and

closing a time line if no longer needed.

13. The method of claim 1, further comprising the steps of

providing a built-in 3D viewer;

displaying a 3D map in addition to the navigation and camera images;

locking the viewpoints of three images to each other so that zooming and panning on the image will affect the 3D view and vice versa;

adjusting the 3d map, navigation map, and image to match the pan/tilt/zoom commands of the operator.

14. The method of claim 1, further comprising the steps of

adding one or more tags to a mission;

annotating the image data with one or more tags;

searching one or more missions by the tag.

15. The method of claim 14, further comprising the steps of

providing a map server;

providing a map user interface;

providing controls so that the map display can be zoomed and panned using mouse actions;

selecting the time ranges to search using calendar inputs placed below the map display;

displaying status information to the right of the date/time selection;

providing a timeline featured at the bottom of the UI map display;

conducting a search,

when a search is conducted, events will appear on the timeline;

these events can be selected, and pertinent information is displayed above the timeline.

16. The method of claim 1, further comprising the steps of

executing the method on a mobile phone application running on a mobile phone;

writing to a phone'"'"'s SD card after the mission is completed

using post-processed data

displaying the data in real-time on a live interface on the mobile phone;

streaming the images as they come in from the UGV to the mobile phone;

automatically refreshing the view on the display screen;

choosing to continue watching the live stream;

panning and zooming on the imagery; and

pausing the live streaming and reviewing older images.

17. The method of claim 16, further comprising the steps of

recording images in JPEG format;

compressing the JPEG images;

sending the compressed JPEG images as User Datagram Protocol (UDP) messages;

providing a set of configurable parameters that set the desired image properties;

one configurable parameter is tile size;

sending an image as a series of tiles form the UGV to the mobile phone;

reconstructing on the phone the series of tiles to form a full image; and

storing each image sent to the phone locally on the phone'"'"'s sdcard.

18. The method of claim 17, further comprising the steps of

providing a slide bar at the bottom of the display screen providing a reference as to where the currently viewed image is located with respect to the last image received;

clicking or dragging the slide bar to pause the live stream;

reviewing the imagery as desired, using either the forwards and backwards buttons or the slide bar; and

returning to the live stream by pressing the button on the right side of the screen or dragging the slide bar all the way to the end of the slide bar.

19. The method of claim 18, further comprising the steps of

selecting or clicking on a static object of interest (OOI) through two or more frames;

using the navigation information of the robot in those frames, along with the positions in the image clicked or selected;

triangulating the object to its global position;

adjusting the virtual camera to this new focus point;

moving forward and backward in the video stream;

centering the camera image by panning and tilting the virtual camera around the object; and

viewing the object from multiple angles.

20. The method of claim 19, further comprising the steps of

displaying path of the robots from both missions are displayed on the map in two different colors;

clicking the map on an overlapping location;

navigating to that location using the buttons on the app brings up the two images side by side;

using navigation information from the time the images were taken to align the orientation of the images so that the images shown on the screen are showing the same area of the environment;

locking the images so that moving a finger on the screen changes the pan/tilt/zoom of both images; and

comparing images from two separate missions.

21. The method of claim 20, further comprising the steps of

providing a menu option to unlock and re-lock the images in case a heading error produces an incorrect alignment.

22. The method of claim 20, wherein two missions can be compared after-action or a previous mission can be compared to the live mission.

23. A program storage device readable by a computer machine, tangibly embodying program instructions executable by the computer machine to perform method steps for visualization enhancement for situational awareness, said method steps comprising:

providing a graphical user interface enabling a user to interact with the software program running on the computer machine running;

the computer machine providing a database of vehicle mission recordings;

selecting one or more mission recordings;

displaying a GUI window comprising;

a navigation map associated with the mission recording wherein,

the navigation map displays the traveled path traveled of an Unmanned Ground Vehicle (UGV) in first color, if available

the current position of the vehicle is indicated by a triangle in a second Color,

the front of the vehicle is designated by the point of the triangle,

the field of view (FOV) of the camera is also indicated on the triangle by the two adjacent lines, shown in a third color when;

a status display; and

taking one or more camera images from the mission recording;

overlaying on the top of the image display is the cardinal direction of the displayed camera image;

changing the direction with camera panning within the still frame image;

progressing through the timeline of a mission using a horizontal scroll bar slider wherein each slider increment is equivalent to moving one second in time

allowing a user to jump to any point from the beginning to the end of the mission;

quickly incrementing or decrementing of the slider;

providing ‘

Front’ and

‘

Back’

buttons next to the scroll bar;

allowing the user to toggle between the front camera and the rear camera of the vehicle;

providing a set of playback controls that allow the user to play, pause, rewind, and fast forward through the images,

repeatedly pressing the rewind or fast-forward buttons increases the speed;

adding one or more tags to a mission;

annotating the image data with one or more tags;

searching one or more missions by the tag, providing a map server;

providing a map user interface;

providing controls so that the map display can be zoomed and panned using mouse actions;

selecting the time ranges to search using calendar inputs placed below the map display;

displaying status information to the right of the date/time selection;

providing a timeline featured at the bottom of the UI map display;

conducting a search,

when a search is conducted, events will appear on the timeline;

these events can be selected, and pertinent information is displayed above the timeline;

querying a specific location;

dragging a bounding box over the map;

the resulting navigation paths will be drawn on the display map;

search results summarizes summarizing the collections are found in the bounding box;

colored lines on the timeline represent the start time of each collection in the search results, where each collection is a ten minute segment of data; and

when an event on the timeline is selected, the collection information is displayed in the search results section.

24. The method of claim 23, further comprising the steps of

downloading and displaying 360 degree imagery from the database;

and

displaying multiple images from different parts of the timeline simultaneously.

25. The method of claim 24, further comprising the steps of

providing a search server;

using a combination of spatial and temporal indexes at varying scales to answer search queries that may include time, distance or producer identity constraints;

sending a query to a search server;

using a query optimizer to determine the best search strategy for the constraints given by

the search server;

conducting a Time/ID search through global linear indices;

conducting a spatial search through indices in a hierarchy of grids where the server looks at a grid hierarchy that splits the surface of the planet into distinct cells,

data are associated with cells at certain levels of the hierarchy;

the server performs a recursive search within this hierarchy, identifying the largest cells that are entirely within the search radius;

cells that are only partially covered by the search area are subdivided until either a completely covered cell is found or the process reaches the lowest level of grid cells;

the server takes its list of matching cells and processes each one;

any cell that has been populated will appear in the root-level tree that maps cell IDs to a set of local indices;

when the particular cell is completely enclosed within the search area, the server looks in the cell'"'"'s local temporal index for collection units that match the time constraint, otherwise, the server will look in either a spatial tree for queries that match only part of the cell but do not specify a time range, or a local spatiotemporal index for queries that specify both a part of the cell and a time range; and

the search server returns a result set in which it lists each collection or 10-minute unit it found.

26. The method of claim 25, further comprising the steps of

querying a collect from the search server;

listing each collection'"'"'s data;

and when the client requests 10-minute units, the server lists each unit'"'"'s start time as well as its sequential number within the collection and all of the data that would be returned for the corresponding collection; and

including a summary of the unit or collection, consisting of a list of time-stamped points at a specified resolution.