System and method for enhanced situation awareness and visualization of environments

• US 20070070069A1
• Filed: 09/26/2006
• Published: 03/29/2007
• Est. Priority Date: 09/26/2005
• Status: Abandoned Application

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Samarasekera, Supun, Kumar, Rakesh, Sawhney, Harpreet, Aggarwal, Manoj, Oskiper, Taragay
• Application Number:
  US11/527,987
• Publication Number:
  US 20070070069A1
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 17/05 Geographic models

Abstract

The present invention provides a system and method for processing real-time rapid capture, annotation and creation of an annotated hyper-video map for environments. The method includes processing video, audio and GPS data to create the hyper-video map which is further enhanced with textual, audio and hyperlink annotations that will enable the user to see, hear, and operate in an environment with cognitive awareness. Thus, this annotated hyper-video map provides a seamlessly navigable, situational awareness and indexable high-fidelity immersive visualization of the environment.

Claims

1. A method for providing an immersive visualization of an environment, the method comprising:

providing a map of the environment;

receiving a plurality of captured video streams of the environment via a video camera mounted on a moving platform;

associating navigation data with said captured video streams;

said navigation data includes location and orientation data of the moving platform for each said captured video stream; and

retrieving the associated navigation data with said video stream to compute a metadata, wherein said metadata comprise 3D visualization of the location and orientation of the moving platform for each of the captured video stream; and

automatically processing said video streams with said associated navigation data and the 3D visualization with the map to create an-hypervideo map;

wherein said hyper-video map provides a navigable and indexable high-fidelity visualization of the environment.

2. The method of claim 1 further comprising:

receiving audio data of the environment and the moving platform;

filtering the audio data of the moving platform; and

synchronizing the filtered audio data with said video streams.

3. The method of claim 1 wherein said navigation data comprises a global positioning satellite data of the moving platform for each of the captured video frame.

4. The method of claim 1 wherein said navigation data comprises an inertial measurement data of an altitude, location, and motion of the moving platform for each of the captured video frame.

5. The method of claim 1 wherein said metadata of 3D visualization is computed by detecting and tracking the multiple video streams to establish point correspondences over time and employing 3D motion constraints between the multiple video streams to hypothesize and test numerous pose hypotheses to produce 3D motion poses of the moving platform.

6. The method of claim 1 wherein said processing comprising:

scanning the metadata to generate a graph having nodes corresponding to a trajectory followed by the moving platform;

extracting from the video stream a corresponding video clip for each said node and storing a pointer to the video clip in the node; and

generating a hyper-video map displaying a road structure of the map of each video frame with highlighted road segments corresponding to the stored pointer for each video clip in the node.

7. The method of claim 1 further comprising recognizing landmarks in said video streams based on the landmark database to identify the location of the moving platform.

8. The method of claim 1 further comprising:

compressing the video data and storing in a format to enable seamless playback of the video frames.

9. The method of claim 1 further comprising:

identifying sites within the video streams and processing said metadata to measure distances between sites in the environment.

10. The method of claim 1 further comprising:

tracking and classifying objects within the video streams; and

providing annotation of the hyper-video map displaying said objects.

11. The method of claim 1 further comprising:

storyboarding of the captured video streams of the environment, said storyboarding comprising a virtual summarization of a route laid out on the annotated hyper-video map

12. The method of claim 1 further comprising:

providing at least two routes displayed on the annotated hyper-video map; and

comparing the at least two routes to identify any changes in the environment.

13. The method of claim 11 further comprising:

highlighting said changes in the video stream and the hyper-video map.

14. The method of claim 1 further comprising:

extracting a structure of the environment from the video streams;

said structure comprising route and objects; and

processing said extracted structure to render a 3D image of the structure of the environment.

15. A method for providing a real-time immersive visualization of an environment, the method comprising:

providing a map of the environment

receiving in real-time a continuous plurality of captured video streams of the environment via a video camera mounted on a moving platform;

associating navigation data with said captured video streams;

said navigation data includes location and orientation data of the moving platform for each said captured video stream; and

retrieving the associated navigation data with said video stream to compute a metadata, wherein said metadata comprise 3D visualization of the location and orientation of the moving platform for each of the captured video stream; and

automatically processing said video streams with said associated navigation data and the 3D visualization with the map to create an-hypervideo map;

wherein said hyper-video map provides a navigable and indexable high-fidelity visualization of the environment.

16. The method of claim 15 further comprising:

receiving in real time a continuous audio data of the environment and the moving platform;

filtering the audio data of the moving platform; and

synchronizing the filtered audio data with said video streams.

17. The method of claim 15 wherein said navigation data comprises a global positioning satellite data of the moving platform for each of the captured video frame.

18. The method of claim 15 wherein said navigation data comprises an inertial measurement data of an altitude, location, and motion of the moving platform for each of the captured video frame.

19. The method of claim 15 wherein said metadata of 3D visualization is computed by detecting and tracking the multiple video streams to establish point correspondences over time and employing 3D motion constraints between the multiple video streams to hypothesize and test numerous pose hypotheses to produce 3D motion poses of the moving platform.

20. The method of claim 15 wherein said processing comprising:

scanning the metadata to generate a graph having nodes corresponding to a trajectory followed by the moving platform;

extracting from the video stream a corresponding video clip for each said node and storing a pointer to the video clip in the node; and

generating a hyper-video map displaying a road structure of the of each video frame with highlighted road segments corresponding to the stored pointer for each video clip in the node.

21. The method of claim 15 further comprising recognizing landmarks in said video streams based on the landmark database to identify the location of the moving platform.

22. The method of claim 15 further comprising:

compressing the video data and storing in a format to enable seamless playback of the video frames.

23. The method of claim 15 further comprising:

identifying sites within the video streams and processing said metadata to measure distances between sites in the environment.

24. The method of claim 15 further comprising:

tracking and classifying objects within the video streams; and

providing annotation of the hyper-video map displaying said objects.

25. The method of claim 15 further comprising:

storyboarding of the captured video streams of the environment, said storyboarding comprising a virtual summarization of a route laid out on the annotated hyper-video map

26. The method of claim 15 further comprising:

providing at least two routes displayed on the annotated hyper-video map; and

comparing the at least two routes to identify any changes in the environment.

27. The method of claim 26 further comprising:

highlighting said changes in the video stream and the hyper-video map.

28. The method of claim 15 further comprising:

extracting a structure of the environment from the video streams;

said structure comprising route and objects; and

processing said extracted structure to render a 3D image of the structure of the environment.

29. A system for providing an immersive visualization of an environment, the system comprising:

a capture device comprising at least one video sensor mounted in a moving platform to capture a plurality of video streams of the environment and a navigation unit mounted on the moving platform to provide location and orientation data of the environment for each said captured video stream;

a hyper-video database linked to the capture device for storing the captured video stream and the navigation data;

said database comprises a map of the environment;

a vision aided navigation processing tool coupled to the capture device and the hyper-video database for retrieving the combined video stream and the navigation data to compute a metadata, said metadata comprises a 3D visualization of the location and orientation of the moving platform for each said captured video stream, said 3D visualization is stored in the hyper-video database; and

a hyper-video map and route visualization processing tool coupled to the hyper-video database for automatically processing the video stream, the metadata and the 3D visualization with the map of the environment to generate a hyper-video map of the environment.

30. The system of claim 29 wherein said video streams of the environment are captured in real-time.

31. The system of claim 29 wherein said hyper-video map and route visualization processing tool provides to a user a graphical user interface of the hyper-video map of the environment.

32. The system of claim 29 wherein said capture device further comprises an audio sensor for capturing an audio data of the environment and the moving platform.

33. The system of claim 32 wherein said audio data is captured in real time.

34. The system of claim 29 further comprising an audio processing tool for filtering the audio data of the moving platform and reducing the noise in the filtered audio data.

35. The system of claim 33 wherein said audio processing tool provides a 3D virtual audio rendering of the audio data.

36. The system of claim 33 wherein said audio processing tool provides for an interface for interfacing audio information into a video stream for visualization.

37. The system of claim 33 wherein said video sensor comprises a 360 degrees video camera for capturing video data of the environment at any given location and time for a complete 360 degree viewpoints.

38. The system of claim 33 wherein said video sensor comprises a lidar scanner for capturing the image to provide an absolute position of the moving platform.

39. The system of claim 33 wherein said navigation unit comprises a GPS antenna for providing a satellite global positioning of the moving platform for each of the captured video frame.

40. The system of claim 33 wherein said navigation unit comprises an inertial measuring unit for providing an altitude, location, and motion of the moving platform for each of the captured video frame.