Methods and systems relating to an augmented virtuality environment

• US 8,732,592 B2
• Filed: 06/08/2009
• Issued: 05/20/2014
• Est. Priority Date: 06/08/2009
• Status: Active Grant

• Current Assignee: Humatics Corporation
• Original Assignee: Battelle Energy Alliance LLC (Battelle Memorial Institute)
• Inventors: Nielsen, Curtis W., Anderson, Matthew O., McKay, Mark D., Wadsworth, Derek C., Boyce, Jodie R., Hruska, Ryan C., Koudelka, John A., Whetten, Jonathon, Bruemmer, David J.
• Primary Examiner(s): Bashore, William
• Assistant Examiner(s): NABI, REZA U
• Application Number:
  US12/480,459
• Publication Number:
  US 20100313146A1
• Time in Patent Office: 1,807 Days
• Field of Search: 715/757
• US Class Current: 715/757
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems and methods relating to an augmented virtuality system are disclosed. A method of operating an augmented virtuality system may comprise displaying imagery of a real-world environment in an operating picture. The method may further include displaying a plurality of virtual icons in the operating picture representing at least some assets of a plurality of assets positioned in the real-world environment. Additionally, the method may include displaying at least one virtual item in the operating picture representing data sensed by one or more of the assets of the plurality of assets and remotely controlling at least one asset of the plurality of assets by interacting with a virtual icon associated with the at least one asset.

Claims

1. A method of operating an augmented virtuality system, comprising:

displaying imagery of a real-world environment in an operating picture by;

displaying substantially static still base imagery, developed from real-world base imagery data as part of the operating picture;

acquiring dynamic real-time imagery depicting a selected region of the substantially static still base imagery;

displaying the dynamic real-time imagery to substantially coincide with the selected region as it appears on the operating picture;

displaying a plurality of virtual icons in the operating picture representing at least some assets of a plurality of assets positioned in the real-world environment; and

displaying at least one virtual item in the operating picture representing data sensed by one or more assets of the plurality of assets at a position in the operating picture where the data was sensed; and

navigating at least one asset of the plurality of assets positioned within the real-world environment, the navigating comprising;

remotely controlling the at least one asset of the plurality of assets by interacting with a virtual icon in the operating picture associated with the at least one asset; and

generating and updating a path plan in substantially real-time for the at least one asset of the plurality of assets as it moves in the real-world environment.

2. The method of claim 1, wherein remotely controlling at least one asset of the plurality of assets comprises teleoperating a single asset of the plurality of assets.

3. The method of claim 2, wherein teleoperating a single asset of the plurality of assets further comprises displaying a view from a perspective of the single asset.

4. The method of claim 1, wherein displaying at least one virtual item in the operating picture representing data sensed by one or more assets of the plurality of assets comprises displaying at least one of a color, alphanumeric text, and a three-dimensional virtual object representing the data sensed by one or more assets of the plurality of assets.

5. The method of claim 1, wherein the dynamic live video imagery is captured by one or more assets of the plurality of assets.

6. The method of claim 1, wherein remotely controlling at least one asset of the plurality of assets comprises sending at least one control signal to the at least one asset of the plurality of assets over a wireless communication medium.

7. The method of claim 6, further comprising adjusting a level of control allocated to the at least one asset of the plurality of assets in carrying out a task.

8. The method of claim 1, wherein displaying at least one virtual item in the operating picture representing data sensed by one or more of the assets of the plurality of assets comprises displaying at least one virtual item representing data sensed by one or more sensors operably coupled to the at least one or more assets of the plurality of assets.

9. The method of claim 8, wherein displaying at least one virtual item in the operating picture representing data sensed by one or more sensors comprises displaying at

least one virtual item representing data sensed by at least one of one or more image sensors, one

or more tracking sensors, one or more range sensors, one or more temperature sensors, one or more radiation sensors, one or more smoke detectors, one or more gas sensors, one or more wind sensors, one or more pressure sensors, one or more humidity sensors, one or more infrared sensors, one or more counter-mine sensors, one or more explosive sensors, one or more position sensors, one or more lasers, and one or more sonar devices.

10. The method of claim 1, further comprising capturing sensor data with one or more sensors operably coupled to the at least one or more assets of the plurality of assets.

11. The method of claim 10, further comprising transmitting the captured sensor data to at least one remote computer.

12. The method of claim 10, further comprising storing the captured sensor data in at least one computer communicatively coupled to the at least one or more assets of the plurality of assets.

13. The method of claim 1, further comprising capturing sensor data with at least one or more assets of the plurality of assets and substantially simultaneously displaying one or more virtual items in the operating picture representing the sensor data captured by the at least one or more assets of the plurality of assets.

14. A method of operating an augmented virtuality system, comprising:

displaying imagery of a real-world environment in an operating picture by;

displaying substantially static still base imagery developed from real-world base imagery data as part of the operating picture;

acquiring dynamic real-time imagery depicting a selected region of the substantially static still base imagery;

displaying the dynamic real-time imagery to substantially coincide with the selected region as it appears on the operating picture; and

displaying virtual data in the operating picture, wherein at least some of the virtual data represents a plurality of assets positioned in the real-world environment and at least some of the virtual data represents data sensed by one or more assets of the plurality of assets at a position in the operating picture where the data was sensed; and

navigating at least one asset of the plurality of assets positioned within the real-world environment, the navigating comprising;

remotely controlling the at least one asset of the plurality of assets by interacting with a virtual icon in the operating picture associated with the at least one asset; and

generating and updating a path plan in substantially real-time for the at least one asset of the plurality of assets as it moves in the real-world environment.

15. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying at least one virtual item representing sensor data captured by one or more of the assets of the plurality of assets.

16. The method of claim 15, wherein displaying at least one virtual item representing sensor data comprises displaying at least one of a three-dimensional virtual object, alphanumeric text, a three-dimensional virtual icon, a video, and aerial imagery in the operating picture representing sensor data gathered from at least one asset of the plurality of assets.

17. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying at least one color in the operating picture representing sensor data gathered from at least one asset of the plurality of assets.

18. The method of claim 14, further comprising capturing sensor data with at least one or more assets of the plurality of assets and substantially simultaneously displaying

virtual representations of the sensor data captured by the at least one or more assets of the plurality of assets.

19. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying at least one virtual object overlaying the imagery and at a location based on a predefined geospatial position of the at least one virtual object.

20. The method of claim 14, wherein the substantially static still base imagery is obtained from a satellite positioned within the real-world environment.

21. The method of claim 14, wherein displaying imagery of the real-world environment comprises displaying at least one of a live video and at least one still image obtained from at least one sensor coupled to at least one asset of the plurality of assets positioned with the real-world environment.

22. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying a plurality of virtual icons representing at least one of one or more unmanned ground vehicles, one or more unmanned surface vehicles, one or more unmanned air vehicles, one or more unattended ground sensors, and one or more personnel positioned within the real-world environment.

23. The method of claim 14, wherein navigating at least one asset of the plurality of assets comprises remotely controlling at least one asset of the plurality of assets positioned within the real-world environment by interacting with the operating picture.

24. The method of claim 23, wherein remotely controlling at least one asset of the plurality of assets comprises remotely controlling at least one asset of the plurality of assets by interacting with a touch screen display device displaying the operating picture.

25. The method of claim 23, wherein remotely controlling at least one asset of the plurality of assets comprises directly controlling a single asset by teleoperation.

26. The method of claim 14, wherein navigating at least one asset of the plurality of assets further comprises generating a path plan in real-time with the at least one asset of the plurality of assets as the at least one asset of the plurality of assets navigates through the real-world environment.

27. An augmented virtuality system, comprising:

a plurality of assets positioned within a real-world environment;

at least one computer communicatively coupled to each asset of the plurality of assets and including a processor and a non-transitory computer-readable medium coupled to the processor;

a display device operably coupled to the at least one computer; and

an application program stored in the non-transitory computer-readable medium, the application program, when executed by the processor, configured to;

displaying imagery of a real-world environment in an operating picture by;

displaying substantially static still base imagery developed from real-world base imagery data as part of the operating picture;

acquiring dynamic real-time image depicting a selected region of the substantially static still base imagery;

displaying the dynamic real-time imagery to substantially coincide with the selected region as it appears on the operating picture; and

displaying virtual data overlaying imagery of the real-world environment displayed, wherein at least some of the virtual data represents data sensed by at least one asset of the plurality of assets at a position in the operating picture where the data was sensed; and

navigating at least one asset of the plurality of assets positioned within the real-world environment, the navigating comprising;

remotely controlling the at least one asset of the plurality of assets by interacting with a virtual icon in the operating picture associated with the at least one asset; and

generating and updating a path plan in substantially real-time for the at least one asset of the plurality of assets as it moves in the real-world environment.

28. The augmented virtuality system of claim 27, wherein the display device comprises one of a video wall, a three-dimensional video table, a holographic video display, and a monitor.

29. The augmented virtuality system of claim 27, wherein the display device comprises a touch-screen device.

30. A non-transitory computer-readable media storage medium storing instructions that when executed by a processor cause the processor to perform instructions for operating an augmented virtuality system, the instructions comprising:

displaying imagery of a real-world environment in an operating picture by;

displaying substantially static still base imagery developed from real-world base imagery data as part of the operating picture;

acquiring dynamic real-time imagery depicting a selected region of the substantially static still base imagery;

displaying the dynamic real-time imagery to substantially coincide with the selected region as it appears on the operating picture; and

displaying a plurality of virtual items in the operating picture overlaying the imagery, wherein at least one virtual item represents an asset of a plurality of assets positioned in the real-world environment, wherein at least one other virtual item represents sensor data captured by one or more assets of the plurality of assets and presented at a position in the operating picture where the sensor data was captured; and

navigating at least one asset of the plurality of assets positioned within the real-world environment, the navigating comprising;

remotely controlling the at least one asset of the plurality of assets by interacting with a virtual icon in the operating picture associated with the at least one asset; and

generating and updating a path plan in substantially real-time for the at least one asset of the plurality of assets as it moves in the real-world environment.

31. The non-transitory computer-readable media storage medium of claim 30, wherein displaying a plurality of virtual items in the operating picture comprises displaying at least one of a three-dimensional virtual object, a three-dimensional virtual icon, imagery, an image, and alphanumeric text in the operating picture.

32. A method of generating an augmented virtuality environment, comprising:

displaying imagery of a real-world environment in an operating picture by;

displaying substantially static still base imagery developed from real-world base imagery data as part of the operating picture;

acquiring dynamic real-time imagery depicting a selected region of the substantially static still base imagery;

displaying the dynamic real-time imagery to substantially coincide with the selected region as it appears on the operating picture;

displaying at least one virtual icon in the operating picture representing an asset of a plurality of assets positioned in the real-world environment; and

displaying virtual data in the operating picture representing data sensed by at least two assets of the plurality of assets at a position in the operating picture where the data was sensed; and

navigating at least one asset of the plurality of assets positioned within the real-world environment, the navigating comprising;

remotely controlling the at least one asset of the plurality of assets by interacting with a virtual icon in the operating picture associated with the at least one asset; and

generating and updating a path plan in substantially real-time for the at least one asset of the plurality of assets as it moves in the real-world environment.

33. The method of claim 32, wherein displaying real-world imagery comprises:

displaying at least one of satellite imagery and local aerial imagery as the substantially static still base imagery; and

displaying a live video captured by an asset of the plurality of assets and mapped over the substantially static still base imagery.

34. The method of claim 32, further comprising displaying at least one virtual object in the operating picture representing geospatial data received from a geospatial information system (GIS).