METHODS AND SYSTEMS RELATING TO AN AUGMENTED VIRTUALITY ENVIRONMENT

• US 20100313146A1
• Filed: 06/08/2009
• Published: 12/09/2010
• Est. Priority Date: 06/08/2009
• Status: Active Grant

• Current Assignee: Humatics Corporation
• Original Assignee: Battelle Energy Alliance LLC (Battelle Memorial Institute)
• Inventors: Anderson, Matthew O., McKay, Mark D., Wadsworth, Derek C., Boyce, Jodie R., Koudelka, John A., Whetten, Jonathon, Hruska, Ryan C., Bruemmer, David J., Nielsen, Curtis W.
• Granted Patent: US 8,732,592 B2
• Time in Patent Office: Days
• US Class Current: 715/757
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems and methods relating to an augmented virtuality system are disclosed. A method of operating an augmented virtuality system may comprise displaying imagery of a real-world environment in an operating picture. The method may further include displaying a plurality of virtual icons in the operating picture representing at least some assets of a plurality of assets positioned in the real-world environment. Additionally, the method may include displaying at least one virtual item in the operating picture representing data sensed by one or more of the assets of the plurality of assets and remotely controlling at least one asset of the plurality of assets by interacting with a virtual icon associated with the at least one asset.

Claims

1. A method of operating an augmented virtuality system, comprising:

displaying imagery of a real-world environment in an operating picture;

displaying a plurality of virtual icons in the operating picture representing at least some assets of a plurality of assets positioned in the real-world environment;

displaying at least one virtual item in the operating picture representing data sensed by one or more assets of the plurality of assets; and

remotely controlling at least one asset of the plurality of assets by interacting with a virtual icon in the operating picture associated with the at least one asset.

2. The method of claim 1, wherein remotely controlling at least one asset of the plurality of assets comprises teleoperating a single asset of the plurality of assets.

3. The method of claim 2, wherein teleoperating a single asset of the plurality of assets further comprises displaying a view from a perspective of the single asset.

4. The method of claim 1, wherein displaying at least one virtual item in the operating picture representing data sensed by one or more assets of the plurality of assets comprises displaying at least one of a color, alphanumeric text, and a three-dimensional virtual object representing the data sensed by one or more assets of the plurality of assets.

5. The method of claim 1, wherein displaying imagery comprises displaying a live video of the real-world environment captured by one or more assets of the plurality of assets.

6. The method of claim 1, wherein remotely controlling at least one asset of the plurality of assets comprises sending at least one control signal to the at least one asset of the plurality of assets over a wireless communication medium.

7. The method of claim 6, further comprising adjusting a level of control allocated to the at least one asset of the plurality of assets in carrying out a task.

8. The method of claim 1, wherein displaying at least one virtual item in the operating picture representing data sensed by one or more of the assets of the plurality of assets comprises displaying at least one virtual item representing data sensed by one or more sensors operably coupled to the at least one or more assets of the plurality of assets.

9. The method of claim 8, wherein displaying at least one virtual item in the operating picture representing data sensed by one or more sensors comprises displaying at least one virtual item representing data sensed by at least one of one or more image sensors, one or more tracking sensors, one or more range sensors, one or more temperature sensors, one or more radiation sensors, one or more smoke detectors, one or more gas sensors, one or more wind sensors, one or more pressure sensors, one or more humidity sensors, one or more infrared sensors, one or more counter-mine sensors, one or more explosive sensors, one or more position sensors, one or more lasers, and one or more sonar devices.

10. The method of claim 1, further comprising capturing sensor data with one or more sensors operably coupled to the at least one or more assets of the plurality of assets.

11. The method of claim 10, further comprising transmitting the captured sensor data to at least one remote computer.

12. The method of claim 10, further comprising storing the captured sensor data in at least one computer communicatively coupled to the at least one or more assets of the plurality of assets.

13. The method of claim 1, further comprising capturing sensor data with at least one or more assets of the plurality of assets and substantially simultaneously displaying one or more virtual items in the operating picture representing the sensor data captured by the at least one or more assets of the plurality of assets.

14. A method of operating an augmented virtuality system, comprising:

displaying imagery of a real-world environment in an operating picture;

displaying virtual data in the operating picture, wherein at least some of the virtual data represents a plurality of assets positioned in the real-world environment; and

navigating at least one asset of the plurality of assets positioned within the real-world environment.

15. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying at least one virtual item representing sensor data captured by one or more of the assets of the plurality of assets.

16. The method of claim 15, wherein displaying at least one virtual item representing sensor data comprises displaying at least one of a three-dimensional virtual object, alphanumeric text, a three-dimensional virtual icon, a video, and aerial imagery in the operating picture representing sensor data gathered from at least one asset of the plurality of assets.

17. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying at least one color in the operating picture representing sensor data gathered from at least one asset of the plurality of assets.

18. The method of claim 14, further comprising capturing sensor data with at least one or more assets of the plurality of assets and substantially simultaneously displaying virtual representations of the sensor data captured by the at least one or more assets of the plurality of assets.

19. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying at least one virtual object overlaying the imagery and at a location based on a predefined geospatial position of the at least one virtual object.

20. The method of claim 14, wherein displaying imagery of the real-world environment comprises displaying imagery of the real-world environment obtained from a satellite positioned within the real-world environment.

21. The method of claim 14, wherein displaying imagery of the real-world environment comprises displaying at least one of a live video and at least one still image obtained from at least one sensor coupled to at least one asset of the plurality of assets positioned with the real-world environment.

22. The method of claim 14, wherein displaying virtual data in the operating picture comprises displaying a plurality of virtual icons representing at least one of one or more unmanned ground vehicles, one or more unmanned surface vehicles, one or more unmanned air vehicles, one or more unattended ground sensors, and one or more personnel positioned within the real-world environment.

23. The method of claim 14, wherein navigating at least one asset of the plurality of assets comprises remotely controlling at least one asset of the plurality of assets positioned within the real-world environment by interacting with the operating picture.

24. The method of claim 23, wherein remotely controlling at least one asset of the plurality of assets comprises remotely controlling at least one asset of the plurality of assets by interacting with a touch screen display device displaying the operating picture.

25. The method of claim 23, wherein remotely controlling at least one asset of the plurality of assets comprises directly controlling a single asset by teleoperation.

26. The method of claim 23, wherein navigating at least one asset of the plurality of assets further comprises updating a path plan in real-time with the at least one asset of the plurality of assets as the at least one asset of the plurality of assets navigates through the real-world environment.

27. The method of claim 14, wherein navigating at least one asset of the plurality of assets further comprises generating a path plan in real-time with the at least one asset of the plurality of assets as the at least one asset of the plurality of assets navigates through the real-world environment.

28. An augmented virtuality system, comprising:

a plurality of assets positioned within a real-world environment;

at least one computer communicatively coupled to each asset of the plurality of assets and including a processor and a computer-readable medium coupled to the processor;

a display device operably coupled to the at least one computer; and

an application program stored in the computer-readable medium, the application program, when executed by the processor, configured to;

display virtual data overlaying imagery of a real-world environment displayed within an operating picture on the display device, wherein at least some of the virtual data represents data sensed by at least one asset of the plurality of assets; and

navigate one or more assets of the plurality of assets positioned within the real-world environment.

29. The augmented virtuality system of claim 28, wherein the display device comprises one of a video wall, a three-dimensional video table, a holographic video display, and a monitor.

30. The augmented virtuality system of claim 28, wherein the display device comprises a touch-screen device.

31. A computer-readable media storage medium storing instructions that when executed by a processor cause the processor to perform instructions for operating an augmented virtuality system, the instructions comprising:

displaying imagery of a real-world environment in an operating picture;

displaying a plurality of virtual items in the operating picture overlaying the imagery, wherein at least one virtual item represents an asset of a plurality of assets positioned in the real-world environment, wherein at least one other virtual item represents sensor data captured by one or more assets of the plurality of assets; and

navigating at least one asset of the plurality of assets positioned within the real-world environment.

32. The computer-readable media storage medium of claim 31, wherein displaying a plurality of virtual items in the operating picture comprises displaying at least one of a three-dimensional virtual object, a three-dimensional virtual icon, imagery, an image, and alphanumeric text in the operating picture.

33. A method of generating an augmented virtuality environment, comprising:

displaying real-world imagery of a real-world environment in an operating picture;

displaying at least one virtual icon in the operating picture representing an asset of a plurality of assets positioned in the real-world environment; and

displaying virtual data in the operating picture representing data sensed by at least two assets of the plurality of assets.

34. The method of claim 33, wherein displaying real-world imagery comprises at least one of displaying satellite imagery, local aerial imagery, and a live video captured by an asset of the plurality of assets.

35. The method of claim 33, further comprising displaying at least one virtual object in the operating picture representing geospatial data received from a geospatial information system (GIS).