Interacting augmented reality and virtual reality

• US 6,917,370 B2
• Filed: 05/13/2002
• Issued: 07/12/2005
• Est. Priority Date: 05/13/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Charles Benton
• Inventors: Benton, Charles
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Kumar, Srilakshmi K
• Application Number:
  US10/144,467
• Publication Number:
  US 20030210832A1
• Time in Patent Office: 1,156 Days
• Field of Search: 345/8, 345/204, 345629-633, 345/7, 345/419, 340/705
• US Class Current: 345/633
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A combination of interacting virtual reality (VR) and augmented reality (AR) system is provided where the system automatically switches from VR to AR when the user selected viewpoint coincides, within tolerance, with the observable AR viewpoint. The invention discloses an apparatus adapted to provide easy access and automatic switching between AR and VR, and method thehrefor.

Claims

1. A visual information display apparatus for displaying virtual reality and augmented reality scenes, the apparatus comprising:

a display;

a computer coupled to said display, and operable to generate thereupon a representation of a computerized environment model or a portion thereof responsive to user selected viewpoint;

an image sensor adapted to generate an actual observed scene image from a scene viewpoint determined by the location and orientation of said image sensor;

a viewpoint comparator adapted to compare the user selected virtual viewpoint and the observed scene viewpoint, and to issue a switch signal when said observed scene viewpoint coincide within predetermined proximity with said user selectable virtual viewpoint; and

,

an image integrator adapted to selectively merge at least a portion of said environment model with the observed scene image;

wherein said computer is adapted to switch between displaying a virtual reality scene and an augmented reality scene, in response to said switch signal.

2. The visual information display apparatus of claim 1, further comprising a viewpoint selector operable to vary said user selected virtual, viewpoint continuously.

3. The visual information display apparatus of claim 1, further comprising an orientation sensor and a location sensor coupled to said image sensor;

and a wherein said observed scene viewpoint is determined by outputs of said location and orientation sensors.

4. The visual information display apparatus of claim 1, wherein said image sensor comprises a camera.

5. The visual information display apparatus of claim 1, wherein the display is selected from a group consisting of a head mounted display, a television display, a computer display, a handheld display, a virtual display, a heads up display, an optical combiner display, a stereoscopic display, a monocular display, a binocular display, a selectively opaque display mounted in the user field of view, a retinal illumination display, projective display, or a combination hereof.

6. The visual information display apparatus of claim 1 wherein said computerized environment model comprises cartographic data.

7. The visual information display apparatus of claim 1 wherein said computerized environment model comprises navigational data.

8. The visual information display apparatus of claim 1 wherein said computerized data environment model comprises pre-captured image data, or computer generated image data, or a combination thereof.

9. The visual information display apparatus of claim 1 wherein said computerized environment model is further adapted to receive dynamic object information, and display a representation thereof.

10. The visual information display apparatus of claim 9 wherein said dynamic object information comprises radar obtained information.

11. The visual information display apparatus of claim 9 wherein said dynamic object information comprises sonar obtained information.

12. The visual information display apparatus of claim 9 wherein said dynamic object information is manually entered.

13. The visual information display apparatus of claim 9 wherein said dynamic object information are provided by an external information system.

14. A visual information display apparatus for displaying virtual reality scenes and augmented reality scenes, adapted to operate in conjunction with a computer and a display device, said apparatus comprising:

a computerized image generator adapted to display a virtual representation of a computerized environment model or a portion thereof from a plurality of user selected viewpoints;

an image integrator adapted to automatically merge a representation of at least a portion of said environment model with an actual observed scene input viewed from an observed scene viewpoint, when the user selected virtual viewpoint is within a predetermined proximity to the actual observed scene viewpoint.

15. The visual information display apparatus of claim 14, wherein said observed scene is inputted as see-through viewing.

16. The visual information display apparatus of claim 14 wherein said observed scene is inputted as video-through viewing.

17. The visual information system of claim 14, wherein light received from the observed scene is amplified.

18. The visual information display apparatus of claim 14, wherein the display is selected from a group consisting of a head mounted display, a television display, a computer display, a handheld display, a virtual display, a heads up display, an optical combiner display, a stereoscopic display, a monocular display, a binocular display, a selectively opaque display mounted in the user field of view, a retinal illumination display, projective display, or a combination hereof.

19. The visual information display apparatus of claim 14, further comprising a viewpoint selector operable to vary said user selected virtual viewpoint continuously.

20. The visual information display apparatus of claim 14 wherein said computerized environment model comprises cartographic data.

21. The visual information display apparatus of claim 14 wherein said computerized environment model comprises navigational data.

22. The visual information display apparatus of claim 14 wherein said computerized data environment model comprises pre-captured image data, or computer generated image data, or a combination thereof.

23. The visual information display apparatus of claim 14 wherein said computerized environment model or said image integrator, or a combination thereof, is further adapted to receive dynamic object information, and display a representation thereof.

24. The visual information display apparatus of claim 23 wherein said dynamic object information comprises radar obtained information.

25. The visual information display apparatus of claim 23 wherein said dynamic object information comprises sonar obtained information.

26. The visual information display apparatus of claim 23 wherein said dynamic object information is manually entered.

27. The visual information display apparatus of claim 22 wherein said dynamic object information is provided by an external information system.

28. The visual information display apparatus of claim 23 wherein said dynamic objects comprise information obtained using a sensor selected from a group consisting of magnetic sensor, beat sensor, infrared sensor, echo location sensor, radio location sensor, remote location reporting sensor, proximity sensor, or a combination thereof.

29. The visual information display apparatus of claim 14 further comprising a location sensor and an orientation sensor, forming an actual observed viewpoint sensor.

30. The visual information display apparatus of claim 29 wherein said actual observed viewpoint sensor or a portion thereof is mounted on a user head, and is responsive to said user head orientation and location.

31. The visual information display apparatus of claim 29 wherein said observable viewpoint sensor or a portion thereof is responsive to user eye movement.

32. A method for automatic switching between virtual reality scenes and augmented reality scenes, the method comprising the steps of:

displaying a virtual image representing a computerized environment model or a portion thereof on a display device;

selecting a virtual viewpoint, and modifying said virtual image in accordance with the selected virtual viewpoint;

sensing an actual observed scene from an actual observed viewpoint;

comparing said selected virtual viewpoint and actual observed viewpoint;

merging at least a portion of said environment model virtual image with the actual observed scene, if the user selectable virtual viewpoint and the actual observed scene viewpoint are within a predetermined proximity to each other.

33. A method according to claim 32 further comprising the step of blocking the image of the actual observed scene when the selected virtual viewpoint and the actual observed scene viewpoint are in separated above a predetermined proximity.

34. A method according to claim 32 wherein said step of merging further comprise the step of moving the user selectable virtual viewpoint to the actual observed scene viewpoint.

35. A method according to claim 32 wherein said step of displaying further comprises displaying information regarding dynamic objects.

36. A method according to claim 35 wherein said information regarding dynamic objects is provided by an external information system.

37. A method according to claim 32 wherein said step of displaying further comprise the step of displaying dynamic objects, or representations thereof.

38. A visual information display apparatus for displaying virtual reality scenes and augmented reality scenes, adapted to operate in conjunction with a computer and a display device, said apparatus having:

an environment model for providing information about an environment;

selector means for selecting a selected virtual viewpoint;

image generator means for generating a computerized image from said environment model responsive to said selected virtual viewpoint;

visual input means for receiving an actual observed image, said visual input means having an actual viewpoint;

said apparatus characterized by;

controllable merging means, for selectively merging said observed image with said computerized image, for display on said display device;

a viewpoint comparator means for comparing said selected virtual viewpoint with said actual observed viewpoint, and controlling said merging means to switch between augmented reality and virtual reality responsive to the relationship between said viewpoints.