Interacting augmented reality and virtual reality

• US 20030210832A1
• Filed: 05/13/2002
• Published: 11/13/2003
• Est. Priority Date: 05/13/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Charles Benton
• Inventors: Benton, Charles
• Granted Patent: US 6,917,370 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A combination of interacting virtual reality (VR) and augmented reality (AR) system is provided where the system automatically switches from VR to AR when the user selected viewpoint coincides, within tolerance, with the observable AR viewpoint. The invention discloses an apparatus adapted to provide easy access and automatic switching between AR and VR, and method thehrefor.

Claims

1. A visual information display apparatus comprising:

a display;

a computer coupled to said display, and operable to generate thereupon a representation of a computerized environment model or a portion thereof responsive to user selected viewpoint;

an image sensor adapted to generate an observed scene image from a scene viewpoint determined by the location and orientation of said image sensor;

a viewpoint comparator adapted to compare the user selected viewpoint and the scene viewpoint, and to issue a switch signal when said scene viewpoint coincide within predetermined proximity with said user selectable viewpoint; and

,

an image integrator adapted to selectively merge at least a portion of said environment model with the observed scene image responsive to said switch signal.

2. The visual information display apparatus of claim 1, further comprising a viewpoint selector operable to vary said user selected viewpoint continuously.

3. The visual information display apparatus of claim 1, further comprising an orientation sensor and a location sensor coupled to said image sensor;

and a wherein said scene viewpoint is determined by outputs of said location and orientation sensors.

4. The visual information display apparatus of claim 1, wherein said image sensor comprises a camera

5. The visual information display apparatus of claim 1, wherein the display is selected from a group consisting of a head mounted display, a television display, a computer display, a handheld display, a virtual display, a heads up display, an optical combiner display, a stereoscopic display, a monocular display, a binocular display, a selectively opaque display mounted in the user field of view, a retinal illumination display, projective display, or a combination hereof.

6. The visual information display apparatus of claim 1 wherein said computerized environment model comprises cartographic data.

7. The visual information display apparatus of claim 1 wherein said computerized environment model comprises navigational data.

8. The visual information display apparatus of claim 1 wherein said computerized data environment model comprises pre-captured image data, or computer generated image data, or a combination thereof.

9. The visual information display apparatus of claim 1 wherein said computerized environment model is further adapted to receive dynamic object information, and display a representation thereof.

10. The visual information display apparatus of claim 9 wherein said dynamic object information comprises radar obtained information.

11. The visual information display apparatus of claim 9 wherein said dynamic object information comprises sonar obtained information.

12. The visual information display apparatus of claim 9 wherein said dynamic object information is manually entered.

13. The visual information display apparatus of claim 9 wherein said dynamic object information are provided by an external information system.

14. A visual information display apparatus adapted to operate in conjunction with a computer and a display device, said apparatus comprising:

a computerized image generator adapted to display a representation of a computerized environment model or a portion thereof from a plurality of user selected viewpoints;

an image integrator adapted to automatically merge a representation of at least a portion of said environment model with an observed scene input viewed from an observed scene viewpoint, when the user selected viewpoint is within a predetermined proximity to the observed scene viewpoint.

15. The visual information display apparatus of claim 14, wherein said observed scene is inputted as see-through viewing.

16. The visual information display apparatus of claim 14 wherein said observed scene is inputted as video-through viewing.

17. The visual information system of claim 14, wherein light received from the observed scene is amplified.

18. The visual information display apparatus of claim 14, wherein the display is selected from a group consisting of a head mounted display, a television display, a computer display, a handheld display, a virtual display, a heads up display, an optical combiner display, a stereoscopic display, a monocular display, a binocular display, a selectively opaque display mounted in the user field of view, a retinal illumination display, projective display, or a combination hereof.

19. The visual information display apparatus of claim 14, further comprising a viewpoint selector operable to vary said user selected viewpoint continuously.

20. The visual information display apparatus of claim 14 wherein said computerized environment model comprises cartographic data.

21. The visual information display apparatus of claim 14 wherein said computerized environment model comprises navigational data.

22. The visual information display apparatus of claim 14 wherein said computerized data environment model comprises pre-captured image data, or computer generated image data, or a combination thereof.

23. The visual information display apparatus of claim 14 wherein said computerized environment model or said image integrator, or a combination thereof, is further adapted to receive dynamic object information, and display a representation thereof.

24. The visual information display apparatus of claim 23 wherein said dynamic object information comprises radar obtained information.

25. The visual information display apparatus of claim 23 wherein said dynamic object information comprises sonar obtained information.

26. The visual information display apparatus of claim 23 wherein said dynamic object information is manually entered.

27. The visual information display apparatus of claim 22 wherein said dynamic object information is provided by an external information system.

28. The visual information display apparatus of claim 23 wherein said dynamic objects comprise information obtained using a sensor selected from a group consisting of magnetic sensor, heat sensor, infra red sensor, echo location sensor, radio location sensor, remote location reporting sensor, proximity sensor, or a combination thereof.

29. The visual information display apparatus of claim 14 further comprising a location sensor and an orientation sensor, forming an observable viewpoint sensor.

30. The visual information display apparatus of claim 29 wherein said observable viewpoint sensor or a portion thereof is mounted on a user head, and is responsive to said user head orientation and location.

31. The visual information display apparatus of claim 29 wherein said observable viewpoint sensor or a portion thereof is responsive to user eye movement.

32. A method for automatic switching between virtual reality and augmented reality, the method comprising the steps of:

displaying an image representing a computerized environment model or a portion thereof on a display device;

selecting a viewpoint, and modifying said representation in accordance with the user selectable viewpoint;

sensing observable scene viewpoint;

comparing said user selectable viewpoint and observable viewpoint;

merging at least a portion of said environment model image with the observable scene if the user selectable viewpoint and the observable scene viewpoint are within a predetermined proximity to each other.

33. A method according to claim 32 further comprising the step of blocking the image of the observable scene when the selected viewpoint and the observed scene viewpoint are in separated above a predetermined proximity.

34. A method according to claim 32 wherein said step of merging further comprise the step of moving the viewpoint to the observable scene viewpoint.

35. A method according to claim 32 wherein said step of displaying further comprises displaying information regarding dynamic objects.

36. A method according to claim 35 wherein said information regarding dynamic objects is provided by an external information system.

37. A method according to claim 32 wherein said step of displaying further comprise the step of displaying dynamic objects, or representations thereof.

38. A visual information display apparatus adapted to operate in conjunction with a computer and a display device, said apparatus having:

an environment model for providing information about an environment;

selector means for selecting a selected viewpoint;

image generator means for generating a computerized image from said environment model responsive to said selected viewpoint;

visual input means for receiving an observed image, said visual input means having a viewpoint;

said apparatus characterized by;

controllable merging means, for selectively merging said observed image with said computerized image, for display on said display device;

a viewpoint comparator means for comparing said selected viewpoint with said observed viewpoint, and controlling said merging means responsive to the relationship between said viewpoints.