LOW-LATENCY FUSING OF VIRTUAL AND REAL CONTENT

• US 20120105473A1
• Filed: 10/27/2010
• Published: 05/03/2012
• Est. Priority Date: 10/27/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bar-Zeev, Avi, Goossen, J. Andrew, Tardif, John, Grossman, Mark S., Singh, Harjit
• Granted Patent: US 9,348,141 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels

Abstract

A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content. In one embodiment, the processing unit is in communication with a hub computing device. The processing unit and hub may collaboratively determine a map of the mixed reality environment. Further, state data may be extrapolated to predict a field of view for a user in the future at a time when the mixed reality is to be displayed to the user. This extrapolation can remove latency from the system.

Claims

1. A system for presenting a mixed reality experience to one or more users, the system comprising:

one or more display devices for the one or more users, each display device including a first set of sensors for sensing data relating to a position of the display device and a display unit for displaying a virtual image to the user of the display device;

one or more processing units, each associated with a display device of the one or more display devices and each receiving sensor data from the associated display device; and

a hub computing system operatively coupled to each of the one or more processing units, the hub computing device including a second set of sensors, the hub computing system and the one or more processing units collaboratively determining a three-dimensional map of the environment in which the system is used, based on data from the first and second sets of sensors.

2. The system of claim 1, the hub computing system and the one or more processing units collaboratively determining a three-dimensional map of the environment in real time.

3. The system of claim 1, the hub computing system providing computer models of the one or more users in the environment and the one or more processing units providing positions of display devices.

4. The system of claim 3, the model of a user and the position of the display device enabling determination by at least one of the hub computing system and a first processing unit of the one or more processing units of a field of view of the display device associated with the first processing unit.

5. The system of claim 4, the first set of sensors including an imaging camera and inertial sensors for providing positions of the display device associated with the first processing device

6. The system of claim 1, one of the hub computing system and a first processing unit of the one or more processing units determining a position of virtual three-dimensional object in a field of view of the display device associated with the first processing unit.

7. The system of claim 6, one of the hub computing system and the first processing unit further determining whether the virtual three-dimensional object is occluded by a real world object in a three-dimensional map of the environment, or whether the virtual three-dimensional object occludes the real world object in the map of the environment, from a first perspective of the display device associated with the first processing unit.

8. The system of claim 7, one of the hub computing system and a second processing unit of the one or more processing units further determining whether the virtual three-dimensional object is occluded by the real world object in the map of the environment, or whether the virtual three-dimensional object occludes the real world object in the map of the environment, from a second perspective of the display device associated with the second processing unit, the second perspective being different than the first perspective.

9. The system of claim 7, further comprising the steps of occluding the real world object by an opacity filter where the real world object is blocked by the virtual object from the first perspective of the display device associated with the first processing unit, and occluding the virtual world object in the map of the environment where the virtual object is blocked by the real world object from the first perspective of the display device associated with the first processing unit.

10. A system for presenting a mixed reality experience to one or more users, the system comprising:

a first head mounted display device including;

a camera for obtaining image data of an environment in which the first head mounted display device is used, inertial sensors for providing inertial measurements of the first head mounted display, and a display device for displaying virtual images to a user of the first head mounted display device, and

a first processing unit, associated with the first head mounted display, for determining a three-dimensional map of the environment in which the first head mounted display device is used and a field of view from which the first head mounted display device views the three-dimensional map.

11. The system of claim 10, wherein the user comprises a first user, the system further comprising:

a second head mounted display device including;

a camera for obtaining image data of an environment in which the second head mounted display device is used, inertial sensors for providing inertial measurements of the second head mounted display, and a display device for displaying virtual images to a second user of the second head mounted display device, and

a second processing unit, associated with the second head mounted display, for collaborating with the first processing unit in determining the three-dimensional map of the environment in which the first and second head mounted display devices are used, and a field of view from which the second head mounted display device views the three-dimensional map, the first and second field of views being different from each other.

12. The system of claim 10, wherein the first processing unit uses state information from the past and/or present to extrapolate the estimated state of a first head mounted display unit at a future time in order to present a rendered frame of image data to the user of the first head mounted display device.

13. The system of claim 12, where in the state information relates to a position of users, real world objects and virtual objects in the three-dimensional map of the environment, and to the user'"'"'s field of view of the three-dimensional map of the environment.

14. A method of presenting a mixed reality experience to one or more users, the system comprising:

(a) determining state information for at least two periods of time, the state information relating to a field of view of a user of an environment, the environment including a mixed reality of one or more real world objects and one or more virtual objects;

(b) extrapolating the state information relating to the field of view of the user of the environment for a third period of time, the third period of time being a time in the future when the one or more virtual objects of the mixed reality are to be displayed to the user; and

(c) displaying at least one virtual object of the one or more virtual objects to the user at the third period of time based on the state information relating to the field of view of the user extrapolated in said step (b).

15. The method of claim 14, said step (b) of extrapolating the state information relating to the field of view for the third period of time comprising the step of extrapolating state information relating to a perspective at which the field of view views the environment.

16. The method of claim 14, said step (b) of extrapolating the state information relating to the field of view for the third period of time comprising the step of extrapolating state information relating to positions of virtual objects within the field of view of the environment.

17. The method of claim 14, the user comprising a first user, the method further comprising the step of displaying the at least one virtual object to a second user different than the first user, from a different perspective than the at least one virtual object is displayed to the first user.

18. The method of claim 14, further comprising the step of collaboratively determining a map of the environment of the mixed reality, the map of the environment include three-dimensional position information relating to the one or more users, real world objects in the environment, and the one or more virtual objects in the environment.

19. The method of claim 18, said step of collaboratively determining a map of the environment comprising the step of determining the map of the environment by a portable processing unit associated with a head mounted display device on which the one or more virtual objects are displayed, and a hub computing system capable of generating computer models for tracking positions of the one or more users in the environment.

20. The method of claim 14, further comprising the steps:

(d) rendering the one or more virtual objects in an image frame for display once a frame, and

(e) re-rendering a portion of the image frame together with a newly rendered portion of the image frame once a frame, the newly rendered portion representing a view of the environment not rendered in said step (d).