Low-latency fusing of virtual and real content

• US 9,348,141 B2
• Filed: 10/27/2010
• Issued: 05/24/2016
• Est. Priority Date: 10/27/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bar-Zeev, Avi, Goossen, J. Andrew, Tardif, John, Grossman, Mark S., Singh, Harjit
• Primary Examiner(s): Hoang, Phi
• Assistant Examiner(s): Shin, Andrew
• Application Number:
  US12/912,937
• Publication Number:
  US 20120105473A1
• Time in Patent Office: 2,036 Days
• Field of Search: 345/419, 345/421, 345/422, 345/427, 345/581, 345/619, 345629-641, 382/103, 382/284, 382/291, 382/293, 382/294, 715/733, 715/734, 715/751, 715/753, 715/757, 715/764, 715/766, 715848-851, 709/204, 709/205, 709/248
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels

Abstract

A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content. In one embodiment, the processing unit is in communication with a hub computing device. The processing unit and hub may collaboratively determine a map of the mixed reality environment. Further, state data may be extrapolated to predict a field of view for a user in the future at a time when the mixed reality is to be displayed to the user. This extrapolation can remove latency from the system.

Claims

1. A system for presenting a mixed reality experience to a user, the system comprising:

a display device including a first set of sensors for sensing data relating to a position of the display device and data relating to positions of real world objects in a field of view of the display device, the display device further including a display unit for displaying a virtual image to the user of the display device;

a processing unit associated with the display device receiving sensor data from the associated display device, the processing unit receiving the data relating to positions of the real world objects in the field of view at a present time within a video frame, and the processing unit extrapolating a position of a virtual three-dimensional object at a future time in the same video frame, the virtual three-dimensional object positioned at least partially behind a position of a real world object, the position extrapolated to the future time to display a proper occlusion of the virtual object by the real world object at the future time when the virtual object is displayed by the display unit.

2. The system of claim 1, the display device and the processing unit collaboratively determining a three-dimensional map of the environment in real time.

3. The system of claim 1, the processing unit providing computer models of the one or more users in the environment and the positions of display devices.

4. The system of claim 3, the model of the user and the position of the display device enabling determination by the processing unit of the field of view of the display device.

5. The system of claim 4, the first set of sensors including an imaging camera and inertial sensors for providing the positions of the display device associated with the processing device.

6. The system of claim 1, the processing unit further determining a degree by which the extrapolated position of the virtual three-dimensional object is occluded by the real world object in a three-dimensional map of the environment from a perspective of the display device associated with the processing unit.

7. A method of presenting a mixed reality experience to one or more users, the system comprising:

(a) determining state information for at least a first period of time from a first video frame, and a second period of time from a second video frame subsequent to the first video frame, the state information relating to a field of view of a user of the one or more users of an environment, the environment including a mixed reality of one or more real world objects and one or more virtual objects;

(b) extrapolating the state information relating to the field of view of the user of the environment for a third period of time in the second video frame, the third period of time being a time in the second video frame, after the second period of time, when the one or more virtual objects of the mixed reality are to be displayed to the user; and

(c) displaying at least one virtual object of the one or more virtual objects to the user at the third period of time based on the state information relating to the field of view of the user extrapolated in said step (b), the at least one virtual object displayed as being partially occluded behind a detected real world object, the extrapolated state information at the third period of time enabling a seamless display of the partially occluded virtual object behind the real world object.

8. The method of claim 7, said step (b) of extrapolating the state information relating to the field of view for the third period of time comprising the step of extrapolating state information relating to a perspective at which the field of view views the environment.

9. The method of claim 7, said step (b) of extrapolating the state information relating to the field of view for the third period of time comprising the step of extrapolating state information relating to positions of virtual objects within the field of view of the environment.

10. The method of claim 7, the user comprising a first user, the method further comprising the step of displaying the at least one virtual object to a second user different than the first user, from a different perspective than the at least one virtual object is displayed to the first user.

11. The method of claim 7, further comprising the step of collaboratively determining a map of the environment of the mixed reality, the map of the environment include three-dimensional position information relating to the one or more users, real world objects in the environment, and the one or more virtual objects in the environment.

12. The method of claim 11, said step of collaboratively determining a map of the environment comprising the step of determining the map of the environment by a portable processing unit associated with a head mounted display device on which the one or more virtual objects are displayed.

13. The method of claim 7, further comprising the steps:

(d) rendering the one or more virtual objects in an image frame for display once a frame, and

(e) re-rendering a portion of the image frame together with a newly rendered portion of the image frame once a frame, the newly rendered portion representing a view of the environment not rendered in said step (d).