LOW-LATENCY FUSING OF COLOR IMAGE DATA

• US 20140176591A1
• Filed: 12/26/2012
• Published: 06/26/2014
• Est. Priority Date: 12/26/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Klein, Georg, Osborne, Timothy R., Service, Douglas Kevin, Silva, Bruno, Michail, Ashraf Ayman, Wong, Tuan, Mehta, Sujeet, Tomlin, Arthur C.
• Application Number:
  US13/727,040
• Publication Number:
  US 20140176591A1
• Time in Patent Office: Days
• US Class Current: 345/589
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G09G 2310/0235 Field-sequential colour display
• CPC Class Codes: G09G 2320/0242 Compensation of deficiencies in the appearance of colours
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence

Abstract

A system and method are disclosed for fusing virtual content with real content to provide a mixed reality experience for one or more users. The system includes a mobile display device communicating with a hub computing system. In examples, the mobile display device includes a color sequential display for displaying an image over a number of color channels. Image data on respective color channels may be adjusted based on a predicted position of the mobile display device at a time the sequential color display projects the image.

Claims

1. A system for presenting a mixed reality experience, the system comprising:

a display device including a color sequential display for projecting a virtual object in two or more color channels;

a sensor for sensing positions of the display device; and

one or more processors for determining image data for the virtual object on each of the two or more color channels for the color sequential display to project the virtual object, the one or more processors predicting a position of the display device at one or more times the color sequential display is to project the virtual object based on input from the sensor, the one or more processors adjusting the image data for the virtual object on first and second color channels of the two or more color channels, independently of each other, based on the predicted position of the display device at the one or more times the color sequential display is to display the virtual object.

2. The system of claim 1, the one or more processors adjusting the image data for the first and second color channels to align with each other at the one or more times the color sequential display is to display the two or more color channels.

3. The system of claim 1, the one or more processors applying a transform to the image data to adjust the image data based on the predicted position of the sensor at the one or more times the color sequential display is to display the two or more color channels.

4. The system of claim 3, the applied transform being one of an integer pixel offset transform, an affine transform, a homography transform and a meshed-based warping algorithm.

5. The system of claim 1, the one or more processors performing the prediction of the position of the display device two or more times between a first refresh of the projected image by the color sequential display and a second immediately subsequent refresh of the projected image by the color sequential display.

6. The system of claim 1, the one or more processors adjusting the image data for each of the first and second color channels two or more times between a first refresh of the projected image by the color sequential display and a second immediately subsequent refresh of the projected image by the color sequential display.

7. The system of claim 1, wherein the sensor comprises an image capture device sensing a position of the display device.

8. The system of claim 1, wherein the sensor comprises an inertial measurement unit sensing movement of the display device.

9. The system of claim 1, wherein the virtual object is registered to a real world object in a field of view of the display device.

10. A system for presenting a mixed reality experience, the system comprising:

a head mounted display device including a color sequential display for projecting a virtual object using first, second and third color channels;

a plurality of sensors for sensing positions of the display device, the plurality of sensors comprising an inertial measurement unit for sensing movement of the head mounted display device and at least one image capture device; and

one or more processors for determining a three-dimensional map of the environment in which the system is used based on data from the plurality of sensors, the one or more processors rendering the first, second and third color channels at a time t₁, the one or more processors predicting a position of the display device at a time t₂, after t₁, when the color sequential display is to display the first color channel, the one or more processors predicting a position of the display device at a time t₃, after t₂, when the color sequential display is to display the second color channel, the one or more processors predicting a position of the display device at a time t₄, after t₃, when the color sequential display is to display the third color channel, the one or more processors adjusting the image data for the virtual object on the first, second and third color channels, independently of each other, based on the predicted position of the display device at the times t₂, t₃ and t₄.

11. The system of claim 10, the virtual object comprising a first virtual object, the one or more processors determining a position at which to project a second virtual object, the one or more processors not adjusting the determined position of the image data for the second virtual object prior to display of the first and second virtual objects.

12. The system of claim 11, wherein the first virtual object is one of a scene-locked virtual object and a dynamic virtual object, and the second virtual object is a head-locked virtual object.

13. The system of claim 10, wherein data indicating the adjustment of the image data for the first, second and third color channels is encoded into initial pixels of the images for the first, second and third color channels.

14. The system of claim 10, the one or more processors adjusting the image data for the first, second and third color channels to align with each other when the respective color channels are displayed at the times t₂, t₃ and t₄.

15. The system of claim 10, the one or more processors applying one of an integer pixel offset transform, an affine transform, a homography transform and a meshed-based warping algorithm to adjust the image data of the respective color channels based on the predicted positions of the sensor at the times t₂, t₃ and t₄.

16. A method of displaying an image using a display device including a color sequential display, the method comprising:

(a) determining a view of the display device;

(b) rendering image data based on the view determined in said step (a);

(c) predicting an updated view of the display device after said step (a); and

(d) applying one or more transforms to image data for color channels of the color sequential display to adjust the image data for the color channels based on the updated view predicted in said step (c), image data for a first color channel adjusted differently than image data for a second color channel.

17. The method of claim 16, said step (c) of predicting the updated view of the display device comprising the step of predicting the updated view of the display device at a time when the display device is to display the image from a color channel.

18. The method of claim 16, said step (c) of predicting the updated view of the display device comprising the step of using previous positions of the display device and movement of the display device to extrapolate a position of the display device at a time when the display device is to display the image from a color channel.

19. The method of claim 16, said step (d) of applying one or more transforms to adjust the image data for the respective color channels so that the color channels display images at one or more times that fuse into a single cohesive color image.

20. The method of claim 16, said step (b) of rendering the image comprises the step of rendering virtual objects for display on a display in a system for presenting a mixed reality experience.