Augmented-reality imaging

• US 10,127,725 B2
• Filed: 09/02/2015
• Issued: 11/13/2018
• Est. Priority Date: 09/02/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kohler, Jeffrey, Demandolx, Denis, Guyman, Will, Michail, Ashraf Ayman, Park, Minshik, Nafziger, Justin, Richards, Eric
• Primary Examiner(s): Mazumder, Tapas
• Application Number:
  US14/843,709
• Publication Number:
  US 20170061693A1
• Time in Patent Office: 1,168 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A two-dimensional augmentation image is rendered from a three-dimensional model from a first virtual perspective. A transformation is applied to the augmentation image to yield an updated two-dimensional augmentation image that approximates a second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model.

Claims

1. On an augmented-reality device, a method of providing a mixed-reality experience, the method comprising:

rendering from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a first real-world location;

outputting the two-dimensional augmentation image;

applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective of the virtual camera spatially registered to a second real-world location different than the first real-world location such that the second virtual perspective is different than the first virtual perspective; and

outputting the updated two-dimensional augmentation image.

2. The method of claim 1, wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image as a layer of a mixed-reality recording, the mixed-reality recording including a second layer including a visible-light image captured by a point-of-view camera.

3. The method of claim 2, wherein the transformation is selected so that the updated two-dimensional augmentation image has a virtual perspective simulating a real-world perspective of the point-of-view camera at the second real-world location when the visible-light image is captured.

4. The method of claim 2, further comprising:

cropping a larger of the updated two-dimensional augmentation image and the visible-light image to match a smaller of the updated two-dimensional augmentation image and the visible light image in the mixed-reality recording.

5. The method of claim 1, wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image to a display of the augmented-reality device for visual presentation of the updated two-dimensional augmentation image by the display.

6. The method of claim 5, wherein the display is an at least partially see-through display, wherein the updated two-dimensional augmentation image is visually presented on the at least partially see-through display while light from a physical space passes through the at least partially see-through display to a user eye.

7. The method of claim 5, wherein the updated two-dimensional augmentation image has dimensions that are larger than dimensions of a field of view of the display, and the method further comprises, cropping the updated two-dimensional augmentation image.

8. The method of claim 1, wherein applying a transformation to the augmentation image includes,

for each pixel of the augmentation image, passing a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,

for each virtual position of the three-dimensional transformation model, mapping the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and

for each updated virtual position in the updated three-dimensional transformation model, passing the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.

9. The method of claim 8, wherein the method further comprises:

if two or more updated virtual positions map to a same pixel, selecting an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.

10. An augmented-reality device, comprising:

an at least partially see-through display;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

render from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a first real-world location;

output the two-dimensional augmentation image;

applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective of the virtual camera spatially registered to a second real-world location different than the first real-world location such that the second virtual perspective is different than the first virtual perspective; and

output the updated two-dimensional augmentation image.

11. The augmented-reality device of claim 10, wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image as a layer of a mixed-reality recording, the mixed-reality recording including a second layer including a visible-light image captured by a point-of-view camera.

12. The augmented-reality device of claim 11, wherein the transformation is selected so that the updated two-dimensional augmentation image has a virtual perspective simulating a real-world perspective of the point-of-view camera at the second real-world location when the visible-light image is captured.

13. The augmented-reality device of claim 11, wherein the storage machine further holds instructions executable by the logic machine to:

crop a larger of the updated two-dimensional augmentation image and the visible-light image to match a smaller of the updated two-dimensional augmentation image and the visible light image in the mixed-reality recording.

14. The augmented-reality device of claim 10, wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image to the at least partially see-through display, and wherein the updated two-dimensional augmentation image is visually presented on the at least partially see-through display while light from a physical space passes through the at least partially see-through display to a user eye.

15. The augmented-reality device of claim 14, wherein the updated two-dimensional augmentation image has dimensions that are larger than dimensions of a field of view of the display, and wherein the storage machine further holds instructions executable by the logic machine to:

crop the updated two-dimensional augmentation image.

16. The augmented-reality device of claim 10, wherein applying a transformation to the augmentation image includes,

for each pixel of the augmentation image, pass a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,

for each virtual position of the three-dimensional transformation model, map the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and

for each updated virtual position in the updated three-dimensional transformation model, pass the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.

17. The augmented-reality device of claim 16, wherein the storage machine further holds instructions executable by the logic machine to:

if two or more updated virtual positions map to a same pixel, select an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.

18. On an augmented-reality device, a method of providing a mixed-reality experience, the method comprising:

rendering from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a real-world perspective of an at least partially see-through display of the augmented reality device;

outputting the two-dimensional augmentation image to the at least partially see-through display;

obtaining, via a point-of-view camera of the augmented-reality device, a visible-light image of a physical space;

applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective different than the first virtual perspective of the virtual camera and corresponding to a real-world perspective of the point-of-view camera when the visible-light image is captured, wherein applying the transformation to the augmentation image includes,

for each pixel of the augmentation image, passing a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,

for each virtual position of the three-dimensional transformation model, mapping the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and

for each updated virtual position in the updated three-dimensional transformation model, passing the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel;

combining the updated two-dimensional augmentation image and the visible light image in a mixed reality recording; and

outputting the mixed reality recording.

19. The method of claim 18, wherein the method further comprises:

if two or more updated virtual positions map to a same pixel, selecting an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.