Optimizing head mounted displays for augmented reality

• US 10,043,319 B2
• Filed: 01/13/2017
• Issued: 08/07/2018
• Est. Priority Date: 11/16/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Eonite Perception, Inc. (Intel Corporation)
• Inventors: Petrovskaya, Anna, Varvak, Peter
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US15/406,652
• Publication Number:
  US 20170206712A1
• Time in Patent Office: 571 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

While many augmented reality systems provide “see-through” transparent or translucent displays upon which to project virtual objects, many virtual reality systems instead employ opaque, enclosed screens. Indeed, eliminating the user'"'"'s perception of the real-world may be integral to some successful virtual reality experiences. Thus, head mounted displays designed exclusively for virtual reality experiences may not be easily repurposed to capture significant portions of the augmented reality market. Various of the disclosed embodiments facilitate the repurposing of a virtual reality device for augmented reality use. Particularly, by anticipating user head motion, embodiments may facilitate scene renderings better aligned with user expectations than naïve renderings generated within the enclosed field of view. In some embodiments, the system may use procedural mapping methods to generate a virtual model of the environment. The system may then use this model to supplement the anticipatory rendering.

Claims

1. A computer system for causing a head mounted display (HMD) to render an augmented reality image, comprising:

at least one processor; and

at least one memory comprising instructions configured to cause the at least one processor to perform a method comprising;

receiving visual data associated with a first orientation of the HMD;

receiving depth frame data associated with the first orientation of the HMD;

receiving IMU data associated with the first orientation of the HMD;

receiving localization results associated with the first orientation of the HMD;

determining a future rendering time when the HMD will present an image to a user;

determining a second, predicted orientation associated with the HMD at the future rendering time based upon the IMU data, the localization results, and the future rendering time;

generating a three-dimensional (3D) mesh from the depth frame data associated with the first orientation and specifying the texture for the 3D mesh from the visual data associated with the first orientation;

determining a transform between the first orientation and the second orientation;

applying the transform to the generated 3D mesh; and

causing at least a portion of the resulting textured 3D mesh to be rendered in the HMD to the user as a final image.

2. The computer system of claim 1, wherein the method further comprises determining pixel stretch distances based upon vertices of the transformed 3D mesh.

3. The computer system of claim 2, wherein the method further comprises blurring pixels in the final image based upon the pixel stretch distances.

4. The computer system of claim 3, wherein the method further comprises creating parts of the final image from at least a portion of a previously created model of an environment surrounding the HMD based upon the pixel stretch distances.

5. The computer system of claim 4, the method further comprising creating the model using a Scaling Series mapping process.

6. The computer system of claim 1, wherein the method further comprises computing Nearest Visible Projected Pixel (NVPP) distances based upon vertices of the transformed 3D mesh.

7. The computer system of claim 6, wherein the method further comprises causing pixels of the final image to be blurred based upon NVPP distances.

8. The computer system of claim 1, wherein the method further comprises reconstructing parts of the final image from at least a portion of a previously created model of an environment surrounding the HMD based upon the NVPP distances.

9. A computer-implemented method for rendering an augmented reality image in a head mounted display (HMD), comprising:

receiving visual data associated with a first orientation of the HMD;

receiving depth frame data associated with the first orientation of the HMD;

receiving IMU data associated with the first orientation of the HMD;

receiving localization results associated with the first orientation of the HMD;

determining a future rendering time when the HMD will present an image to a user;

determining a second, predicted orientation associated with the HMD at the future rendering time based upon the IMU data, the localization results, and the future rendering time;

generating a three-dimensional (3D) mesh from the depth frame data associated with the first orientation and specifying the texture for the 3D mesh from the visual data associated with the first orientation;

determining a transform between the first orientation and the second orientation;

applying the transform to the generated 3D mesh; and

causing at least a portion of the resulting textured 3D mesh to be rendered in the HMD to the user as a final image.

10. The computer-implemented method of claim 9, wherein the method further comprises determining pixel stretch distances based upon vertices of the transformed 3D mesh.

11. The computer-implemented method of claim 10, wherein the method further comprises blurring pixels in the final image based upon the pixel stretch distances.

12. The computer-implemented method of claim 11, wherein the method further comprises creating parts of the final image from at least a portion of a previously created model of an environment surrounding the HMD based upon the pixel stretch distances.

13. The computer-implemented method of claim 12, wherein the method further comprises creating the model using a Scaling Series mapping process.

14. The computer-implemented method of claim 9, wherein the method further comprises computing Nearest Visible Projected Pixel (NVPP) distances based upon vertices of the transformed 3D mesh.

15. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method for rendering an augmented reality image in a head mounted display (HMD), comprising:

receiving visual data associated with a first orientation of the HMD;

receiving depth frame data associated with the first orientation of the HMD;

receiving IMU data associated with the first orientation of the HMD;

receiving localization results associated with the first orientation of the HMD;

determining a future rendering time when the HMD will present an image to a user;

determining a second, predicted orientation associated with the HMD at the future rendering time based upon the IMU data, the localization results, and the future rendering time;

generating a three-dimensional (3D) mesh from the depth frame data associated with the first orientation and specifying the texture for the 3D mesh from the visual data associated with the first orientation;

determining a transform between the first orientation and the second orientation;

applying the transform to the generated 3D mesh; and

causing at least a portion of the resulting textured 3D mesh to be rendered in the HMD to the user as a final image.

16. The non-transitory computer-readable medium of claim 15, wherein the method further comprises determining pixel stretch distances based upon vertices of the transformed 3D mesh.

17. The non-transitory computer-readable medium of claim 16, wherein the method further comprises blurring pixels in the final image based upon the pixel stretch distances.

18. The non-transitory computer-readable medium of claim 17, wherein the method further comprises creating parts of the final image from at least a portion of a previously created model of an environment surrounding the HMD based upon the pixel stretch distances.

19. The non-transitory computer-readable medium of claim 18, wherein the method further comprises creating the model using a Scaling Series mapping process.

20. The non-transitory computer-readable medium of claim 15, wherein the method further comprises computing Nearest Visible Projected Pixel (NVPP) distances based upon vertices of the transformed 3D mesh.