Augmented reality device to warp images

• US 10,347,050 B1
• Filed: 09/22/2017
• Issued: 07/09/2019
• Est. Priority Date: 09/22/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Wang, Paul X., Chang, Ray
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US15/713,246
• Time in Patent Office: 655 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually

Abstract

Displaying a warped area includes obtaining an image of a real environment, obtaining a portion of a field of view comprising an obstruction, determining a portion of the image of the real environment corresponding to the obstruction, applying a warping function to an area surrounding the portion of the image of the real environment corresponding to the obstruction to obtain a warped area, and displaying the warped area.

Claims

1. A method for displaying a warped area, comprising:

obtaining an image of a real environment captured by a front facing camera of a device;

determining a portion of a field of view of the real environment for a user comprising an obstruction based on an optical occlusion of an eye of the user;

determining a portion of the image of the real environment corresponding to the obstruction, comprising;

determining a relationship between the eye and a front-facing camera;

determining a relationship between the optical occlusion and the eye; and

determining the portion of the field of view comprising the obstruction based on the relationship between the eye and the front-facing camera, the relationship between the optical occlusion and the eye, and a gaze vector of the eye;

applying a warping function to an area of the image surrounding the portion of the image of the real environment corresponding to the obstruction to obtain a warped area; and

displaying the warped area,

wherein the portion of the field of view comprising the obstruction is presented in the warped area.

2. The method of claim 1, wherein an image of the eye is captured by a back-facing camera of the device, and wherein the image of the eye is utilized in determining the relationship between the eye and the front facing camera.

3. The method of claim 1, wherein displaying the warped area comprises:

modifying the image of the real environment to include the warped area;

wherein displaying the warped area comprises displaying the modified image of the real environment on a display device.

4. The method of claim 1, wherein displaying the warped area comprises displaying the warped area as an overlay over the real environment on a transparent display device.

5. The method of claim 1, wherein displaying the warped area comprises projecting the warped area onto a retina of an eye.

6. A system, comprising:

one or more cameras;

one or more processors; and

a memory coupled to the one or more processors and comprising instructions executable by the one or more processors to cause the system to;

obtain an image of a real environment from a front facing camera of the one or more cameras;

determine a relationship between the eye and the front facing camera;

determine a relationship between the optical occlusion and the eye;

determine a portion of a field of view for a user comprising an obstruction based on the relationship between the eye and the front-facing camera, the relationship between the optical occlusion and the eye, and a gaze vector of the eye;

determine a portion of the image of the real environment corresponding to the obstruction;

applying a warping function to an area of the image surrounding the portion of the image of the real environment corresponding to the obstruction to obtain a warped area; and

displaying the warped area,

wherein the portion of the field of view comprising the obstruction is presented in the warped area.

7. The system of claim 6, wherein the image of the eye is captured by a back-facing camera of the one of more cameras, and wherein the image of the eye is utilized in determining the relationship between the eye and the front facing camera.

8. The system of claim 6, wherein the instructions to display the warped area comprise instructions to:

modify the image of the real environment to include the warped area,

wherein the instructions to display the warped area comprise instructions to display the modified image of the real environment on a display device.

9. The system of claim 6, wherein the instructions to display the warped area comprise instructions to display the warped area as an overlay over the real environment on a transparent display device.

10. The system of claim 6, wherein the instructions to display the warped area comprises instructions to project the warped area onto a retina of an eye.

11. A non-transitory machine readable medium comprising instructions executable by one or more processors to:

obtain an image of a real environment captured by a front facing camera of a device;

determine a relationship between the eye and the front facing camera;

determine a relationship between the optical occlusion and the eye;

determine a portion of a field of view for a user comprising an obstruction based on the relationship between the eye and the front-facing camera, the relationship between the optical occlusion and the eye, and a gaze vector of the eye;

determine a portion of the image of the real environment corresponding to the obstruction;

apply a warping function to an area of the image surrounding the portion of the image of the real environment corresponding to the obstruction to obtain a warped area; and

display the warped area,

wherein the portion of the field of view comprising the obstruction is presented in the warped area.

12. The non-transitory machine readable medium of claim 11, wherein an image of the eye is captured by a back-facing camera of the device, and wherein the image of the eye is utilized in determining the relationship between the eye and the front facing camera.

13. The non-transitory machine readable medium of claim 11, wherein the instructions to display the warped area comprise instructions to:

modify the image of the real environment to include the warped area,

wherein the instructions to display the warped area comprise instructions to display the modified image of the real environment on a display device.

14. The non-transitory machine readable medium of claim 11, wherein the instructions to display the warped area comprise instructions to display the warped area as an overlay over the real environment on a transparent display device.

15. The non-transitory machine readable medium of claim 11, wherein the instructions to display the warped area comprises instructions to project the warped area onto a retina of an eye.