Technologies for adjusting a perspective of a captured image for display

• US 9,934,573 B2
• Filed: 09/17/2014
• Issued: 04/03/2018
• Est. Priority Date: 09/17/2014
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Reif, Dror, Moran, Amit, Horesh, Nizan
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US14/488,516
• Publication Number:
  US 20160078680A1
• Time in Patent Office: 1,294 Days
• Field of Search: 345 8, 345156, 345158, 345633
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

Technologies for adjusting a perspective of a captured image for display on a mobile computing device include capturing a first image of a user by a first camera and a second image of a real-world environment by a second camera. The mobile computing device determines a position of an eye of the user relative to the mobile computing device based on the first captured image and a distance of an object in the real-world environment from the mobile computing device based on the second captured image. The mobile computing device generates a back projection of the real-world environment captured by the second camera to the display based on the determined distance of the object in the real-world environment relative to the mobile computing device, the determined position of the user'"'"'s eye relative to the mobile computing device, and at least one device parameter of the mobile computing device.

Claims

1. A mobile computing device for adjusting a perspective of a captured image for display, the mobile computing device comprising:

a permanently non-transparent display;

a camera system comprising a first camera and a second camera, the camera system to capture, while the user is holding the mobile computing device, (i) a first image of a user of the mobile computing device with the first camera and (i) a second image of a real-world environment of the mobile computing device with the second camera;

an eye tracking module to determine a position of an eye of the user relative to the mobile computing device based on the first captured image;

an object distance determination module to determine a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image; and

an image projection module to (i) determine an angular size of the mobile computing device from a perspective of the user, (ii) determine a distance of the object in the real-world environment relative to the user, (iii) determine a region of the object occluded by the permanently non-transparent display from the user'"'"'s perspective, (iv) determine a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the permanently non-transparent display and the distance of the object relative to the mobile computing device, and (v) generate, on the permanently non-transparent display of the mobile computing device, a back projection image of the real-world environment captured by the second camera to the permanently non-transparent display based on the corrected zoom magnitude, the determined position of the user'"'"'s eye relative to the mobile computing device, and at least one device parameter of the mobile computing device such that the back projection image is approximately equivalent to the view of the user of the real world that is occluded by the permanently non-transparent display.

2. The mobile computing device of claim 1, wherein to generate the back projection comprises to:

determine, for each display pixel of the permanently non-transparent display, a ray from the user'"'"'s eye through a corresponding display pixel to the object in the real-world environment;

identify, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and

construct a back projection image based on the identified image pixels for display on the permanent non-transparent display of the mobile computing device.

3. The mobile computing device of claim 1, wherein to determine the corrected zoom magnitude comprises to determine an angular size of a region of the object from a perspective of the second camera corresponding with the region of the object occluded by the mobile computing device from the user'"'"'s perspective.

4. The mobile computing device of claim 1, wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user'"'"'s perspective.

5. The mobile computing device of claim 4, wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera having only image pixels corresponding with features of the object from the region of the object occluded by the mobile computing device from the user'"'"'s perspective.

6. The mobile computing device of claim 1, wherein to:

determine the angular size of the mobile computing device from the user'"'"'s perspective comprises to determine an angular size of the mobile computing device from the user'"'"'s perspective based on a distance of the user'"'"'s eye relative to the mobile computing device and a size of the mobile computing device;

determine the distance of the object relative to the user comprises to determine the distance of the object relative to the user based on the distance of the user'"'"'s eye relative to the mobile computing device and the distance of the object relative to the mobile computing device; and

determine the region of the object occluded by the mobile computing device from the user'"'"'s perspective comprises to determine the angular size of the region of the object occluded by the mobile computing device based on the angular size of the mobile computing device from the user'"'"'s perspective and the distance of the object relative to the user.

7. The mobile computing device of claim 1, wherein angular size, δ

, is determined according to

8. The mobile computing device of claim 1, wherein to:

determine the position of the user'"'"'s eye relative to the mobile computing device comprises to determine a position of the user'"'"'s eye relative to the first camera; and

determine the distance of the object in the real-world environment relative to the mobile computing device comprises to determine a distance of the object relative to the second camera.

9. The mobile computing device of claim 1, wherein the first camera has a field of view in a direction opposite a field of view of the second camera about the permanently non-transparent display.

10. The mobile computing device of claim 1, wherein to determine the distance of the object in the real-world environment relative to the mobile computing device comprises to set a distance of the object relative to the mobile computing device to a predefined distance.

11. The mobile computing device of claim 1, further comprising a display module to display an image on the permanently non-transparent display based on the generated back projection of the real-world environment captured by the second camera.

12. The mobile computing device of claim 11, wherein to display the image based on the generated back projection comprises to display an image corresponding with the back projection modified to include augmented reality features.

13. The mobile computing device of claim 1, wherein the at least one device parameter comprises at least one of (i) a focal length of the second camera, (ii) a size of the permanently non-transparent display, (iii) a size of the mobile computing device, or (iv) a location of components of the mobile computing device relative to a reference point.

14. One or more non-transitory machine-readable storage media comprising a plurality of instructions stored thereon that, in response to execution by a mobile computing device, cause the mobile computing device to:

capture, by a first camera of the mobile computing device and while the user is holding the mobile computing device, a first image of a user of the mobile computing device;

determine a position of an eye of the user relative to the mobile computing device based on the first captured image;

capture, by a second camera of the mobile computing device different from the first camera, a second image of a real-world environment of the mobile computing device;

determine a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image;

determine an angular size of the mobile computing device from a perspective of the user based on a distance of the user'"'"'s eye relative to the mobile computing device and a size of the mobile computing device;

determine a distance of the object in the real-world environment relative to the user based on the distance of the user'"'"'s eye relative to the mobile computing device and the distance of the object relative to the mobile computing device;

determine a region of the object occluded by a permanently non-transparent display of the mobile computing device from the user'"'"'s perspective based on the angular size of the mobile computing device from the user'"'"'s perspective and the distance of the object relative to the user; and

determine a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the permanently non-transparent display and the distance of the object relative to the mobile computing device; and

generate, on the permanently non-transparent display of the mobile computing device, a back projection image of the real-world environment captured by the second camera to the permanently non-transparent display of the mobile computing device based on the corrected zoom magnitude, the determined position of the user'"'"'s eye relative to the mobile computing device, and at least one device parameter of the mobile computing device such that the back projection image is approximately equivalent to the view of the user of the real world that is occluded by the permanently non-transparent display.

15. The one or more non-transitory machine-readable storage media of claim 14, wherein to generate the back projection comprises to:

determine, for each display pixel of the permanently non-transparent display, a ray from the user'"'"'s eye through a corresponding display pixel to the object in the real-world environment;

identify, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and

construct a back projection image based on the identified image pixels for display on the permanently non-transparent display of the mobile computing device.

16. The one or more non-transitory machine-readable storage media of claim 14, wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user'"'"'s perspective.

17. The one or more non-transitory machine-readable storage media of claim 14, wherein to:

determine the position of the user'"'"'s eye relative to the mobile computing device comprises to determine a position of the user'"'"'s eye relative to the first camera; and

determine the distance of the object in the real-world environment relative to the mobile computing device comprises to determine a distance of the object relative to the second camera.

18. The one or more non-transitory machine-readable storage media of claim 14, wherein to determine the distance of the object in the real-world environment relative to the mobile computing device comprises to set a distance of the object relative to the mobile computing device to a predefined distance.

19. The one or more non-transitory machine-readable storage media of claim 14, wherein the plurality of instructions further cause the mobile computing device to display an image on the permanently non-transparent display based on the generated back projection of the real-world environment captured by the second camera.

20. The one or more non-transitory machine-readable storage media of claim 14, wherein the at least one device parameter comprises at least one of (i) a focal length of the second camera, (ii) a size of the permanently non-transparent display, (iii) a size of the mobile computing device, or (iv) a location of components of the mobile computing device relative to a reference point.

21. A method for adjusting a perspective of a captured image for display on a mobile computing device, the method comprising:

capturing, by a first camera of the mobile computing device and while the user is holding the mobile computing device, a first image of a user of the mobile computing device;

determining, by the mobile computing device, a position of an eye of the user relative to the mobile computing device based on the first captured image;

capturing, by a second camera of the mobile computing device different from the first camera, a second image of a real-world environment of the mobile computing device;

determining, by the mobile computing device, a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image;

determining an angular size of the mobile computing device from a perspective of the user based on a distance of the user'"'"'s eye relative to the mobile computing device and a size of the mobile computing device;

determining a distance of the object in the real-world environment relative to the user based on the distance of the user'"'"'s eye relative to the mobile computing device and the distance of the object relative to the mobile computing device;

determining a region of the object occluded by a permanently non-transparent display of the mobile computing device from the user'"'"'s perspective based on the angular size of the mobile computing device from the user'"'"'s perspective and the distance of the object relative to the user; and

determining a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the permanently non-transparent display and the distance of the object relative to the mobile computing device;

generating, by the mobile computing device on the permanently non-transparent display of the mobile computing device, a back projection image of the real world environment captured by the second camera to the permanently non-transparent display of the mobile computing device based on the corrected zoom magnitude, the determined position of the user'"'"'s eye relative to the mobile computing device, and at least one device parameter of the mobile computing device such that the back projection image is approximately equivalent to the view of the user of the real world that is occluded by the permanently non-transparent display.

22. The method of claim 21, wherein generating the back projection comprises:

determining, for each display pixel of the permanently non-transparent display, a ray from the user'"'"'s eye through a corresponding display pixel to the object in the real-world environment;

identifying, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and

constructing a back projection image based on the identified image pixels for display on the permanently non-transparent display of the mobile computing device.

23. The method of claim 21, wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user'"'"'s perspective.