Head-mounted device with active optical foveation

• US 10,642,049 B2
• Filed: 09/13/2018
• Issued: 05/05/2020
• Est. Priority Date: 04/25/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Huo, Edward S., Schmuck, David A., Sauers, Jason C., Gross, Kevin A.
• Primary Examiner(s): Patel, Sanjiv D.
• Application Number:
  US16/130,775
• Publication Number:
  US 20190331919A1
• Time in Patent Office: 600 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 26/0825 the reflecting element being a flexible sheet or membrane, e.g. for varying the focus flexible mirrors for cosmetic use A45D42/24
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/10 Beam splitting or combining systems polarising systems G02B27/28; mixing and splitting light signals using optical waveguides G02B6/28; beam shaping, e.g. changing the cross-sectional area, by dividing or superposing multiple light beams G02B27/0905
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/815 for controlling the resolution by using a single image
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout

Abstract

A pass-through camera in a head-mounted device may capture image data for displaying on a display in the head-mounted device. However, only low-resolution image data may be needed to display low-resolution images in the periphery of the user'"'"'s field of view on the display. Therefore, the pass-through camera may only capture high-resolution images that correspond to the portion of the user'"'"'s field-of-view that is being directly viewed and may capture lower resolution image data that corresponds to the real-world objects in the user'"'"'s peripheral vision. To enable the camera module to selectively capture high-resolution images, the pass-through camera may include an image sensor with two or more pixel densities, a distortion lens, and/or one or more planar or curved mirrors. Any of the components in the camera module may be adjusted to change which portion of a scene is captured with high-resolution image data.

Claims

1. A head-mounted device, comprising:

a head-mounted support structure;

a display that is supported by the head-mounted support structure and that is configured to display an image;

a camera module comprising an image sensor that is configured to capture the image based on incident light from real-world objects and an optical component that is interposed in an optical path of the incident light;

a gaze-tracking system that is configured to obtain point of gaze information; and

control circuitry that is configured to position the optical component based on the point of gaze information.

2. The head-mounted device defined in claim 1, wherein the camera module further comprises a lens, wherein the optical component is a mirror that redirects light from the lens to the image sensor, wherein the image sensor has a first portion with a first pixel density and a second portion with a second pixel density that is higher than the first pixel density, and wherein the control circuitry is configured to place the mirror in a position in which a portion of the incident light that corresponds to a point of gaze of a user is directed to the second portion of the image sensor.

3. The head-mounted device defined in claim 1, wherein the optical component is a planar mirror and wherein the control circuitry is configured to rotate the planar mirror based on the point of gaze information.

4. The head-mounted device defined in claim 1, wherein the optical component is a deformable mirror and wherein the control circuitry is configured to control a shape of the deformable mirror based on the point of gaze information.

5. The head-mounted device defined in claim 1, wherein the optical component is a lens.

6. The head-mounted device defined in claim 5, wherein the lens increases an angular resolution of a portion of the incident light.

7. The head-mounted device defined in claim 1, wherein the image sensor has a first portion with a first pixel density and a second portion with a second pixel density that is higher than the first pixel density.

8. The head-mounted device defined in claim 7, wherein positioning the optical component controls which portion of the incident light is directed to the second portion of the image sensor.

9. The head-mounted device defined in claim 8, wherein the control circuitry is configured to place the optical component in a position in which the portion of the incident light that corresponds to a point of gaze of a user is directed to the second portion of the image sensor.

10. The head-mounted device defined in claim 1, wherein the control circuitry is configured to place the optical component in a position in which the image sensor obtains high-resolution image data for a portion of the incident light that corresponds to a point of gaze of a user.

11. The head-mounted device defined in claim 1, further comprising:

positioning equipment configured to adjust a position of the optical component, wherein the control circuitry is configured to use the positioning equipment to position the optical component.

12. A head-mounted device, comprising:

a head-mounted support structure;

a camera module comprising an image sensor, wherein the image sensor is configured to generate first image data with a first resolution and second image data with a second resolution that is higher than the first resolution based on incident light from real-world objects;

a gaze-tracking system that is configured to obtain point of gaze information; and

positioning equipment that is configured to adjust the position of the image sensor based on the point of gaze information.

13. The head-mounted device defined in claim 12, wherein the image sensor has a first portion with a first pixel density and a second portion with a second pixel density that is higher than the first pixel density, wherein the first portion of the image sensor generates the first image data and the second portion of the image sensor generates the second image data, and wherein the positioning equipment is configured to place the image sensor in a position in which a portion of the incident light that corresponds to a point of gaze of a user is directed to the second portion of the image sensor.

14. The head-mounted device defined in claim 12, wherein the camera module further comprises a lens interposed on an optical path of the incident light.

15. The head-mounted device defined in claim 14, wherein the camera module further comprises a mirror interposed on the optical path of the incident light between the lens and the image sensor.

16. The head-mounted device defined in claim 12, wherein the image sensor has a first portion with a first pixel density and a second portion with a second pixel density that is higher than the first pixel density.

17. A head-mounted device, comprising:

a head-mounted support structure;

a camera module configured to capture an image of a real-world scene, wherein the image of the real-world scene has a first-resolution portion and a second-resolution portion that has a higher resolution than the first-resolution portion;

a display that is supported by the head-mounted support structure, wherein the display is configured to display the image of the real-world scene;

a gaze-tracking system that is supported by the head-mounted support structure and that obtains point of gaze information; and

positioning equipment that is configured to adjust a component in the camera module based on the point of gaze information, wherein adjusting the component adjusts which portion of the real-world scene corresponds to the second-resolution portion of the image of the real-world scene.

18. The head-mounted device defined in claim 17, wherein the component is an image sensor of the camera module, wherein the image sensor has a first portion with a first pixel density and a second portion with a second pixel density that is higher than the first pixel density, and wherein adjusting the component comprises moving the image sensor.

19. The head-mounted device defined in claim 17, wherein the component is a mirror of the camera module.

20. The head-mounted device defined in claim 17, wherein the component is a lens of the camera module and wherein adjusting the component comprises adjusting a shape of the lens.