IMAGE CAPTURE SYSTEMS, DEVICES, AND METHODS THAT AUTOFOCUS BASED ON EYE-TRACKING

• US 20180007255A1
• Filed: 06/30/2017
• Published: 01/04/2018
• Est. Priority Date: 06/30/2016
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Thalmic Labs Inc. (Alphabet Inc.)
• Inventors: Tang, Sui Tong
• Application Number:
  US15/639,371
• Publication Number:
  US 20180007255A1
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation

Abstract

Image capture systems, devices, and methods that automatically focus on objects in the user'"'"'s field of view based on where the user is looking/gazing are described. The image capture system includes an eye tracker subsystem in communication with an autofocus camera to facilitate effortless and precise focusing of the autofocus camera on objects of interest to the user. The autofocus camera automatically focuses on what the user is looking at based on gaze direction determined by the eye tracker subsystem and one or more focus property(ies) of the object, such as its physical distance or light characteristics such as contrast and/or phase. The image capture system is particularly well-suited for use in a wearable heads-up display to capture focused images of objects in the user'"'"'s field of view with minimal intervention from the user.

Claims

1. An image capture system comprising:

an eye tracker subsystem to sense at least one feature of an eye of a user and to determine a gaze direction of the eye of the user based on the at least one feature; and

an autofocus camera communicatively coupled to the eye tracker subsystem, the autofocus camera to automatically focus on an object in a field of view of the eye of the user based on the gaze direction of the eye of the user determined by the eye tracker subsystem.

2. The image capture system of claim 1 wherein the autofocus camera includes:

an image sensor having a field of view that at least partially overlaps with the field of view of the eye of the user;

a tunable optical element positioned and oriented to tunably focus on the object in the field of view of the image sensor; and

a focus controller communicatively coupled to the tunable optical element, the focus controller communicatively coupled to apply adjustments to the tunable optical element to focus the field of view of the image sensor on the object in the field of view of the eye of the user based on both the gaze direction of the eye of the user determined by the eye tracker subsystem and a focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera.

3. The image capture system of claim 2, further comprising:

a processor communicatively coupled to both the eye tracker subsystem and the autofocus camera; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable data and/or instructions that, when executed by the processor, cause the processor to effect a mapping between the gaze direction of the eye of the user determined by the eye tracker subsystem and the focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera.

4. The image capture system of claim 2 wherein the autofocus camera includes a focus property sensor to determine the focus property of at least a portion of the field of view of the image sensor, the focus property sensor selected from a group consisting of:

a distance sensor to sense distances to objects in the field of view of the image sensor;

a time of flight sensor to determine distances to objects in the field of view of the image sensor;

a phase detection sensor to detect a phase difference between at least two points in the field of view of the image sensor; and

a contrast detection sensor to detect an intensity difference between at least two points in the field of view of the image sensor.

5. The image capture system of claim 1, further comprising:

a processor communicatively coupled to both the eye tracker subsystem and the autofocus camera; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable data and/or instructions that, when executed by the processor, cause the processor to control an operation of at least one of the eye tracker subsystem and/or the autofocus camera.

6. The image capture system of claim 5 wherein the eye tracker subsystem includes:

an eye tracker to sense the at least one feature of the eye of the user; and

processor-executable data and/or instructions stored in the non-transitory processor-readable storage medium, the processor-executable data and/or instructions which, when executed by the processor, cause the processor to determine the gaze direction of the eye of the user based on the at least one feature of the eye of the user sensed by the eye tracker.

7. The image capture system of claim 1 wherein the at least one feature of the eye of the user sensed by the eye tracker subsystem is selected from a group consisting of:

a position of a pupil of the eye of the user, an orientation of a pupil of the eye of the user, a position of a cornea of the eye of the user, an orientation of a cornea of the eye of the user, a position of an iris of the eye of the user, an orientation of an iris of the eye of the user, a position of at least one retinal blood vessel of the eye of the user, and an orientation of at least one retinal blood vessel of the eye of the user.

8. The image capture system of claim 1, further comprising:

a support structure that in use is worn on a head of the user, wherein both the eye tracker subsystem and the autofocus camera are carried by the support structure.

9. A method of operation of an image capture system, wherein the image capture system includes an eye tracker subsystem and an autofocus camera, the method comprising:

sensing at least one feature of an eye of a user by the eye tracker subsystem;

determining a gaze direction of the eye of the user based on the at least one feature by the eye tracker subsystem; and

focusing on an object in a field of view of the eye of the user by the autofocus camera based on the gaze direction of the eye of the user determined by the eye tracker subsystem.

10. The method of claim 9 wherein sensing at least one feature of an eye of the user by the eye tracker subsystem includes at least one of:

sensing a position of a pupil of the eye of the user by the eye tracker subsystem;

sensing an orientation of a pupil of the eye of the user by the eye tracker subsystem;

sensing a position of a cornea of the eye of the user by the eye tracker subsystem;

sensing an orientation of a cornea of the eye of the user by the eye tracker subsystem;

sensing a position of an iris of the eye of the user by the eye tracker subsystem;

sensing an orientation of an iris of the eye of the user by the eye tracker subsystem;

sensing a position of at least one retinal blood vessel of the eye of the user by the eye tracker subsystem;

or

sensing an orientation of at least one retinal blood vessel of the eye of the user by the eye tracker subsystem.

11. The method of claim 9 wherein the image capture system further includes:

a processor communicatively coupled to both the eye tracker subsystem and the autofocus camera; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, and wherein the non-transitory processor-readable storage medium stores processor-executable data and/or instructions, the method further comprising;

executing the processor-executable data and/or instructions by the processor to cause the autofocus camera to focus on the object in the field of view of the eye of the user based on the gaze direction of the eye of the user determined by the eye tracker subsystem.

12. The method of claim 11 wherein the autofocus camera includes an image sensor, a tunable optical element, and a focus controller communicatively coupled to the tunable optical element, the method further comprising:

determining a focus property of at least a portion of a field of view of the image sensor by the autofocus camera, wherein the field of view of the image sensor at least partially overlaps with the field of view of the eye of the user; and

wherein focusing on an object in a field of view of the eye of the user by the autofocus camera based on the gaze direction of the eye of the user determined by the eye tracker subsystem includes adjusting, by the focus controller of the autofocus camera, the tunable optical element to focus the field of view of the image sensor on the object in the field of view of the eye of the user based on both the gaze direction of the eye of the user determined by the eye tracker subsystem and the focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera.

13. The method of claim 12 wherein the autofocus camera includes a focus property sensor, and wherein determining a focus property of at least a portion of a field of view of the image sensor by the autofocus camera includes at least one of:

sensing a distance to the object in the field of view of the image sensor by the focus property sensor;

determining a distance to the object in the field of view of the image sensor by the focus property sensor;

detecting a phase difference between at least two points in the field of view of the image sensor by the focus property sensor; and

/or

detecting an intensity difference between at least two points in the field of view of the image sensor by the focus property sensor.

14. The method of claim 12, further comprising:

effecting, by the processor, a mapping between the gaze direction of the eye of the user determined by the eye tracker subsystem and the focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera.

15. The method of claim 14 wherein:

determining a gaze direction of the eye of the user by the eye tracker subsystem includes determining, by the eye tracker subsystem, a first set of two-dimensional coordinates corresponding to the at least one feature of the eye of the user;

determining a focus property of at least a portion of a field of view of the image sensor by the autofocus camera includes determining a focus property of a first region in the field of view of the image sensor by the autofocus camera, the first region in the field of view of the image sensor including a second set of two-dimensional coordinates; and

effecting, by the processor, a mapping between the gaze direction of the eye of the user determined by the eye tracker subsystem and the focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera includes effecting, by the processor, a mapping between the first set of two-dimensional coordinates corresponding to the at least one feature of the eye of the user and the second set of two-dimensional coordinates corresponding to the first region in the field of view of the image sensor.

16. The method of claim 12, further comprising:

effecting, by the processor, a mapping between the gaze direction of the eye of the user determined by the eye tracker subsystem and a field of view of an image sensor of the autofocus camera.

17. The method of claim 11, further comprising:

receiving, by the processor, an image capture command from the user; and

in response to receiving, by the processor, the image capture command from the user, executing, by the processor, the processor-executable data and/or instructions to cause the autofocus camera to focus on the object in the field of view of the eye of the user based on the gaze direction of the eye of the user determined by the eye tracker subsystem.

18. The method of claim 9, further comprising:

capturing an image of the object by the autofocus camera while the autofocus camera is focused on the object.

19. A wearable heads-up display (“

WHUD”

) comprising;

a support structure that in use is worn on a head of a user;

a display content generator carried by the support structure, the display content generator to provide visual display content;

a transparent combiner carried by the support structure and positioned within a field of view of the user, the transparent combiner to direct visual display content provided by the display content generator to the field of view of the user; and

an image capture system that comprises;

an eye tracker subsystem to sense at least one feature of an eye of the user and to determine a gaze direction of the eye of the user based on the at least one feature; and

an autofocus camera communicatively coupled to the eye tracker subsystem, the autofocus camera to automatically focus on an object in a field of view of the eye of the user based on the gaze direction of the eye of the user determined by the eye tracker subsystem.

20. The WHUD of claim 19 wherein the autofocus camera includes:

an image sensor having a field of view that at least partially overlaps with the field of view of the eye of the user;

a tunable optical element positioned and oriented to tunably focus on the object in the field of view of the image sensor; and

a focus controller communicatively coupled to the tunable optical element, the focus controller to apply adjustments to the tunable optical element to focus the field of view of the image sensor on the object in the field of view of the eye of the user based on both the gaze direction of the eye of the user determined by the eye tracker subsystem and a focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera.

21. The WHUD of claim 20, further comprising:

a processor communicatively coupled to both the eye tracker subsystem and the autofocus camera; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable data and/or instructions that, when executed by the processor, cause the processor to effect a mapping between the gaze direction of the eye of the user determined by the eye tracker subsystem and the focus property of at least a portion of the field of view of the image sensor determined by the autofocus camera.