In-cell gaze tracking for near-eye display

• US 9,703,374 B1
• Filed: 12/16/2015
• Issued: 07/11/2017
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhang, Zhibin
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US14/971,477
• Publication Number:
  US 20170177075A1
• Time in Patent Office: 573 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A near-eye display device includes a display panel with an array of photon-emitting cells interspersed with photon-detecting cells and a display controller coupled to the display panel, the display controller to control the display panel to display imagery using the array of photon-emitting cells. The device further includes a camera controller coupled to the display panel, the camera controller to control the display panel to capture imagery of an eye of a user using the photon-detecting cells. The device also includes an eye-tracking module coupled to the camera controller, the eye-tracking module to construct a three-dimensional representation of the eye based on the captured imagery.

Claims

1. A near-eye display device comprising:

a display panel comprising an array of photon-emitting cells interspersed with photon-detecting cells;

a display controller coupled to the display panel, the display controller to control the display panel to display imagery using the array of photon-emitting cells; and

a camera controller coupled to the display panel, the camera controller to control the display panel to capture imagery of an eye of a user using the photon-detecting cells; and

an eye-tracking module coupled to the camera controller, the eye-tracking module to construct a three-dimensional representation of the eye based on the captured imagery.

2. The near-eye display device of claim 1, further comprising:

a set of one or more infrared (IR) light sources disposed at corresponding positions in association with the display panel, the set of one or more IR light sources to illuminate the eye during the capture of the imagery of the eye.

3. The near-eye display device of claim 2, wherein:

the camera controller is to activate the set of one or more IR light sources to illuminate the eye during a vertical synchronization period of the display panel for the array of photon-emitting cells.

4. The near-eye display device of claim 2, wherein:

the eye-tracking module is to construct the three-dimensional representation of the eye based on a time-of-flight analysis.

5. The near-eye display device of claim 2, wherein:

the set of one or more IR light sources comprises a set of one or more vertical-cavity surface-emitting lasers.

6. The near-eye display device of claim 1, wherein:

the photon-detecting cells are arranged into a first subset and a second subset, wherein each photon-detecting cell of the first subset is paired with a corresponding photon-detecting cell of the second subset;

the camera controller is to concurrently capture a first image of the eye using the photon-detecting cells of the first subset and to capture a second image of the eye using the photon-detecting cells of the second subset; and

the eye-tracking module is to determine the three-dimensional representation of the eye using parallax-based analysis of the first and second images.

7. The near-eye display device of claim 6, wherein:

the display panel further comprises a microlens disposed over each photon-detecting cell of at least some of the photon-detecting cells of the display panel.

8. The near-eye display device of claim 1, wherein:

the eye-tracking module further is to determine, based on the three-dimensional representation of the eye, at least one of;

a presence of the eye;

a position of an eyelid of the eye;

a position of the eye;

an orientation of the eye; and

a gaze direction of the eye.

9. The near-eye display device of claim 8, wherein:

at least one component of the near-eye display device is controlled based on the determined presence of the eye, position of the eyelid of the eye, position of the eye, orientation of the eye, or gaze direction of the eye.

10. The near-eye display device of claim 1, wherein the near-eye display device is a head-mounted display (HMD) device.

11. In a near-eye display device having a display panel comprising an array of photon-emitting cells interspersed with photon-detecting cells, a method comprising:

controlling, using a display controller of the device, the display panel to display imagery using the array of photon-emitting cells; and

controlling, using a camera controller of the device, the display panel to capture imagery of an eye of a user using the photon-detecting cells; and

constructing, using an eye-tracking module of the device, a three-dimensional representation of the eye based on the captured imagery.

12. The method of claim 11, further comprising:

activating a set of one or more infrared (IR) light sources disposed at corresponding positions in association with the display panel so as to illuminate the eye during the capture of the imagery of the eye.

13. The method of claim 12, wherein:

activating the set of one or more IR light sources comprises activating the set of one or more IR light sources during a vertical synchronization period of the display panel for the array of photon-emitting cells.

14. The method of claim 12, wherein:

constructing the three-dimensional representation of the eye comprises constructing the three-dimensional representation of the eye based on a time-of-flight analysis using the photon-detecting cells.

15. The method of claim 11, wherein:

the photon-detecting cells are arranged into a first subset and a second subset, wherein each photon-detecting cell of the first subset is paired with a corresponding photon-detecting cell of the second subset;

controlling the display panel to capture imagery of the eye comprises controlling the display panel to capture a first image of the eye using the photon-detecting cells of the first subset and to capture a second image of the eye using the photon-detecting cells of the second subset; and

constructing the three-dimensional representation of the eye comprises constructing the three-dimensional representation of the eye using parallax-based processing of the first and second images.

16. The method of claim 11, further comprising:

determining, using the eye-tracking module, at least one of;

a presence of the eye;

a position of an eyelid of the eye;

a position of the eye;

an orientation of the eye; and

a gaze direction of the eye based on the three-dimensional representation of the eye.

17. The method of claim 16, further comprising:

controlling at least one component of the near-eye display device based on the determined presence of the eye, position of the eyelid of the eye, position of the eye, orientation of the eye, or gaze direction of the eye.

18. In a near-eye display device, a method comprising:

displaying imagery to an eye of a user via a set of photon-emitting cells of a display panel of the device;

capturing imagery of the eye via a set of photon-detecting cells interspersed among the photon-emitting cells of the display panel;

determining a gaze characteristic of the eye based on the captured imagery; and

controlling an operation of the device based on the gaze characteristic.

19. The method of claim 18, wherein:

determining the gaze characteristic comprises determining at least one of;

a presence of the eye and a position of an eyelid of the eye; and

controlling an operation of the device comprises at least one of;

activating or deactivating the display panel based on at least one of the presence of the eye and the position of the eyelid.

20. The method of claim 18, wherein:

determining the gaze characteristic comprises determining a gaze direction of the eye; and

controlling an operation of the device comprises modifying the imagery displayed to the eye of the user based on the gaze direction.

21. The method of claim 18, wherein:

determining the gaze characteristic comprises determining a three-dimensional representation of the eye based on the captured imagery.

22. The method of claim 21, wherein:

determining the three-dimensional representation of the eye comprises determining the three-dimensional representation of the eye based on time-of-flight analysis of the captured imagery.

23. The method of claim 21, wherein:

capturing imagery of the eye comprises concurrently capturing a first image of the eye using a first subset of the photon-detecting cells and capturing a second image of the eye using a second subset of the photon-detecting cells; and

determining the three-dimensional representation of the eye comprises determining the three-dimensional representation of the eye based on parallax-based processing of the first and second images.