Adjusting image frames based on tracking motion of eyes

• US 10,338,677 B2
• Filed: 10/28/2015
• Issued: 07/02/2019
• Est. Priority Date: 10/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Guenter, Brian K.
• Primary Examiner(s): Geroleo, Francis
• Application Number:
  US14/925,892
• Publication Number:
  US 20170123489A1
• Time in Patent Office: 1,343 Days
• Field of Search: 348 78
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

In some examples, techniques and architectures for operating a device that may be wearable, such as a head-mounted device, may be used for virtual reality applications. A processor of the device may operate by dynamically tracking the cornea, and thus the gaze direction, of a user'"'"'s eyes. By tracking the gaze direction, retinal velocity of a display image projected onto the retina, measured in the retina coordinate frame may be computed. Display times of image frames viewable by the user may be modified based on the retinal velocity.

Claims

1. A system comprising:

a light emitter to emit light toward an eye of a subject, the eye having a retina and a cornea;

a camera to capture images of the cornea of the eye having glints produced by reflection of the light from the cornea;

a display device operated by a display module to sequentially display image frames; and

a processor to;

determine, based at least in part on a first captured image of the cornea, first locations of the glints at a first time, while the display device is displaying an image frame, and a first gaze direction, and generate a first virtual cornea sphere based on the first locations, with the first virtual cornea sphere having a first center;

determine, based at least in part on a second captured image of the cornea, second locations of the glints at a second time, while the display device is displaying the image frame, and a second gaze direction, and generate a second virtual cornea sphere based on the second locations, with the second virtual cornea sphere having a second center;

generate a virtual sphere having a center, using the first and second centers;

determine a retinal velocity of the eye corresponding to differences between the first gaze direction and the second gaze direction, the determined retinal velocity being velocity of the image frame projected onto the retina, measured in a retinal coordinate frame using the center of the virtual sphere; and

provide signals to the display module to change a display time for the image frame based, at least in part, on the retinal velocity.

2. The system of claim 1, further comprising:

an image processor module accessible by the processor to determine the first locations of the glints in the first captured image and the second locations of the glints in the second captured image.

3. The system of claim 1, wherein if the retinal velocity is greater than a threshold value, the processor is configured to:

determine, based at least in part on additional captured images of the cornea, additional sets of locations of the glints at corresponding additional times while the display is displaying the image frame;

determine additional retinal velocities corresponding to differences among the additional sets of locations of the glints; and

provide signals to the display module to extend the display time for the image frame if the additional retinal velocities are greater than the threshold value or terminate displaying the image frame if the additional retinal velocities are less than the threshold value.

4. The system of claim 3, wherein the processor is configured to determine, based at least in part on the additional sets of locations of the glints, if cortical movement of the eye comprises saccadic motions, and wherein if the eye comprises the saccadic motions, the processor increases a frequency of images that are captured by the camera.

5. The system of claim 1, further comprising instructions stored in memory accessible by the processor to operate the camera to capture the images of the cornea at a frequency that is based, at least in part, on the retinal velocity.

6. The system of claim 1, wherein the image frame comprises a color frame of a color-sequential image.

7. The system of claim 1, wherein the light emitter is configured to emit two or more beams of the light from different respective directions toward the eye of the subject.

8. The system of claim 1, wherein the system comprises a head-mounted display.

9. The system of claim 1, wherein the glints comprise specularly reflected originating from the light emitter.

10. The system of claim 1, wherein the system comprises a virtual reality system.

11. A head-mounted device comprising:

multiple light emitters configured to direct infrared light toward an eye of a wearer of the head-mounted device, the eye having a retina and a cornea;

a camera configured to capture images of the cornea of an eye of the wearer with respect to image frames displayed to the eye; and

a processor to;

determine relative positions of glints on the cornea in images captured by the camera for an image frame of the image frames;

determine multiple gaze directions and multiple virtual cornea spheres, based at least in part on the relative positions of glints, with each virtual cornea sphere having a center and with each gaze direction and each virtual cornea sphere corresponding to a time different from times corresponding to the other gaze directions and other virtual cornea sphere;

generate a virtual sphere having a center of the virtual sphere, using the centers of the virtual cornea spheres;

determine retinal velocity of the eye based, at least in part, on the gaze directions associated with the relative positions of the glints on the cornea, the determined retinal velocity being velocity of the image frame projected onto the retina, measured in a retinal coordinate frame, using the center of the virtual sphere; and

modify display times of image frames based, at least in part, on the retinal velocity.

12. The head-mounted device of claim 11, wherein the processor is configured to determine, based at least in part on gaze direction and a location of an object in the image frames, if the eye is moving in a saccadic motion or if the eye is substantially tracking the location of the object, and wherein if the eye comprises the saccadic motion, the processor increases a frequency of images that are captured by the camera.

13. The head-mounted device of claim 12, wherein the object is a synthetic object of a virtual reality scene.

14. The head-mounted device of claim 11, wherein the multiple light emitters and the camera are positioned relative to one another so that light from the multiple light emitters reflects from the cornea of the eye and enters an aperture of the camera.

15. The head-mounted device of claim 11, wherein relative positions of the glints in the images depend, at least in part, on rotational orientation of the eye.

16. A method comprising:

capturing images of a cornea of an eye of a subject while the subject gazes at a display device, wherein the images include sets of glint points on the cornea produced by specular reflection of light by a surface of the cornea;

determining, based at least in part on a series of the images, if the cornea is moving in a saccadic motion or is moving substantially continuously; and

adjusting durations of image frames displayed on the display device based, at least in part, on whether the cornea is moving in the saccadic motion or moving substantially continuously using a retinal velocity of the eye determined from gaze directions and a center of a virtual sphere calculated from generation of virtual cornea spheres associated with relative positions of the glints captured with respect to an image frame of the image frames displayed, the determined retinal velocity being velocity of the image frame projected onto a retina of the eye, measured in a retinal coordinate frame using the center of the virtual sphere.

17. The method of claim 16, wherein the image frames displayed on the display device comprise color-sequential images.

18. The method of claim 16, further comprising:

recording to a memory device motions of the cornea with respect to a location of an object in the image frames.

19. The method of claim 18, wherein the object is a synthetic object of a virtual reality scene displayed on the display device.

20. The method of claim 16, wherein the light comprises infrared light.