Eye reflection image analysis

• US 9,684,374 B2
• Filed: 02/20/2015
• Issued: 06/20/2017
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Raffle, Hayes Solos, Zhao, Yong, Starner, Thad Eugene
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): PERLMAN, DAVID S
• Application Number:
  US14/627,790
• Publication Number:
  US 20150160461A1
• Time in Patent Office: 851 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

Example methods and devices are disclosed for generating life-logs with point-of-view images. An example method may involve: receiving image-related data based on electromagnetic radiation reflected from a human eye, generating an eye reflection image based on the image-related data, generating a point-of-view image by filtering the eye reflection image, and storing the point-of-view image. The electromagnetic radiation reflected from a human eye can be captured using one or more video or still cameras associated with a suitably-configured computing device, such as a wearable computing device.

Claims

1. A computer-implemented method comprising:

receiving, by a computing device, eye-movement data corresponding to an eye, wherein the eye-movement data is generated by one or more sensors of a head-mountable device (HMD);

generating, by the computing device, point-of-view image data based on electromagnetic radiation reflected from a surface of the eye, wherein the point-of-view image data corresponds to a field of view of the eye;

determining that the point-of-view image data comprises a reflection, from the eye, of a real-world object;

determining, by the computing device, that the eye-movement data corresponds to a stare in a direction of the real-world object;

in response to determining that the eye-movement data corresponds to the stare in the direction of the real-world object, the computing device;

(a) initiating an image-based search on the real-world object, wherein the image-based search provides search results based on a comparison of image data comprising the real-world object and reference image data; and

(b) causing a graphic display to display search result information based on the image-based search; and

while the search result information is displayed on the graphic display;

receiving subsequent eye-movement data;

determining whether or not the subsequent eye-movement data corresponds to a stare in the direction of the search result information;

if the subsequent eye-movement data corresponds to a stare, then responsively storing the search result information to a life-log of a user-account associated with the computing device; and

otherwise, if the subsequent eve-movement data does not correspond to a stare, then refraining from storing the search result information to the life-log.

2. The method of claim 1, wherein generating the point-of-view image based on electromagnetic radiation reflected from a surface of the eye comprises the computing device:

generating an eye reflection image based on the electromagnetic radiation reflected from the surface of the eye, wherein the eye reflection image comprises an image of the surface of the eye with one or more reflections, from the eye, of one or more objects in the field of view of the eye, and wherein the one or more objects comprise the real-world object; and

filtering the eye reflection image to generate the point-of-view image, wherein the filtering comprises determining a difference between the eye reflection image and a pre-determined reference image of an eye surface.

3. The method of claim 2, wherein the eye-movement data comprises the eye reflection image.

4. The method of claim 1, wherein the eye-movement data comprises data generated by one or more proximity sensors that are arranged on the HMD such that the one or more proximity sensors are directed towards an eye area of the eye when the HMD is worn.

5. The method of claim 1, wherein the image-based search request comprises the point-of-view image and an indication of a location of the real-world object within the point-of-view image.

6. The method of claim 1, further comprising, performing image processing on the point-of-view image data to generate image data of the real-world object, wherein the image-based search request comprises the generated image data of the real-world object.

7. The method of claim 1, wherein determining that the eye-movement data corresponds to the stare in the direction of the real-world object comprises:

detecting, based on the eye-movement data, that the eye does not move more than a threshold amount during a predetermined period of time; and

determining that the real-world object remains in the point-of-view image data during the predetermined period of time.

8. The method of claim 1, wherein initiating the image-based search comprises transmitting an image-based search request for an image-based search on the real-world object, the method further comprising:

receiving search-result data corresponding to the image-based search request, wherein the displayed search-result information is based on the received search-result data.

9. A head-mountable device (HMD) comprising:

one or more sensors that are arranged on the HMD so as to generate eye-movement data corresponding to an eye;

at least one processor;

program instructions stored on a non-transitory computer-readable medium and executable by the at least on processor to;

generate point-of-view image data based on electromagnetic radiation reflected from a surface of the eye, wherein the point-of-view image data corresponds to a field of view of the eye;

determine that the point-of-view image data comprises a reflection, from the eye, of a real-world object;

determine that the eye-movement data corresponds to a stare in a direction of the real-world object;

in response to determining that the eye-movement data corresponds to the stare in the direction of the real-world object;

(a) initiate an image-based search on the real-world object, wherein the image-based search provides search results based on a comparison of image data comprising the real-world object and reference image data; and

(b) cause a graphic display to display search result information based on the image-based search; and

while the search result information is displayed on the graphic display;

receive subsequent eye-movement data;

determine whether or not the subsequent eye-movement data corresponds to a stare in the direction of the search result information;

if the subsequent eye-movement data corresponds to a stare, then responsively store the search result information to a life-log of a user-account associated with the computing device; and

otherwise, if the subsequent eye-movement data does not correspond to a stare, then refrain from storing the search result information to the life-log.

10. The HMD of claim 9, wherein the HMD further comprises at least one inward-facing image capture device arranged on the HMD to capture eye reflection image data when the HMD is worn, wherein the program instructions are further executable by the at least on processor to:

operate the at least one inward-facing image capture device to generate an eye reflection image that captures the electromagnetic radiation reflected from the surface of the eye, wherein the eye reflection image comprises an image of the surface of the eye with one or more reflections, from the eye, of one or more objects in the field of view of the eye, and wherein the one or more objects comprise the real-world object; and

filter the eye reflection image to generate the point-of-view image, wherein the filtering comprises a determination of a difference between the eye reflection image and a pre-determined reference image of an eye surface.

11. The HMD of claim 10, wherein the one or more sensors comprise the inward-facing image capture device, and wherein the wherein the eye-movement data comprises the eye reflection image.

12. The HMD of claim 10, wherein the one or more sensors comprise one or more proximity sensors that are arranged on the HMD such that the one or more proximity sensors are directed towards an eye area of the eye when the HMD is worn.

13. The HIVID of claim 9, wherein the one or more sensors comprise one or more proximity sensors that are arranged on the HMD such that the one or more proximity sensors are directed towards an eye area of the eye when the HMD is worn.

14. The HIVID of claim 9, wherein, to determine that the eye-movement data corresponds to the stare in the direction of the real-world object, the program instructions are executable to:

determine, based on the eye-movement data, that the eye has not moved more than a threshold amount during a predetermined period of time; and

determine that the real-world object remains in the point-of-view image data during the predetermined period of time.

15. A computer-implemented method comprising:

receiving, by a computing device, eye-movement data corresponding to an eye, wherein the eye-movement data is generated by one or more sensors of a head-mountable device (HMD);

generating, by the computing device, point-of-view image data based on electromagnetic radiation reflected from a surface of the eye, wherein the point-of-view image data corresponds to a field of view of the eye;

determining that the point-of-view image data comprises a reflection, from the eye, of a real-world object;

determining, by the computing device, that the eye-movement data corresponds to a stare in a direction of the real-world object;

in response to determining that the eye-movement data corresponds to the stare in the direction of the real-world object, the computing device;

(a) initiating an image-based search on the real-world object, wherein the image-based search provides search results based on a comparison of image data comprising the real-world object and reference image data; and

(b) storing the search result information corresponding to the image-based search in a life-log; and

while the search result information is displayed on the graphic display;

receiving subsequent eye-movement data;

determining whether or not the subsequent eye-movement data corresponds to a stare in the direction of the search result information;

if the subsequent eye-movement data corresponds to a stare, then responsively storing the search result information to a life-log of a user-account associated with the computing device; and

otherwise, if the subsequent eve-movement data does not correspond to a stare, then refraining from storing the search result information to the life-log.