Gaze tracking system

• US 8,510,166 B2
• Filed: 05/11/2011
• Issued: 08/13/2013
• Est. Priority Date: 05/11/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Neven, Hartmut
• Primary Examiner(s): Coppola, Jacob C.
• Application Number:
  US13/105,701
• Publication Number:
  US 20120290401A1
• Time in Patent Office: 825 Days
• Field of Search: 705/14.68
• US Class Current: 705/14.68
• CPC Class Codes: A61B 3/112 for measuring diameter of pupils
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A gaze tracking technique is implemented with a head mounted gaze tracking device that communicates with a server. The server receives scene images from the head mounted gaze tracking device which captures external scenes viewed by a user wearing the head mounted device. The server also receives gaze direction information from the head mounted gaze tracking device. The gaze direction information indicates where in the external scenes the user was gazing when viewing the external scenes. An image recognition algorithm is executed on the scene images to identify items within the external scenes viewed by the user. A gazing log tracking the identified items viewed by the user is generated.

Claims

1. A method comprising:

receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device, the scene images received at a server via a network;

receiving gaze direction information from the head mounted gaze tracking device along with the scene images, the gaze direction information indicating where in the external scenes the user was gazing when viewing the external scenes, the gaze direction information received at the server via the network;

executing an image recognition algorithm on the scene images to identify items within the external scenes viewed by the user;

generating a gazing log tracking the identified items viewed by the user;

performing latent pre-searches on at least a portion of the items viewed by the user to generate latent search results, wherein the latent pre-searches are automatically triggered while the associated items are in the user'"'"'s peripheral view and without affirmative requests on a per search basis by the user; and

caching the latent search results.

2. The method of claim 1, wherein generating the gazing log further comprises storing an indication of whether the user looked directly at the identified items based at least in part upon the gaze direction information.

3. The method of claim 2, wherein the user is deemed to have looked directly at a given identified item if a position of the given identified item within a given scene image correlates to the gaze direction information associated with the given scene image and if the user looked directly at the given identified item for a threshold period of time.

4. The method of claim 2, wherein generating the gazing log further comprises storing an indication of how long and when the user looked at one or more of the identified items based upon the gaze direction information.

5. The method of claim 1, further comprising:

receiving pupil dilation information from the head mounted gaze tracking device along with the scene images, the pupil dilation information indicating a pupil dilation of the user while viewing the external scenes, the pupil dilation information received at the server via the network.

6. The method of claim 5, further comprising:

inferring an emotional state of the user while viewing the external scenes based at least in part upon the pupil dilation information; and

storing an emotional state indication associated with one or more of the identified items.

7. The method of claim 1, further comprising:

determining which, if any, of the identified items within the external scenes viewed by the user are advertisements; and

charging advertisers associated with the advertisements based at least in part on a per gaze basis.

8. The method of claim 7, wherein charging the advertisers further comprises charging a given advertiser associated with a given advertisement based at least in part upon whether the user looked directly at the given advertisement as determined by the gaze direction information and how long the user looked at the given advertisement.

9. The method of claim 1, further comprising:

determining which, if any, of the identified items within the external scenes viewed by the user are advertisements; and

charging advertisers for analytical information generated based upon the gazing direction information.

10. The method of claim 9, further comprising:

combining the gazing direction information acquired from the user wearing the head mounted device with other gazing direction information acquired from other users wearing other head mounted devices to generate batched information; and

charging advertisers for the analytical information generated based upon the batched information.

11. The method of claim 10, wherein the analytical information includes the tendency of a given advertisement to draw user gazes or to hold the user gazes.

12. The method of claim 10, wherein the analytical information includes the tendency of a given advertisement to evoke an emotional response.

13. The method of claim 1, further comprising:

providing selected ones of the cached latent search results to the user upon request.

14. The method of claim 2, further comprising:

providing the user access to a personally viewing history search feature; and

searching the gazing log upon receiving a viewing history search request from the user.

15. The method of claim 14, wherein the viewing history search feature provides the user with a list of historical influences by advertisements.

16. The method of claim 1, wherein the head mounted gaze tracking device comprises eyeglasses including side-arms that engage ears of the user, a nose bridge that engages a nose of the user, and lenses through which the user views the external scenes,

wherein the scene images are captured in real-time while the user views the external scenes with at least one forward facing scene camera disposed on the eyeglasses,

wherein the gaze direction information is generated based upon eye images of at least one eye of the user captured in real-time while the user views the external scenes with at least one gaze tracking camera disposed on the eyeglasses.

17. The method of claim 16, wherein the gaze direction information is captured based upon the eye images captured with the at least one gaze tracking camera being substantially forward facing and disposed on at least one of the side-arms of the eyeglasses, the eye images captured based upon reflections off of at least one of the lenses through which the user perceives the external scene.

18. A tangible machine-readable storage medium that provides instructions that, when executed by a machine, will cause the machine to perform operations comprising:

receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device, the scene images received at a server via a network;

receiving gaze direction information from the head mounted gaze tracking device along with the scene images, the gaze direction information indicating where in the external scenes the user was gazing when viewing the external scenes, the gaze direction information received at the server via the network;

executing an image recognition algorithm on the scene images to identify items within the external scenes viewed by the user;

generating a gazing log tracking the identified items viewed by the user;

performing latent pre-searches on at least a portion of the items viewed by the user to generate latent search results, wherein the latent pre-searches are automatically triggered while the associated items are in the user'"'"'s peripheral view and without affirmative requests on a per search basis by the user; and

caching the latent search results.

19. The tangible machine-readable storage medium of claim 18, wherein generating the gazing log further comprises storing an indication of whether the user looked directly at the identified items based at least in part upon the gaze direction information.

20. The tangible machine-readable storage medium of claim 19, wherein the user is deemed to have looked directly at a given identified item if a position of the given identified item within a given scene image correlates to the gaze direction information associated with the given scene image and if the user looked directly at the given identified item for a threshold period of time.

21. The tangible machine-readable medium of claim 18, further providing instructions that, when executed by the machine, will cause the machine to perform further operations, comprising:

receiving pupil dilation information from the head mounted gaze tracking device along with the scene images, the pupil dilation information indicating a pupil dilation of the user while viewing the external scenes, the pupil dilation information received at the server via the network.

22. The tangible machine-readable medium of claim 21, further providing instructions that, when executed by the machine, will cause the machine to perform further operations, comprising:

inferring emotional state of the user while viewing the external scenes based at least in part upon the pupil dilation information; and

storing an emotional state indication associated with one or more of the identified items.

23. The tangible machine-readable medium of claim 18, further providing instructions that, when executed by the machine, will cause the machine to perform further operations, comprising:

determining which, if any, of the identified items within the external scenes viewed by the user are advertisements; and

charging advertisers associated with the advertisements based at least in part on a per gaze basis and whether the user looked directly at the given advertisement as determined by the gaze direction information and how long the user looked at the given advertisement.

24. The tangible machine-readable medium of claim 18, further providing instructions that, when executed by the machine, will cause the machine to perform further operations, comprising:

providing selected ones of the cached latent search results to the user upon request.

25. The tangible machine-readable medium of claim 18, further providing instructions that, when executed by the machine, will cause the machine to perform further operations, comprising:

providing the user access to a personally viewing history search feature; and

searching the gazing log upon receiving a viewing history search request from the user.