Audio system

• US 8,893,164 B1
• Filed: 05/16/2012
• Issued: 11/18/2014
• Est. Priority Date: 05/16/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Teller, Eric
• Primary Examiner(s): Goodarzi, Nasser
• Assistant Examiner(s): Pierorazio, Michael B
• Application Number:
  US13/472,902
• Time in Patent Office: 916 Days
• Field of Search: 725/12
• US Class Current: 725/12
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 21/123 by using dedicated hardware, e.g. dongles, smart cards, cryptographic processors, global positioning systems [GPS] devices
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 30/0252 based on events or environment, e.g. weather or festivals
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

Methods and systems for providing audio information based on gaze information are described. In an example, a computing device may be configured to receive gaze information indicative of a wearer-view associated with the computing device. Based on the gaze information, the computing device may be configured to determine content displayed in the wearer-view. Further, the computing device may be configured to determine audio information relating to the content and cause an audio stream associated with the audio information to be played at the computing device.

Claims

1. A method, comprising:

receiving, at a wearable computing device, gaze information comprising;

(i) an image or a video indicative of a wearer-view through a field of view of the wearable computing device, and (ii) data indicating a view direction associated with the wearer-view;

identifying a video advertisement that is playing, in a physical advertisement space captured in the image or the video, at a location in the image or the video that corresponds to the view direction, wherein identifying the video advertisement comprises;

generating a search query based on (i) a portion of the image or the video at the location that corresponds to the view direction, and (ii) an amount of time during which the location in the image or the video corresponds to the view direction,

providing the search query to a server, and

receiving an identification of the video advertisement;

in response to identifying the video advertisement that is playing in the physical advertisement space that corresponds to the view direction, determining an audio advertisement relating to the video advertisement; and

causing the audio advertisement to be played at the wearable computing device.

2. The method of claim 1, wherein causing the audio advertisement to be played at the wearable computing device comprises causing the audio advertisement to be played at the wearable computing device in response to determining that the amount of time exceeds a predetermined threshold time period.

3. The method of claim 1, further comprising:

determining, based on the gaze information, a change in the view direction, wherein the change is indicative that the video advertisement is disregarded;

causing the audio advertisement to stop playing; and

causing the audio advertisement to resume playing in response to determining that the view direction has changed back to correspond to the location at which video advertisement is playing.

4. The method of claim 1, wherein receiving the gaze information comprises receiving information relating to a gaze axis from an eye-tracking system coupled to the wearable computing device, wherein the data indicating the view direction is based on the gaze axis.

5. The method of claim 4, further comprising determining the wearer-view based on the gaze axis.

6. The method of claim 1, wherein the audio advertisement is synchronized to the video advertisement playing in the physical advertisement space.

7. A system, comprising:

an eye-tracking system;

a camera coupled to a head-mounted display (HMD), wherein the camera is configured to capture a field of view of the HMD; and

a computing device in communication with the eye-tracking system and the camera, wherein the computing device is configured to;

receive, from the eye-tracking system, gaze information comprising;

(i) an image or a video indicative of a wearer-view through a field of view of the wearable computing device, and (ii) data indicating a view direction associated with the wearer-view;

identify an animated content that is playing, on a display device captured in the image or the video, at a location in the image or the video that corresponds to the view direction, wherein, to identify the animated content, the computing device is configured to;

generate a search query based on (i) a portion of the image or the video at the location that corresponds to the view direction, and (ii) an amount of time during which the location in the image or the video corresponds to the view direction,

provide the search query to a server, and

receive an identification of the animated content;

in response to identifying the animated content that is playing on the display device that corresponds to the view direction, determine audio information relating to the animated content; and

cause an audio stream associated with the audio information to be played at the computing device.

8. The system of claim 7, wherein the computing device is further configured to:

determine, based on the gaze information, a change in the view direction, wherein the change is indicative of the animated content playing on the display device being disregarded; and

cause the audio stream to stop playing.

9. The system of claim 8, wherein the computing device is further configured to cause the audio stream to resume playing in response to determining that the view direction has changed back to correspond to the location at which the animated content is playing.

10. The system of claim 7, wherein the eye-tracking system comprises a given camera, wherein the gaze information comprises one or more images captured by the given camera, and wherein the computing device is further configured to:

determine a gaze axis based on the one or more images; and

determine the view direction based on the determined gaze axis.

11. The system of claim 10, wherein the image or the video are captured for the determined wearer-view.

12. The system of claim 7, wherein the audio stream is synchronized to the animated content.

13. The system of claim 7, wherein the animated content comprises multiple items displayed on the display device, and wherein the computing device is further configured to:

identify, based on the gaze information, a respective item of the multiple items; and

cause a respective audio stream relating to the respective item to be played at the computing device.

14. The system of claim 13, wherein the computing device is further configured to:

determine, based on the gaze information, a change in the view direction, wherein the change is indicative that the respective item is disregarded;

cause the respective audio stream item to stop playing;

identify, based on the gaze information, a given item gazed at in the changed view direction; and

cause a given audio stream relating to the given item to be played at the computing device.

15. The system of claim 7, wherein the computing device is further configured to:

determine, based on the gaze information, a change in the view direction, wherein the change is indicative that the animated content on the display device is disregarded;

cause the audio stream to stop playing;

identify, based on the gaze information, a respective content depicted on a respective display device in the changed view direction; and

cause a respective audio stream to be played at the computing device.

16. The system of claim 7, wherein the animated content comprises one or more of a video, an interactive media element, a game, an advertisement, and a graphical user-interface.

17. A non-transitory computer readable medium having stored thereon instructions executable by a wearable computing device to cause the wearable computing device to perform functions comprising:

receiving gaze information comprising;

(i) an image or a video indicative of a wearer-view through a field of view of the wearable computing device, and (ii) data indicating a view direction associated with the wearer-view;

identifying a video advertisement that is playing, in a physical advertisement space captured in the image or the video, at a location in the image or the video that corresponds to the view direction, wherein identifying the video advertisement comprises;

generating a search query based on (i) a portion of the image or the video at the location that corresponds to the view direction, and (ii) an amount of time during which the location in the image or the video corresponds to the view direction,

providing the search query to a server, and

receiving an identification of the video advertisement;

in response to identifying the video advertisement that is playing in the physical advertisement space that corresponds to the view direction, determining an audio advertisement relating to the video advertisement; and

causing the audio advertisement to be played at the wearable computing device.

18. The non-transitory computer readable medium of claim 17, wherein the instructions are further executable by the wearable computing device to cause the wearable computing device to perform functions comprising:

determining, based on the gaze information, a focus metric, wherein the focus metric is based on the amount of time during which the location in the image or the video corresponds to the view direction; and

causing the audio advertisement to be played at the wearable computing device based on a comparison of the focus metric to a predetermined focus threshold.