Audio system

• US 9,208,516 B1
• Filed: 10/21/2014
• Issued: 12/08/2015
• Est. Priority Date: 05/16/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Teller, Eric
• Primary Examiner(s): Goodarzi, Nasser
• Assistant Examiner(s): Pierorazio, Michael B
• Application Number:
  US14/519,265
• Time in Patent Office: 413 Days
• Field of Search: 725/12, 725/32, 725/34, 725/35
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 21/123 by using dedicated hardware, e.g. dongles, smart cards, cryptographic processors, global positioning systems [GPS] devices
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 30/0252 based on events or environment, e.g. weather or festivals
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

Methods and systems for providing audio information based on gaze information are described. In an example, a computing device may be configured to receive gaze information indicative of a wearer-view associated with the computing device. Based on the gaze information, the computing device may be configured to determine content displayed in the wearer-view. Further, the computing device may be configured to determine audio information relating to the content and cause an audio stream associated with the audio information to be played at the computing device.

Claims

1. A method, comprising:

receiving, at a wearable computing device, data comprising;

(i) image data indicative of a field of view associated with the wearable computing device, and (ii) view data indicating a view direction in the field of view;

detecting, at a location in the image data that corresponds to the view direction, a physical advertisement space;

determining, based on the image data, that video from a media advertisement is playing in the physical advertisement space, wherein the media advertisement comprises the video and corresponding audio;

in response to determining that the video from the media advertisement is playing in the physical advertisement space, determining a portion of the audio that corresponds to the video from the media advertisement that is being played in the physical advertisement space; and

causing the determined audio portion to be played at the wearable computing device such that the determined audio portion is synchronized to the video of the media advertisement.

2. The method of claim 1, further comprising:

determining, based on the view data, an amount of time during which the location corresponds to the view direction, wherein causing the determined audio portion to be played at the wearable computing device comprises causing the determined audio portion to be played at the wearable computing device in response to determining that the amount of time exceeds a predetermined threshold time period.

3. The method of claim 1, further comprising:

determining, based on the view data, a change in the view direction, wherein the change is indicative that the video that is playing in the physical advertisement space is disregarded;

causing the determined audio portion to stop playing; and

causing the determined audio portion to resume playing in response to determining that view direction has changed back to correspond to the physical advertisement space at which the video from the media advertisement is playing.

4. The method of claim 1, wherein determining the portion of the audio comprises:

generating a search query based on a portion of the image data that corresponds to the view direction;

providing the search query to a server; and

receiving an identification of the media advertisement.

5. The method of claim 4, wherein generating the search query is further based on an amount of time during which the location corresponds to the view direction.

6. The method of claim 1, wherein the image data comprises one or more images of the video from the media advertisement, wherein the one or more images are captured from a point-of-view (POV) of the wearable computing device, and wherein determining the portion of the audio comprises identifying the video based on the one or more images.

7. The method of claim 1, wherein the media advertisement comprises multiple video items that are playing in the physical advertisement space, the method further comprising:

identifying, based on the data, a respective video item of the multiple video items; and

causing a respective audio portion relating to the respective video item to be played at the wearable computing device.

8. The method of claim 7, further comprising:

determining, based on the view data, a change in the view direction, wherein the change is indicative that the respective video item is disregarded;

causing the respective audio portion to stop playing;

identifying, based on the view data, a given video item gazed at in the changed view direction; and

causing a given audio portion relating to the given video item to be played at the wearable computing device.

9. A non-transitory computer readable medium having stored thereon instructions that, when executed a wearable computing device, cause the wearable computing device to perform functions comprising:

receiving data comprising;

(i) image data indicative of a field of view associated with the wearable computing device, and (ii) view data indicating a view direction in the field of view;

detecting, at a location in the image data that corresponds to the view direction, a physical advertisement space;

determining, based on the image data, that video from a media advertisement is playing in the physical advertisement space, wherein the media advertisement comprises the video and corresponding audio;

in response to determining that the video from the media advertisement is playing in the physical advertisement space, determining a portion of the audio that corresponds to the video from the media advertisement that is being played in the physical advertisement space; and

causing the determined audio portion to be played at the wearable computing device such that the determined audio portion is synchronized to the video of the media advertisement.

10. The non-transitory computer readable medium of claim 9, wherein the functions further comprise:

determining, based on the view data, a focus metric that is based on an amount of time during which the location corresponds to the view direction, wherein the function of causing the determined audio portion to be played comprises causing the determined audio portion to be played based on a comparison of the focus metric to a predetermined focus threshold.

11. The non-transitory computer readable medium of claim 9, wherein the functions further comprise:

determining, based on the view data, a change in the view direction, wherein the change is indicative that the video that is playing in the physical advertisement space is disregarded;

causing the determined audio portion to stop playing; and

causing the determined audio portion to resume playing in response to determining that view direction has changed back to correspond to the physical advertisement space at which the media advertisement is playing.

12. The non-transitory computer readable medium of claim 9, wherein the function of determining the portion of the audio comprises:

generating a search query based on (i) a portion of the image data that corresponds to the view direction, and (ii) an amount of time during which the location corresponds to the view direction;

providing the search query to a server; and

receiving an identification of the media advertisement.

13. The non-transitory computer readable medium of claim 9, wherein the media advertisement comprises multiple video items that are playing in the physical advertisement space, and wherein the functions further comprise:

identifying, based on the data, a respective video item of the multiple video items; and

causing a respective audio portion relating to the respective video item to be played at the computing device.

14. The non-transitory computer readable medium of claim 13, wherein the functions further comprise:

determining, based on the view data, a change in the view direction, wherein the change is indicative that the respective video item is disregarded;

causing the respective audio portion to stop playing;

identifying, based on the gaze information, a given video item gazed at in the changed view direction; and

causing a given audio portion relating to the given video item to be played at the computing device.

15. A system, comprising:

a head-mountable display (HMD);

an eye-sensing device coupled to the HMD;

an image-capture device coupled to the HMD, wherein the image-capture device is configured to capture a field of view of the HMD;

at least one processor in communication with the HMD, the eye-sensing device, and the image-capture device; and

data storage comprising instructions that, when executed by the at least one processor, cause the system to perform functions comprising;

receiving, from the image-capture device, image data indicative of the field of view of the HMD,

receiving, from the eye-sensing device, view data indicating a view direction in the field of view;

detecting, at a location in the image data that corresponds to the view direction, a physical advertisement space;

determining that video from a media advertisement is playing in the physical advertisement space, wherein the media advertisement comprises the video and corresponding audio;

in response to determining that the video from the media advertisement is playing in the physical advertisement space, determining a portion of the audio that corresponds to the video from the media advertisement that is being played in the physical advertisement space; and

causing the determined audio portion to be played at the HMD such that the determined audio portion is synchronized to the video of the media advertisement.

16. The system of claim 15, wherein the functions further comprise:

determining, based on the view data, an amount of time during which the location corresponds to the view direction, wherein the function of causing the determined audio portion to be played at the HMD comprises causing the determined audio portion to be played at the HMD in response to determining that the amount of time exceeds a predetermined threshold time period.

17. The system of claim 15, wherein the functions further comprise:

determining, based on the view data, a change in the view direction, wherein the change is indicative that the video that is playing in the physical advertisement space is disregarded;

causing the determined audio portion to stop playing; and

causing the determined audio portion to resume playing in response to determining that view direction has changed back to correspond to the physical advertisement space at which the media advertisement is playing.

18. The system of claim 15, wherein the function of determining the portion of the audio comprises:

generating a search query based on (i) a portion of the image data that corresponds to the view direction, and (ii) an amount of time during which the location corresponds to the view direction;

providing the search query to a server; and

receiving an identification of the media advertisement.

19. The system of claim 15, wherein the image data comprises one or more images of the video, wherein the one or more images are captured from a point-of-view of the HMD, and wherein the function of determining the portion of the audio comprises identifying the video based on the one or more images.

20. The system of claim 15, wherein the media advertisement comprises multiple video items that are playing in the physical advertisement space, and wherein the functions further comprise:

identifying a first video item of the multiple video items;

causing a respective audio portion relating to the first video item to be played;

determining, based on the view data, a change in the view direction, wherein the change is indicative that the first video item is disregarded;

causing the respective audio portion to stop playing;

identifying a second video item gazed at in the changed view direction; and

causing a respective audio portion relating to the second video item to be played.