Systems and methods for automatically suggesting media accompaniments based on identified media content

• US 10,467,287 B2
• Filed: 12/12/2013
• Issued: 11/05/2019
• Est. Priority Date: 12/12/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Hume, Thomas Weedon, Lerda, Flavio, Koser, Mikkel Crone
• Primary Examiner(s): Chokshi, Pinkal R
• Application Number:
  US14/104,157
• Publication Number:
  US 20150169747A1
• Time in Patent Office: 2,154 Days
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G06F 16/687 using geographical or spatial information, e.g. location
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 21/8113 comprising music, e.g. song in MP3 format

Abstract

The disclosed technology includes automatically suggesting audio, video, or other media accompaniments to media content based on identified objects in the media content. Media content may include images, audio, video, or a combination. In one implementation, one or more images representative of the media content may be extracted. A visual search may be run across the images to identify objects or characteristics present in or associated with the media content. Keywords may be generated based on the identified objects and characteristics. The keywords may be used to determine suitable audio tracks to accompany the media content, for example by performing a search based on the keywords. The determined tracks may be presented to a user, or automatically arranged to match the media content. In another implementation, an aural search may be run across samples of the audio data to similarly identify objects and characteristics of the media content.

Claims

1. A method, comprising:

identifying, by a computing device, a video item comprising a plurality of frames, wherein the video item was recorded using a video camera;

analyzing the plurality of frames of the video item to identify visual objects in the video item;

determining a plurality of keywords representative of the visual objects in the video item;

determining weights for the plurality of keywords based on a frequency of appearance of corresponding visual objects in the analyzed plurality of frames, wherein a weight for a respective keyword of the plurality of keywords increases corresponding to a higher frequency of appearance of a corresponding visual object in the analyzed plurality of frames of the video item;

identifying an audio item that is related to one or more keywords of the plurality of keywords associated with higher weights than other keywords of the plurality of keywords, wherein identifying the audio item comprises;

providing a plurality of audio items that are related to the one or more keywords associated with higher weights than other keywords of the plurality of keywords for selection by a user, and

receiving an indication of a user selection of the audio item from the plurality of audio items; and

providing the audio item for playback, wherein the playback of the audio item is concurrent with playback of the video item.

2. The method of claim 1, further comprising:

determining, based on the identified visual objects in the video item, a geographic location depicted in the video item, wherein the audio item is identified based on the geographic location.

3. The method of claim 2, further comprising:

identifying that the visual objects in the video item comprise a landmark, wherein the geographic location is determined based on the identified landmark.

4. The method of claim 2, further comprising:

identifying that the visual objects in the video item comprise a building, wherein the geographic location is determined based on the identified building.

5. The method of claim 2, further comprising:

identifying that the visual objects in the video item comprise a sign, wherein the geographic location is determined based on the identified sign.

6. The method of claim 2, further comprising:

identifying the visual objects in the video item comprise text, wherein the geographic location is determined based on the identified text.

7. The method of claim 1, further comprising:

determining, based on the visual objects in the video item, a time period during which the video item was recorded, wherein the audio item is identified based on the time period during which the video item was recorded.

8. The method of claim 7, further comprising:

identifying the visual objects in the video item comprise a time-keeping device, wherein the time period during which the video item was recorded is determined based on the identified time-keeping device.

9. The method of claim 7, further comprising:

identifying the visual objects in the video item comprise a celestial body, wherein the time period during which the video item was recorded is determined based on the identified celestial body.

10. A non-transitory computer-readable medium that stores instructions that, responsive to execution by a computing device, cause the computing device to perform operations comprising:

identifying, by the computing device, a video item comprising a plurality of frames;

analyzing the plurality of frames of the video item to identify visual objects in the video item;

determining a plurality of keywords representative of the visual objects in the video item;

determining weights for the plurality of keywords based on a frequency of appearance of corresponding visual objects in the analyzed plurality of frames, wherein a weight for a respective keyword of the plurality of keywords increases corresponding to a higher frequency of appearance of a corresponding visual object in the analyzed plurality of frames of the video item;

identifying an audio item that is related to one or more keywords of the plurality of keywords associated with higher weights than other keywords of the plurality of keywords, wherein identifying the audio item comprises;

providing a plurality of audio items that are related to the one or more keywords associated with higher weights than other keywords of the plurality of keywords for selection by a user, and

receiving an indication of a user selection of the audio item from the plurality of audio items; and

providing the audio item for playback, wherein the playback of the audio item is concurrent with playback of the video item.

11. The non-transitory computer-readable medium of the claim 10, further comprising:

determining, based on the identified visual objects in the video item, a geographic location depicted in the video item, wherein the audio item is identified based on the geographic location.

12. The non-transitory computer-readable medium of the claim 11, further comprising:

identifying that the visual objects in the video item comprise a landmark, wherein the geographic location is determined based on the identified landmark.

13. The non-transitory computer-readable medium of claim 11, further comprising:

identifying that the visual objects in the video item comprise a sign, wherein the geographic location is determined based on the identified sign.

14. The non-transitory computer-readable medium of claim 10, further comprising:

determining, based on the visual objects in the video item, a time period during which the video item was recorded, wherein the audio item is identified based on the time period during which the video item was recorded.

15. A system comprising:

a memory; and

a computing device, coupled to the memory, the computing device to;

identify a video item comprising a plurality of frames, wherein the video item was recorded using a video camera;

analyze the plurality of frames of the video item to identify visual objects in the video item;

determine a plurality of keywords representative of the visual objects in the video item;

determine weights for the plurality of keywords based on a frequency of appearance of corresponding visual objects in the analyzed plurality of frames, wherein a weight for a respective keyword of the plurality of keywords increases corresponding to a higher frequency of appearance of a corresponding visual object in the analyzed plurality of frames of the video item;

identify an audio item that is related to one or more keywords of the plurality of keywords associated with higher weights than other keywords of the plurality of keywords, wherein identifying the audio item comprises;

providing a plurality of audio items that are related to the one or more keywords associated with higher weights than other keywords of the plurality of keywords for selection by a user, and

receiving an indication of a user selection of the audio item from the plurality of audio items; and

provide the audio item for playback, wherein the playback of the audio item is concurrent with playback of the video item.

16. The system of claim 15, the computing device further to:

determine, based on the identified visual objects in the video item, a geographic location depicted in the video item, wherein the audio item is identified based on the geographic location.

17. The system of claim 15, the computing device further to:

determine, based on the visual objects in the video item, a time period during which the video item was recorded, wherein the audio item is identified based on the time period during which the video item was recorded.