SYSTEMS AND METHODS FOR AUTOMATICALLY SUGGESTING MEDIA ACCOMPANIMENTS BASED ON IDENTIFIED MEDIA CONTENT

• US 20150169747A1
• Filed: 12/12/2013
• Published: 06/18/2015
• Est. Priority Date: 12/12/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hume, Thomas Weedon, Lerda, Flavio, Koser, Mikkel Crone
• Granted Patent: US 10,467,287 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G06F 16/687 using geographical or spatial information, e.g. location
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 21/8113 comprising music, e.g. song in MP3 format

Abstract

The disclosed technology includes automatically suggesting audio, video, or other media accompaniments to media content based on identified objects in the media content. Media content may include images, audio, video, or a combination. In one implementation, one or more images representative of the media content may be extracted. A visual search may be run across the images to identify objects or characteristics present in or associated with the media content. Keywords may be generated based on the identified objects and characteristics. The keywords may be used to determine suitable audio tracks to accompany the media content, for example by performing a search based on the keywords. The determined tracks may be presented to a user, or automatically arranged to match the media content. In another implementation, an aural search may be run across samples of the audio data to similarly identify objects and characteristics of the media content.

Claims

1. A method, comprising:

receiving, at a computing device, one or more images embodying content containing representations of one or more objects;

determining, by the computing device, an identity of at least one object of the one or more objects;

determining, based on the identified at least one object, one or more keywords representative of the one or more images;

determining, based on the one or more keywords, one or more audio tracks for presentation with the one or more images; and

outputting an indication of the determined one or more audio tracks.

2. The method of claim 1, further comprising:

associating one or more weights with at least one keyword of the one or more keywords, the determining the one or more audio tracks further based on the associated one or more weights.

3. The method of claim 2, further comprising:

determining the one or more weights for the at least one keyword based on a frequency of occurrence of the identified at least one object corresponding to the at least one keyword in the one or more images.

4. The method of claim 1, further comprising:

determining, based on the identified at least one object, one or more languages associated with the one or more images, the determining the one or more audio tracks further based on the determined one or more languages.

5. The method of claim 4, the one or more objects including textual characters, the determining the one or more associated languages based on identifying textual characters in the one or more images.

6. The method of claim 1, further comprising:

determining, based on the identified at least one object, one or more locations associated with the one or more images, the determining the one or more audio tracks further based on the determined locations.

7. The method of claim 6, the one or more objects including landmarks, the determining the one or more associated locations based on identifying landmarks in the one or more images.

8. The method of claim 6, the one or more objects including buildings, the determining the one or more associated locations based on identifying buildings in the one or more images.

9. The method of claim 6, the one or more objects including signs, the determining the one or more associated locations based on identifying signs in the one or more images.

10. The method of claim 6, the one or more objects including textual characters, the determining the one or more associated locations based on identifying textual characters in the one or more images.

11. The method of claim 1, further comprising:

determining, based on the identified at least one object, one or more time periods associated with the one or more images, the determining the one or more audio tracks further based on the determined time periods.

12. The method of claim 11, the one or more objects including time-keeping devices, the determining the one or more associated time periods based on identifying time-keeping devices in the one or more images.

13. The method of claim 11, the one or more objects including time displays, the determining the one or more associated time periods based on identifying time displays in the one or more images

14. The method of claim 11, the one or more objects including celestial bodies, the determining the one or more associated time periods based on identifying celestial bodies in the one or more images.

15. A computer-readable medium that stores instructions that, when executed by at least one processor in a system, cause the system to perform a method comprising:

receiving, at the at least one processor, a media sequence comprising audio data, the audio data containing representations of one or more objects;

determining, by the at least one processor, an identity of at least one object of the one or more objects;

determining, based on the identified at least one object, one or more keywords representative of the media sequence;

determining, based on the one or more keywords, one or more audio tracks for presentation with the media sequence; and

outputting an indication of the determined one or more audio tracks.

16. The computer-readable medium of claim 15, the method further comprising:

determining, based on a frequency of occurrence of the identified audio objects in the media sequence corresponding to at least one keyword of the one or more keywords, one or more weights for the at least one keyword; and

associating the one or more weights with the at least one keyword.

17. The computer-readable medium of claim 15, the one or more identified objects including spoken words, the one or more determined keywords including a form of at least one of the identified spoken words.

18. The computer-readable medium of claim 17, the method further comprising determining the identified spoken words correspond to lyrics of a song, the determined one or more audio tracks including a version of the song.

19. The computer-readable medium of claim 15, the method further comprising:

determining, based on the identified at least one object, one or more languages associated with the media sequence, the determining the one or more audio tracks further based on the determined one or more languages.

20. The computer-readable medium of claim 19, the one or more objects including spoken words, the determining the one or more associated languages based on identifying spoken words in the media sequence.

21. The computer-readable medium of claim 15, the method further comprising:

determining, based on the identified at least one object, one or more locations associated with the media sequence, the determining the audio track further based on the determined locations.

22. The computer-readable medium of claim 21, the one or more objects including spoken words, the determining the one or more associated locations based on identifying spoken words in the media sequence.

23. The computer-readable medium of claim 21, the one or more objects including reference signals, the determining the one or more associated locations based on identifying reference signals in the media sequence.

24. The computer-readable medium of claim 15, the method further comprising:

determining, based on the identified at least one object, one or more time periods associated with the media sequence, the determining the one or more audio tracks further based on the determined time periods.

25. A system comprising:

at least one memory operatively coupled to at least one processor and configured for storing data and instructions that, when executed by the at least one processor, cause the system to;

receive, at the at least one processor, a media sequence containing representations of one or more objects;

determine, by the at least one processor, an identity of at least one object of the one or more objects;

determine, based on the identified at least one object, one or more keywords representative of the media sequence;

determine, based on the one or more keywords, one or more audio tracks for presentation with the media sequence; and

output an indication of the determined one or more audio tracks.

26. The system of claim 25, the data and instructions further causing system to:

receive an indication of a collection of audio tracks stored at least partially on the least one memory, the determined one or more audio tracks being from the collection of audio tracks.

27. The system of claim 25, the media sequence received over a network from a computing device, the data and instructions further causing system to:

receive an indication of a collection of audio tracks stored at least partially on a memory associated with the computing device, the determined one or more audio tracks being from the collection of audio tracks.

28. The system of claim 25, the data and instructions further causing system to:

receive an indication of a collection of audio tracks associated with an online service, the determined one or more audio tracks from the collection of audio tracks.

29. The system of claim 28, the determined one or more audio tracks from a subset of the collection of audio tracks, the subset associated with a user account of the online service.

30. The system of claim 25, the media sequence associated with one or more metadata tags, the data and instructions further causing system to determine, based on the identified at least one object and the metadata tags, one or more of a language, location, and time period associated with the media sequence.

31. The system of claim 25, the media sequence associated with one or more metadata tags, the determining the identity of the objects based at least partially on the metadata tags.