Media fingerprinting for content determination and retrieval

• US 9,264,785 B2
• Filed: 04/01/2010
• Issued: 02/16/2016
• Est. Priority Date: 04/01/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Mallinson, Dominic S.
• Primary Examiner(s): MENDOZA, JUNIOR O
• Application Number:
  US12/752,979
• Publication Number:
  US 20110247042A1
• Time in Patent Office: 2,147 Days
• Field of Search: 725 40- 43
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: H04N 21/218 Source of audio or video content , e.g. local disk arrays details of retrieval in video databases G06F16/739
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/2225 Local VOD servers
• CPC Class Codes: H04N 21/232 Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]

Abstract

Audio fingerprinting and other media matching technologies can be used to identify broadcast media, such as television shows and radio broadcasts. A user device can record image, audio, and/or video information and upload information to a matching service that is able to use matching technology to identify the media and provide supplemental content or information to the user. The user might receive information identifying a product in an advertisement, identifying an actor on screen in a movie at a particular time, or other such information. In some embodiments, the user can receive access to a digital copy of the captured media, such as the ability to download a copy of a program in which a user expressed interest. Since a user might capture media information after the point of interest, a device can buffer a window of recently captured media in order to attempt to identify the intended media.

Claims

1. A computer-implemented method of providing supplemental content to a user, comprising:

monitoring, by a server, a broadcast of media content;

separating the broadcast of media content into one or more segments of programs or commercials;

analyzing, using one or more processing devices, the one or more segments of broadcast media content to attempt to determine identifying information for each of the segments;

if the one or more segments have not been previously identified, storing the one or more segments of broadcast media content and the identifying information to at least one database;

if no identifying information is determined, storing the one or more segments of the broadcast media content to the at least one database for subsequent content matching analysis for a predetermined period of time or a number of attempts before being removed from the at least one database;

receiving a request including captured media file information, the captured media file information including a portion of at least one of audio and video data recorded from a video presentation;

comparing, using at least one processing device, the captured media file information to instances of stored media data and the one or more segments of the stored broadcast media content to attempt to locate a first match for the captured media file information, wherein the first match of an instance of stored media data corresponds to identified video content;

processing the captured media file information, using at least one processing device, to determine additional textual content if no match is located with one of the instances of stored media data or the one or more segments of stored broadcast media content, and comparing the additional textual content to instances of stored textual content to locate a second match for the captured media file information, wherein the second match of an instance of stored textual data corresponds to the identified video content;

determining, using at least one processing device, supplemental information for the identified video content if the first match or second match is located; and

providing at least one of the determined supplemental information or additional textual content in response to the received request.

2. The computer-implemented method of claim 1, wherein the comparing is performed using at least one of an audio fingerprinting process or an image recognition process.

3. The computer-implemented method of claim 1, wherein the captured media file information comprises a portion of at least one of audio and video data captured for a first period of time before a user action triggered generation of the request.

4. The computer-implemented method of claim 3, wherein the captured media file information further comprises a portion of at least one of audio and video data captured for a second period of time after the user action triggered generation of the request.

5. The computer-implemented method of claim 1, wherein the determined supplemental information is provided to a device other than a user device that recorded the captured media file information.

6. The computer-implemented method of claim 1, wherein the determined supplemental information includes at least one of advertising information, product information, trivia information, purchasing information, download information, a media file, or an address or link to related content.

7. The computer-implemented method of claim 1, further comprising:

prompting a user device when more than one potential match is determined.

8. The computer-implemented method of claim 1, wherein comparing includes determining a timestamp received with the request, and determining video content broadcast at substantially that time.

9. The computer-implemented method of claim 1, further comprising:

storing information including at least one of the captured media file information or the determined additional information; and

if no match is located using the determined additional textual content with one of the instances of stored textual content;

analyzing the stored information to determine supplemental information to provide for a future request that matches the stored information; and

including the stored information in the instances of stored media data.

10. The computer-implemented method of claim 1, wherein the one or more segments of the broadcast media content correspond to advertisements between program segments.

11. The computer-implemented method of claim 1, wherein the determined supplemental information includes at least one of identities of actors or locations included in a program.

12. The computer-implemented method of claim 1, wherein the request includes a geographic location, and wherein the geographic location is used in comparing the captured media file information to the one or more segments of the broadcast media content.

13. The computer-implemented method of claim 1, wherein the media content comprises audio and video data.

14. A system for providing supplemental content to a user, comprising:

at least one processor; and

memory including instructions that, when executed by the processor, cause the system to;

monitor a broadcast of media content;

separate the broadcast of media content into one or more segments of programs or commercials;

analyze the one or more segments of broadcast media content to attempt to determine identifying information for each of the segments;

if the one or more segments have not been previously identified, store the one or more segments of broadcast media content and the identifying information to at least one database;

if no identifying information is determined, store the one or more segments of the broadcast media content to the at least one database for subsequent content matching analysis for a predetermined period of time or a number of attempts before being removed from the at least one database;

receive a request including captured media file information, the captured media file information including a portion of at least one of audio and video data recorded from a video presentation;

compare the captured media file information to instances of stored media data and the one or more segments of the stored broadcast media content to attempt to locate a first match for the captured media file information, wherein the first match of an instance of stored media data or stored broadcast media content corresponds to identified video content;

process the captured media file information to determine additional textual content if no match is located with one of the instances of stored media data or the one or more segments of stored broadcast media content, and compare the additional textual content to instances of stored textual content to locate a second match for the captured media file information, wherein the second match of an instance of stored textual data corresponds to the identified video content;

determine supplemental information for the identified video content if the first match or the second match is located; and

provide at least one of the determined supplemental information or additional textual content in response to the received request.

15. The system of claim 14, wherein the captured media file information is compared to the instances of stored media data using at least one of an audio fingerprinting process or an image recognition process.

16. The system of claim 14, wherein the determined supplemental information includes at least one of advertising information, product information, trivia information, purchasing information, download information, search results, a media file, or an address or link to related content.

17. The system of claim 14, wherein the captured media file information includes a portion of at least one of audio and video data recorded over a period of time between about 5 seconds and about 20 seconds.

18. A device for identifying video content, comprising:

a processor;

at least one user input capture element operable to receive at least one user input;

at least one media capture element operable to capture at least one of audio and video media information; and

memory including instructions that, when executed by the processor, cause the device to;

while in a capture mode of operation, continually capture media information corresponding to a presentation of media content using the at least one media capture element, the device buffering the captured media information for at least a recent specified period of time, the buffered media information continually updating with newly captured media information and deleting the buffered media information captured outside the specified period of time;

process the captured media information to determine additional textual content from the at least one of audio and video information;

in response to activation of the at least one user input, send a request to obtain supplemental information corresponding to the presentation of media content based at least in part upon, textual content, the request including a portion of the buffered media information, wherein the supplemental information is determined by a server monitoring a broadcast of media content separating the broadcast of media content into one or more segments of programs or commercials;

analyze the one or more segments of broadcast media content to attempt to determine identifying information for each of the segments;

if the one or more segments have not been previously identified, storing the one or more segments of broadcast media content and the identifying information to at least one database;

if no identifying information is determined, store the one or more segments of the broadcast media content to the at least one database for subsequent content matching analysis for a predetermined period of time or a number of attempts before being removed from the at least one database;

compare identified contents of the captured media information to instances of stored media data and one or more segments of stored broadcast media content to attempt to locate a match; and

in response to receiving supplemental information corresponding to the presentation of media content if the match is located, provide at least a portion of the supplemental content to the user.

19. The device according to claim 18, wherein the request further includes data based at least in part on captured media information captured for an additional period of time after receiving the at least one user input.