System and method for audio identification and metadata retrieval

• US 9,996,612 B2
• Filed: 08/08/2007
• Issued: 06/12/2018
• Est. Priority Date: 08/08/2007
• Status: Active Grant

• Current Assignee: Saturn Licensing LLC (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Candelore, Brant, Hardacker, Robert, Read, Christopher Jensen
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): HASAN, SYED Y
• Application Number:
  US11/835,954
• Publication Number:
  US 20090041418A1
• Time in Patent Office: 3,961 Days
• Field of Search: 386341, 386 E9016, 348 E5103, 348 E7071, 348 E5105, 348 E7061, 379 8814, 434236, 434323, 700 15045, 700 94, 700500, 704200, 7042001, 704270, 704275, 707722, 707899, 707999003, 707999102, 7079991, 709229, 709231, 709235, 713176, 715764, 725 62, 725100, 725131, 725151, 725 86, 725 91, 725110, 725105, 725115, 725 39, 725 40, 725 41, 725 52, 725 53, 725 31, 725 46, 725109, 725136, 4551851
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: H04N 21/23106 involving caching operations prefetching while addressing of a memory level in which the access to the desired data or data block requires associative addressing means within memory systems or architectures G06F12/0862; caching at an intermediate stage in a data network H04L67/568
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4398 involving reformatting operations of audio signals details of audio signal transcoding G10L19/173
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

Herein, a display device, system and method are described for identifying digital content through comparison of one or more audio thumbnails when the digital content is provided without metadata. According to one embodiment, a display device comprises a display, audio conversion logic and a transceiver. The audio conversion logic is adapted to produce one or more audio thumbnails, which are compact representations of a snippet of audio from that incoming digital content without the metadata. The transceiver is adapted to transmit the one or more audio thumbnails in order to (i) retrieve metadata associated with the digital content from an external source separate from a source of the digital content and (ii) display information provided by the metadata on the display.

Claims

1. A system comprising:

at least one video display adapted to receive, from a source of digital content, digital content and to display the digital content, the digital content comprising at least one audio-video (A/V) program including video, the digital content comprising at least one A/V program not including metadata comprising information associated with video in the A/V program;

at least one computer memory that is not a transitory signal and that comprises instructions executable by at least one processor;

at least one processor adapted to access the instructions to;

produce from the A/V program including video at least one audio thumbnail, the at least one audio thumbnail being a representation of audio from digital content;

route the at least one audio thumbnail to a first database different from the source of digital content for comparison of the at least one audio thumbnail to pre-stored audio thumbnails in the first database to determine whether at least one match is found between the at least one audio thumbnail and one of the pre-stored audio thumbnails;

receive metadata associated with the A/V program from a metadata base different from the first database, responsive to at least one match being found between the at least one audio thumbnail and a first pre-stored audio thumbnail in the first database, the metadata being retrieved by an identifier, received by the metadata base from the first database, corresponding to the first pre-stored audio thumbnail;

the video display being adapted to display the metadata with the A/V program, wherein the metadata comprises information identifying a particular location for the scene of the content so that the metadata, when processed, provides information concerning the particular location such that a playback position of the video associated with the audio thumbnail is ascertained by identifying a location of the audio thumbnail within the content, the system further comprising;

a transceiver that is adapted to retrieve the metadata.

2. The system of claim 1, wherein the system includes a transceiver that is adapted to retrieve metadata that, when processed, produces the metadata as background images along an outer border of a display area while the A/V program is displayed as foreground images.

3. The system of claim 1, wherein the system includes a transceiver that is adapted to retrieve metadata that, when processed, produces the one or more displayable images that enable the viewer to purchase the digital content stored in a machine readable medium.

4. The system of claim 1 further comprising:

audio extraction logic that is adapted to separate digital audio from the digital content into one or more audio snippets and to output the one or more audio snippets to the audio conversion logic.

5. The system of claim 4, wherein the audio extraction logic further performs pre-processing operations on the digital audio to normalize the digital audio before generating the one or more audio snippets.

6. The system of claim 1, comprising a transceiver adapted to retrieve the metadata that, when processed, produces one or more displayable images, wherein at least one of the displayable images is a uniform resource locator.

7. The system of claim 1, wherein the video display is adapted to use the metadata to display additional information pertaining to the A/V program and/or links (URLs) to the additional information.

8. The system of claim 1, wherein the at least one computerized management assembly is adapted to:

conduct an initial search of the first database for audio from first programs based on the first televised programs being scheduled to be aired at a predetermined time, the initial search not including audio in the first database from second programs.

9. A method comprising:

receiving, from a source of digital content, digital content and to display the digital content, the digital content comprising at least one audio-video (A/V) program including video but not including metadata comprising information associated with video in the A/V program;

producing from the A/V program including video at least one audio thumbnail, the at least one audio thumbnail being a representation of audio from digital content;

routing the at least one audio thumbnail to a first database different from the source of digital content for comparison of the at least one audio thumbnail to pre-stored audio thumbnails in the first database to determine whether at least one match is found between the at least one audio thumbnail and one of the pre-stored audio thumbnails;

responsive to at least one match being found between the at least one audio thumbnail and a first pre-stored audio thumbnail in the first database, receiving, from a metadata base different from the first database, metadata associated with the A/V program;

the metadata being retrieved from the metadata base using an identifier received from the first database corresponding to the first pre-stored audio thumbnail; and

displaying the metadata with the A/V program, wherein the metadata comprises information pertaining to context within a scene of content including video associated with audio thumbnails, the information comprising information identifying a location for the scene of the content so that the metadata, when processed, provides information concerning the location such that a playback position of the video associated with the at least one audio thumbnail is ascertained by identifying a location of the audio thumbnail within the content.

10. The method of claim 9, wherein segmenting of digital audio is performed on a time-basis through video frame synchronization.

11. The method of claim 9, wherein segmenting of digital audio is performed using local maxima and/or minima of the digital audio as reference points.

12. The method of claim 9, wherein converting audio snippets into a plurality of audio thumbnails is conducted by performing a Fast Fourier Transform on each of the plurality of audio snippets, each of the plurality of audio thumbnails being a representation of a corresponding audio snippet and having a bit size lesser than the corresponding audio snippet.

13. The method of claim 9, wherein displaying of at least a portion of metadata on the display device during playback of digital content includes producing a displayable image using the metadata that is displayed with the digital content that enables a viewer to purchase the digital content stored in a machine readable medium.