System and Method for Audio Identification and Metadata Retrieval

• US 20090041418A1
• Filed: 08/08/2007
• Published: 02/12/2009
• Est. Priority Date: 08/08/2007
• Status: Active Grant

• Current Assignee: Saturn Licensing LLC (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hardacker, Robert, Read, Christopher Jensen, Candelore, Brant
• Granted Patent: US 9,996,612 B2
• Time in Patent Office: Days
• US Class Current: 386/341
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: H04N 21/23106 involving caching operations prefetching while addressing of a memory level in which the access to the desired data or data block requires associative addressing means within memory systems or architectures G06F12/0862; caching at an intermediate stage in a data network H04L67/568
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4398 involving reformatting operations of audio signals details of audio signal transcoding G10L19/173
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

Herein, a display device, system and method are described for identifying digital content through comparison of one or more audio thumbnails when the digital content is provided without metadata. According to one embodiment, a display device comprises a display, audio conversion logic and a transceiver. The audio conversion logic is adapted to produce one or more audio thumbnails, which are compact representations of a snippet of audio from that incoming digital content without the metadata. The transceiver is adapted to transmit the one or more audio thumbnails in order to (i) retrieve metadata associated with the digital content from an external source separate from a source of the digital content and (ii) display information provided by the metadata on the display.

Claims

1. A display device, comprising:

a display;

audio conversion logic to produce one or more audio thumbnails each being a compact representation of a snippet of audio from incoming digital content excluding metadata associated with the digital content; and

a transceiver to transmit the one or more audio thumbnails in order to (i) retrieve metadata associated with the digital content from an external source separate from a source of the digital content and (ii) display information provided by the metadata on the display

2. The display device of claim 1, wherein the transceiver is adapted to retrieve the metadata that, when processed, produces displayable images that are selectable by the viewer and are separate from display images produced from the digital content.

3. The display device of claim 2, wherein the transceiver is adapted to retrieve the metadata that, when processed, produces the displayable images positioned alongside a display area reserved for the display images produced from the digital content.

4. The display device of claim 3, wherein the transceiver is adapted to retrieve the metadata that, when processed, produces the displayable images as background images along an outer border of the display area while the display images of the digital content are displayed as foreground images.

5. The display device of claim 1, wherein the transceiver is adapted to retrieve the metadata including information pertaining to context within a scene within an audio-video program of the digital content, the metadata, when processed, produces displayable images related to the scene within the audio-video program.

6. The display device of claim 5, wherein the information pertaining to context within the scene is information identifying a particular location for the scene so that the metadata, when processed, produces a displayable image that is selectable to provide further information concerning the particular location.

7. The display device of claim 1, wherein the transceiver is adapted to retrieve the metadata that, when processed, produces a displayable image that enables the viewer to purchase the digital content stored in a machine readable medium.

8. The display device of claim 1 further comprising:

audio extraction logic that is adapted to separate digital audio from the digital content into one or more audio snippets and to output the one or more audio snippets to the audio conversion logic.

9. The display device of claim 1, wherein the audio extraction logic may further perform pre-processing operations on the digital audio to normalize the digital audio before generating the one or more audio snippets.

10. A method comprising:

segmenting digital audio from incoming digital content into a plurality of audio snippets, the digital content including video and audio without metadata;

converting the plurality of audio snippets into a plurality of audio thumbnails, each audio thumbnail being a representation of a selected one of the plurality of audio snippets;

using at least one of the audio thumbnails to retrieve metadata associated with the incoming digital content; and

displaying at least a portion of the metadata on a display device during playback of the incoming digital content.

11. The method of claim 10, wherein the segmenting of the digital audio is performed on a time-basis through video frame synchronization.

12. The method of claim 10, wherein the segmenting of the digital audio is performed using local maxima and/or minima of the digital audio as reference points.

13. The method of claim 10, wherein the converting of the plurality of audio snippets into the plurality of audio thumbnails is conducted by performing a Fast Fourier Transform on each of the plurality of audio snippets, each of the plurality of audio thumbnails being a representation of a corresponding audio snippet and having a bit size lesser than the corresponding audio snippet.

14. The method of claim 10, wherein the using of the at least one audio thumbnail to retrieve metadata associated with the incoming digital content comprises:

transmitting the at least one audio thumbnail to a first database to compare the at least one audio thumbnail with pre-stored information;

retrieving the metadata associated with the pre-stored information if a match is detected between the at least one audio thumbnail and the pre-stored information.

15. The method of claim 14, wherein the retrieving of the metadata associated with the pre-stored information including extracting an index associated with the pre-stored information from the first database and using the index to extract the metadata from a second database.

16. The method of claim 10, wherein the displaying of at least a portion of the metadata on the display device during playback of the digital content includes producing a displayable image using the metadata that is displayed with the digital content that enables a viewer to purchase the digital content stored in a machine readable medium.

17. An entertainment system comprising:

a networking device to receive and output content excluding metadata associated with the content; and

a digital device in communication with the networking device, the digital device including (i) audio conversion logic to produce a representation of each snippet of audio from the content received from the networking device, and (ii) a transceiver to transmit the one or more representations in order to retrieve metadata from an external source for subsequent display of the metadata with the content.

18. The entertainment system of claim 17 wherein the networking device is a set-top box and the display device is a television.

19. The entertainment system of claim 18, wherein the digital device further comprises audio extraction logic to segregate audio in digital form as part of the content into a plurality of snippets of audio and the representations being produced as a result of audio-to-text conversion.

20. The entertainment system of claim 19, wherein audio extraction logic of the display device to perform pre-processing operations on the digital audio including decompression and decryption of the digital audio before subsequent conversion into the one or more representations.