Method and device for generating and detecting a fingerprint functioning as a trigger marker in a multimedia signal

• US 9,066,114 B2
• Filed: 01/10/2014
• Issued: 06/23/2015
• Est. Priority Date: 07/11/2003
• Status: Active Grant

• Current Assignee: Roku, Inc.
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Oostveen, Job Cornelis, Ten Kate, Warner Rudolph Theophile, Denissen, Adrianus Johannes Maria, Roberts, David K.
• Primary Examiner(s): LEWIS, LISA C
• Application Number:
  US14/151,879
• Publication Number:
  US 20140130077A1
• Time in Patent Office: 529 Days
• Field of Search: 380/204, 380/54, 713/150, 713176-180, 725/32, 725 34- 35, 725/86
• US Class Current: 1/1
• CPC Class Codes: G06F 21/6227 where protection concerns the structure of data, e.g. records, types, queries
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/433 Content storage operation, e.g. storage operation in response to a pause request, caching operations
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4586 Content update operation triggered locally, e.g. by comparing the version of software modules in a DVB carousel to the version stored locally deployment, distribution, installation, update of software G06F8/65; error detection or correction of the data by redundancy during software upgrading G06F11/1433; arrangements for updating broadcast information or broadcast-related information H04H60/25
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

A device is configured to perform a method that detects a trigger marker for an action corresponding to a segment of a multimedia signal. A fingerprint is generated based on the segment of the multimedia signal at a trigger time point. The generated fingerprint is stored in a database and communicated to the device. During playback of the multimedia signal, fingerprints of segments of the multimedia signal are generated and matched against fingerprints in the database. When a match is found, one or more associated actions for the segment are retrieved by the device. The trigger time point may be determined as a time point near or at the segment of the multimedia signal with the matched fingerprint. In this way, trigger markers for actions may be enabled without modifying the multimedia signal.

Claims

1. A method comprising:

determining a plurality of trigger fingerprints from content being played back on a playback device, each trigger fingerprint identifying a corresponding trigger time point of a sequence of trigger time points in the content and at which trigger time point at least one corresponding action is to be triggered, the determining being performed by a processor;

accessing a database that includes a plurality of reference fingerprints, previously derived from the content, and a plurality of reference actions, each reference fingerprint identifying at least one corresponding reference action to be performed at a reference time point identified by the reference fingerprint;

identifying the corresponding reference action by obtaining a match in the database between a trigger fingerprint among the plurality of trigger fingerprints and a reference fingerprint among the plurality of reference fingerprints; and

performing a reference action that corresponds to the reference fingerprint on the playback device.

2. The method of claim 1, wherein:

different corresponding actions are to be performed at different trigger time points in the sequence of trigger time points in the content.

3. The method of claim 1, wherein:

the trigger fingerprint is determined from a segment of the content, the segment ending at the trigger time point at which the reference action is to be performed.

4. The method of claim 1, wherein:

the trigger fingerprint is determined from a segment of the content, the segment beginning at the trigger time point at which the reference action is to be performed.

5. The method of claim 1, wherein:

the trigger fingerprint is determined from a segment of the content, the segment beginning or ending at a predetermined distance before or after the trigger time point at which the reference action is to be performed.

6. The method of claim 1, wherein:

the reference action corresponds to multiple reference fingerprints that include the reference fingerprint.

7. The method of claim 1, wherein:

multiple reference actions correspond to the reference fingerprint and are to be performed at the trigger time point; and

the performing of the reference action is part of performing the multiple reference actions that correspond to the reference fingerprint.

8. A system comprising:

a fingerprint detector configured to determine a plurality of trigger fingerprints from content being played back on a playback device, each trigger fingerprint identifying a corresponding trigger time point of a sequence of trigger time points in the content and at which trigger time point at least one corresponding action is to be triggered; and

a processor configured to;

access a database that includes a plurality of reference fingerprints, previously derived from the content, and a plurality of reference actions, each reference fingerprint identifying at least one corresponding reference action to be performed at a reference time point identified by the reference fingerprint;

identify the corresponding reference action by obtaining a match in the database between a trigger fingerprint among the plurality of trigger fingerprints and a reference fingerprint among the plurality of reference fingerprints; and

perform a reference action that corresponds to the reference fingerprint on the playback device.

9. The system of claim 8, wherein:

the reference action corresponds to multiple reference fingerprints that include the reference fingerprint.

10. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

determining a plurality of trigger fingerprints from content being played back on a playback device, each trigger fingerprint identifying a corresponding trigger time point of a sequence of trigger time points in the content and at which trigger time point at least one corresponding action is to be triggered;

accessing a database that includes a plurality of reference fingerprints, previously derived from the content, and a plurality of reference actions, each reference fingerprint identifying at least one corresponding reference action to be performed at a reference time point identified by the reference fingerprint;

identifying the corresponding reference action by obtaining a match in the database between a trigger fingerprint among the plurality of trigger fingerprints and a reference fingerprint among the plurality of reference fingerprints; and

performing a reference action that corresponds to the reference fingerprint on the playback device.

11. A method comprising:

accessing segments of content, a segment among the segments being associated with a corresponding action to be triggered during playback of the segment;

generating a reference fingerprint of the segment associated with the corresponding action, the generating being performed by a processor;

assigning the reference fingerprint of the segment to the corresponding action that is to be triggered during playback of the segment of the content;

providing access to the reference fingerprint of the segment to a playback device configured to match the reference fingerprint to a playback fingerprint derived from the segment by the playback device; and

providing the identifier of the corresponding action to the playback device based on the reference fingerprint matching the playback fingerprint, the provided identifier causing the playback device to trigger the corresponding action during the playback of the segment.

12. The method of claim 11, wherein:

the assigning of the reference fingerprint to the identifier of the corresponding action includes storing the reference fingerprint with the identifier of the corresponding action in a database; and

the playback device is configured to search the database to match the reference fingerprint to the playback fingerprint.

13. The method of claim 11, wherein:

the playback device is configured to trigger the corresponding action at a time point that is a predetermined distance from a start of the segment of the multimedia content.

14. The method of claim 11, wherein:

the playback device is configured to trigger the corresponding action at a time point that is a predetermined distance from an end of the segment of the multimedia content.

15. The method of claim 11, wherein:

the playback device is further configured to present the corresponding action and await confirmation of the corresponding action in response to the playback device triggering the corresponding action during the playback of the segment of the content.

16. The method of claim 11, wherein:

the corresponding action for the reference fingerprint and for the segment of the content includes retrieving and displaying on a display additional information selected from a group consisting of;

biography data for an actor that appears in the segment of the content,

a link to a website,

a local weather forecast, and

a local commercial.

17. The method of claim 11, wherein:

the corresponding action for the reference fingerprint and for the segment of the content includes retrieving and playing via a speaker additional information selected from a group consisting of;

biography data of an actor that appears in the segment of the content,

a local weather forecast, and

a local commercial.

18. The method of claim 11, wherein:

the corresponding action for the reference fingerprint and for the segment of the content is selected from a group consisting of;

replacing the content with different content for a period of time,

pausing playback of the content,

stopping playback of the content,

executing a control command, and

preparing the playback device to receive user input.

19. The method of claim 11, wherein:

the content includes at least one of audio content or video content.

20. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing segments of content, a segment among the segments being associated with a corresponding action to be triggered during playback of the segment;

generating a reference fingerprint of the segment associated with the corresponding action;

assigning the reference fingerprint of the segment to the corresponding action that is to be triggered during playback of the segment of the content;

providing access to the reference fingerprint of the segment to a playback device configured to match the reference fingerprint to a playback fingerprint derived from the segment by the playback device; and

providing the identifier of the corresponding action to the playback device based on the reference fingerprint matching the playback fingerprint, the provided identifier causing the playback device to trigger the corresponding action during the playback of the segment.

21. A system comprising:

an input module configured to access segments of content, a segment among the segments being associated with a corresponding action to be triggered during playback of the segment; and

a fingerprint module implemented in a processor and configured to;

generate a reference fingerprint of the segment associated with the corresponding action;

assign the reference fingerprint of the segment to the corresponding action that is to be triggered during playback of the segment of the content;

provide access to the reference fingerprint of the segment to a playback device configured to match the reference fingerprint to a playback fingerprint derived from the segment by the playback device; and

provide the identifier of the corresponding action to the playback device based on the reference fingerprint matching the playback fingerprint, the provided identifier causing the playback device to trigger the corresponding action during the playback of the segment.

22. The system of claim 21, wherein:

the system further comprises a database;

the fingerprint module is configured to store the reference fingerprint with the identifier of the corresponding action in the database; and

the playback device is configured to search the database to match the reference fingerprint to the playback fingerprint.

23. The system of claim 21, wherein:

the corresponding action is to be triggered by the playback device at a time point that is a predetermined distance from a start of the segment of the multimedia content.

24. The system of claim 21, wherein:

the corresponding action is to be triggered by the playback device at a time point that is a predetermined distance from an end of the segment of the multimedia content.

25. The system of claim 21, wherein:

the corresponding action is to be presented by the playback device in response to the playback device triggering the corresponding action during the playback of the segment of the content, the playback device being further configured to await confirmation of the corresponding action prior to executing the corresponding action.