Time-synchronizing a parallel feed of secondary content with primary media content

• US 9,118,951 B2
• Filed: 06/26/2012
• Issued: 08/25/2015
• Est. Priority Date: 06/26/2012
• Status: Active Grant

• Current Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: ARRIS Technology, Inc. (CommScope Holding Company, Inc.)
• Inventors: Gandhi, Bhavan, Fonseca, Benedito J. Jr.
• Primary Examiner(s): Bates, Kevin
• Assistant Examiner(s): McCray, Clarence D
• Application Number:
  US13/533,309
• Publication Number:
  US 20130346631A1
• Time in Patent Office: 1,155 Days
• Field of Search: 709/248, 709/232
• US Class Current: 1/1
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]

Abstract

Disclosed is a method of operating a secondary device in a manner associated with operation of a primary device including obtaining first information corresponding to a media asset being output by the primary device, processing the first information to determine local media-signature information, transmitting the first information for receipt by a server, receiving secondary information from the server, wherein the secondary information includes a plurality of asset-media signatures that respectively correspond to respective portions of the media asset, attempting to determine a time-based correlation between at least one portion of the local media-signature information and at least one of the asset-media signatures, and outputting one or more portions of time-relevant asset streams from the secondary device, the one or more portions being determined at least indirectly based upon the correlation.

Claims

1. A method of operating a secondary device in a manner associated with operation of a primary device, the method comprising:

(a) obtaining first information corresponding to a media asset being output by the primary device;

(b) processing the first information to determine local media-signature information;

(c) transmitting the first information for receipt by a server;

(d) receiving secondary information from the server, wherein the secondary information includes a plurality of asset-media signatures that respectively correspond to respective portions of the media asset;

(e) determining a time-based correlation between at least one portion of the local media-signature information and at least one of the asset-media signatures; and

based at least in part upon the plurality of asset-media signatures, further determining whether the media asset is no longer being output by the primary device; and

(f) outputting one or more portions of time-relevant asset streams from the secondary device, the one or more portions being determined at least indirectly based upon the correlation;

wherein steps (a), (b), and (e) are each repeated periodically, and wherein each repeated step (e) includes, in the secondary device, comparing the first information with a first subset of the plurality of asset-media signatures, the first subset corresponding to one or more time points close to a latest synchronization time point, to identify a match between the first information and one or more of the first subset of the plurality of asset-media signatures at the one or more time points.

2. The method of claim 1 wherein the secondary information includes one or more of links or pointers to the time-relevant asset streams associated with respective portions of the media asset.

3. The method of claim 2 wherein the one or more portions of the time-relevant asset streams are received by the secondary device as a parallel feed.

4. The method of claim 1 wherein the first information includes at least one audio sample.

5. The method of claim 1 further comprising storing the received secondary information including the plurality of asset-media signatures.

6. The method of claim 5 wherein the secondary device is a mobile device selected from the group consisting of:

a cell phone, a personal digital assistant, and a tablet computer device.

7. The method of claim 1 further including during step (e):

if no match is found, comparing the first information to a second subset of asset-media signatures.

8. The method of claim 1 wherein the obtaining is performed by at least one input device of the secondary device, the at least one input device being selected from the group consisting of:

a camera and a microphone.

9. The method of claim 1 wherein the secondary information additionally includes a plurality of data portions respectively identifying respective times at which to perform the obtaining of the first information so as to enhance user privacy and reduce battery consumption.

10. The method of claim 1 wherein an input device of the secondary device is activated so as to result in the obtaining of the first information at during collection intervals identified in the secondary information.

11. The method of claim 1 further comprising determining, based at least in part upon the plurality of asset-media signatures as stored at the secondary device, one or more collection intervals during which to perform the obtaining of the first information.

12. The method of claim 11 wherein an input device of the secondary device is activated so as to result in the obtaining of the first information at the one or more collection intervals.

13. The method of claim 1 wherein the determining step (e) further comprises determining, based at least in part upon the as set-media signatures, at least one of:

whether a first media asset being output by the primary device has changed to a second media asset being output by the primary device, and whether a fault has occurred.

14. The method of claim 1 wherein the received secondary information additionally includes at least one of:

a first sequence of media signatures, a second sequence of microphone signatures, and a third sequence of camera signatures.

15. The method of claim 1:

wherein third information is obtained that includes one or more as set-media signatures that include only audio-sample information; and

wherein the secondary information includes the first sequence of media signatures.

16. The method of claim 1 wherein (a), (b), (c), (d), and (e) are repeated until, and (f) is not performed until, the determining of (e) produces a time correspondence between the at least one portion of the local media signature with the at least one of the asset-media signatures.

17. The method of claim 1 further comprising, following (f):

retrieving future time-relevant asset streams at times determined from the sequence of microphone or camera signatures to obtain updated asset-media signatures.

18. A mobile device comprising:

an input device by which first information corresponding to primary content being output by an external device is periodically and repeatedly obtained, the first information including video or audio information;

a processing device coupled at least indirectly to the input device, the processing device configured to periodically and repeatedly develop local media-signature information based at least in part upon the first information;

a wireless transceiver configured to receive communication signals originating at a server, the communication signals including one or more portions of secondary information that includes a plurality of time-relevant asset-media signatures that respectively correspond to respective portions of the primary content; and

a memory device configured to store one or more portions of the secondary information;

wherein the processing device is further configured to periodically and repeatedly determine a time-based correlation between at least one portion of the media-signature information and at least one of the time-relevant asset-media signatures, wherein each repeated determining includes, in the mobile device, comparing the first information with a first subset of the plurality of time-relevant asset-media signatures, the first subset corresponding to one or more time points close to a latest synchronization time point, to identify a match between the first information and one or more of the first subset of the plurality of time-relevant asset-media signatures at the one or more time points, and based at least in part upon the plurality of time-relevant asset-media signatures, further determining whether the primary content is no longer being output by the primary device;

the processing device being further coupled at least indirectly to each of the wireless transceiver and the memory device; and

an output device by which one or more portions of time-relevant asset streams are output from the mobile device, the one or more portions being determined at least indirectly based upon the correlation.

19. The mobile device of claim 18:

wherein the mobile device is selected from the group consisting of;

a cell phone, a personal digital assistant, a mobile computer device, and a tablet computer device; and

wherein the input device is selected from the group consisting of;

a microphone and a camera.

20. A method of synchronizing secondary content presented on a mobile device with primary content viewable on a primary device, the method comprising:

(a) obtaining first information corresponding to a media asset being output by a primary device;

(b) processing the first information to determine local media-signature information;

(c) transmitting the first information for receipt by a server;

(d) receiving secondary information from the server, wherein the secondary information includes a plurality of asset-media signatures that respectively correspond to respective portions of the media asset;

(e) determining a time-based correlation between at least one portion of the local media-signature information and at least one of the asset-media signatures; and

, based at least in part upon the plurality of asset-media signatures, further determining whether the media asset is no longer being output by the primary device; and

(f) outputting one or more portions of time-relevant asset streams from the mobile device, the one or more portions being determined at least indirectly based upon the correlation;

wherein steps (a), (b), and (e) are each repeated periodically, and wherein each repeated step (e) includes, in the mobile device, comparing the first information with a first subset of the plurality of asset-media signatures, the first subset corresponding to one or more time points close to a latest synchronization time point, to identify a match between the first information and one or more of the first subset of the plurality of asset-media signatures at the one or more time points.

21. A method of synchronizing secondary content presented on a mobile device with primary content viewable on a primary device, the method comprising:

periodically and repeatedly capturing at least one of a video sample and an audio sample using one or more input devices of a mobile device, wherein the at least one of a video sample and an audio sample is taken from a primary device that outputs a media asset;

generating local media signatures from a local media-signature generator on the mobile device;

receiving one or more asset-media signatures related to the media asset, where the asset-media signatures substantially span the time length of the media asset;

storing the one or more asset-media signatures on the mobile device;

periodically and repeatedly correlating the local media signatures against the asset-media signatures wherein each repeated correlation includes, in the mobile device, comparing one or more of the local media signatures with a first subset of the one or more asset-media signatures, the first subset corresponding to one or more time points close to a latest synchronization time point, to identify a match between the one or more of the local media signatures and one or more of the first subset of the one or more asset-media signatures at the one or more time points;

establishing a time reference based on the match between the local media signatures and the asset-media signatures; and

periodically updating the time reference, and based at least in part upon the one or more asset-media signatures, determining whether the primary content is no longer being output by the primary device.