Synchronizing out-of-band content with a media stream

• US 10,231,033 B1
• Filed: 07/30/2015
• Issued: 03/12/2019
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bumgarner, William M., Seely, Blake, Oklota, Christopher L., Taylor, Dale A., Priyadarshan, Eswar
• Primary Examiner(s): Mendoza, Junior O
• Application Number:
  US14/814,230
• Time in Patent Office: 1,321 Days
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/26258 for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list retrieval of multimedia data based on playlists G06F16/40
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/4825 using a list of items to be played back in a given order, e.g. playlists
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

Receiving and processing out-of-band (OOB) data along with a media stream is disclosed. The received OOB data can be extracted from an OOB data stream and correlated with the media stream using timestamps. Display and other operations can then be performed when a timestamp of the OOB data matches a current timestamp of the media stream. In addition, timestamp notification requests can be received and processed such that when the current timestamp of the media stream matches a requested timestamp, a notification can be transmitted, and data can be received in response to the notification.

Claims

1. A method of integrating an out-of-band (OOB) data stream with a first media stream including one or more events and a first plurality of timestamps of a first type, the method comprising:

determining whether the first media stream contains event metadata comprising information about the one or more events in the first media stream;

in accordance with a determination that the first media stream does not contain any of the event metadata for the one or more events in the first media stream;

transmitting a request for the OOB data stream;

after transmitting the request for the OOB data stream, receiving the OOB data stream that contains event data and a second plurality of timestamps of a second type, different than the first type;

matching the OOB data stream with the first media stream using a media identification (ID) of the first media stream;

extracting OOB event metadata from the OOB data stream, the OOB event metadata including information about the one or more events in the first media stream;

converting the second plurality of timestamps into a plurality of converted timestamps of the first type; and

correlating the OOB event metadata with the first media stream using the plurality of converted timestamps of the OOB event metadata and the first plurality of timestamps of the first media stream; and

in accordance with a determination that the first media stream contains the event metadata for the one or more events in the media stream, forgoing transmitting the request for the OOB data stream.

2. The method of claim 1, further comprising performing an operation when a timestamp of the plurality of converted timestamps of the OOB event metadata matches a current timestamp of the first media stream.

3. The method of claim 2, wherein the operation comprises retrieving data and performing display operations in accordance with the data.

4. The method of claim 1, wherein:

the determination that the first media stream does not contain any of the event metadata for the one or more events in the first media stream includes detecting an instruction that was transmitted with the first media stream to request the OOB data stream.

5. The method of claim 1, wherein:

the first media stream is received from a first source;

the OOB data stream is received from a second source; and

the media ID of the first media stream is received from a server, different from the first and second source.

6. The method of claim 1, further comprising:

after correlating the OOB event metadata with the first media stream, storing the OOB event metadata separately from the first media stream.

7. The method of claim 1, further comprising:

after correlating the OOB event metadata with the first media stream, embedding the OOB event metadata within the first media stream.

8. A device for integrating an out-of-band (OOB) data stream with a first media stream including one or more events and a first plurality of timestamps of a first type, the device comprising:

a network interface configured for receiving the OOB data and the first media stream; and

a processor communicatively coupled to the network interface and capable of;

determining whether the first media stream contains event metadata comprising information about the one or more events in the first media stream;

in accordance with a determination that the first media stream does not contain any of the event metadata for the one or more events in the first media stream;

transmitting a request for the OOB data stream;

after transmitting the request for the OOB data stream, receiving the OOB data stream that contains event data;

matching the OOB data stream with the first media stream using a media identification (ID) of the first media stream and a second plurality of timestamps of a second type, different than the first type;

extracting OOB event metadata from the OOB data stream, the OOB event metadata including information about the one or more events in the first media stream;

converting the second plurality of timestamps into a plurality of converted timestamps of the first type; and

correlating the OOB event metadata with the first media stream using the plurality of converted timestamps of the OOB event metadata and the first plurality of timestamps of the first media stream; and

in accordance with a determination that the first media stream contains the event metadata for the one or more events in the media stream, forgoing transmitting the request for the OOB data stream.

9. The device of claim 8, the processor further capable of performing an operation when a timestamp of the plurality of converted timestamps of the OOB event metadata matches a current timestamp of the first media stream.

10. The device of claim 8, wherein:

the determination that the first media stream does not contain any of the event metadata for the one or more events in the first media stream includes detecting an instruction that was transmitted with the first media stream to request the OOB data stream.

11. The device of claim 8, wherein:

the first media stream is received from a first source;

the OOB data stream is received from a second source; and

the media ID of the first media stream is received from a server, different from the first and second source.

12. The device of claim 8, wherein the processor further capable of:

after correlating the OOB event metadata with the first media stream, storing the OOB event metadata separately from the first media stream.

13. The device of claim 8, wherein the processor further capable of after correlating the OOB event metadata with the first media stream, embedding the OOB event metadata within the first media stream.

14. A non-transitory computer-readable storage medium containing instructions that, when executed, perform a method for integrating an out-of-band (OOB) data stream with a first media stream including one or more events and a first plurality of timestamps of a first type, the method comprising:

determining whether the first media stream contains event metadata comprising information about the one or more events in the first media stream;

in accordance with a determination that the first media stream does not contain any of the event metadata for the one or more events in the first media stream;

transmitting a request for the OOB data stream;

after transmitting the request for the OOB data stream, receiving the OOB data stream that contains event data and a second plurality of timestamps of a second type, different than the first type;

matching the OOB data stream with the first media stream using a media identification (ID) of the first media stream;

extracting OOB event metadata from the OOB data stream, the OOB event metadata including information about the one or more events in the first media stream;

converting the second plurality of timestamps into a plurality of converted timestamps of the first type; and

correlating the OOB event metadata with the first media stream using the plurality of converted timestamps of the OOB event metadata and the first plurality of timestamps of the first media stream; and

in accordance with a determination that the first media stream contains the event metadata for the one or more events in the media stream, forgoing transmitting the request for the OOB data stream.

15. The non-transitory computer-readable storage medium of claim 14, the method further comprising performing an operation when a timestamp of the plurality of converted timestamps of the OOB event metadata matches a current timestamp of the first media stream.

16. The non-transitory computer-readable storage medium of claim 14, wherein:

the determination that the first media stream does not contain any of the event metadata for the one or more events in the first media stream includes detecting an instruction that was transmitted with the first media stream to request the OOB data stream.

17. The non-transitory computer-readable storage medium of claim 14, wherein:

the first media stream is received from a first source;

the OOB data stream is received from a second source; and

the media ID of the first media stream is received from a server, different from the first and second source.

18. The non-transitory computer-readable storage medium of claim 14, the method further comprising:

after correlating the OOB event metadata with the first media stream, storing the OOB event metadata separately from the first media stream.

19. The non-transitory computer-readable storage medium of claim 14, the method further comprising:

after correlating the OOB event metadata with the first media stream, embedding the OOB event metadata within the first media stream.