Synchronization of user interactive events with on-screen events during playback of multimedia stream

• US 9,043,829 B2
• Filed: 10/07/2009
• Issued: 05/26/2015
• Est. Priority Date: 10/07/2009
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Harp, Gregory O., Rushing, James D., Klein, Craig Arthur
• Primary Examiner(s): Featherstone, Mark D
• Application Number:
  US12/575,017
• Publication Number:
  US 20110081965A1
• Time in Patent Office: 2,057 Days
• Field of Search: None
• US Class Current: 725/32
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/338 using television networks
• CPC Class Codes: A63F 13/88 Mini-games executed independently while main games are being loaded
• CPC Class Codes: A63F 2300/409 Data transfer via television network
• CPC Class Codes: A63F 2300/6036 for offering a minigame in combination with a main game
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A method includes receiving, at a multimedia device, a data stream representing multimedia content, the data stream comprising first timing information for the multimedia content, and metadata representing interactive event information associated with the multimedia content, the metadata comprising second timing information for the interactive event information. During the playback of the multimedia content, a first user interactive event is identified that is associated with a first time point in the playback of the multimedia content based on the interactive event information, and the first user interactive event is performed concurrent with the first time point in the playback of the multimedia content.

Claims

1. A method comprising:

receiving, at a multimedia device, a data stream representing multimedia content, the data stream comprising a first plurality of timing indicators for the multimedia content embedded within and distributed throughout the data stream;

receiving, at the multimedia device, metadata representing interactive event information associated with the multimedia content, the metadata comprising a second plurality of timing indicators for the interactive event information;

processing, at the multimedia device, the data stream to provide playback of the multimedia content to a user; and

during the playback of the multimedia content;

providing, at the multimedia device, progress of the playback of the multimedia content, wherein the progress of the playback of the multimedia content comprises timing indicators of the first plurality of timing indicators;

identifying, at the multimedia device, a first user interactive event that is associated with a first time point in the playback of the multimedia content based on the interactive event information;

determining, from the progress of the playback of the multimedia content, when the playback of the multimedia content is approaching the first time point;

determining, at the multimedia device, a subset of the interactive event information that is associated with the first time point based on timing indicators of the second plurality of timing indicators in response to determining, based on the first plurality of timing indicators, that the playback of the multimedia content is approaching the first time point; and

performing, at the multimedia device, the first user interactive event concurrent with the first time point in the playback of the multimedia content,

wherein synchronization of the first user interactive event and the playback of the multimedia content is maintained when a tuning event or a trickplay event occurs.

2. The method of claim 1,

wherein the identifying comprises identifying the first user interactive event from the subset of the interactive event information, and

wherein the trickplay event includes a pause, a resume, a fast-forward, a reverse, or a combination thereof.

3. The method of claim 1, wherein the first plurality of timing indicators comprises a first set of time stamps interspersed substantially equally throughout the data stream and the second plurality of timing indicators comprises a second set of time stamps interspersed within the metadata.

4. The method of claim 1, wherein the metadata is received while receiving the data stream, and wherein the first plurality of timing indicators distributed throughout the data stream comprises the first plurality of timing indicators equally interspersed throughout the data stream.

5. The method of claim 1, wherein receiving the metadata at the multimedia device comprises accessing, at the multimedia device, the metadata from a portable storage medium.

6. The method of claim 1, further comprising:

providing, via the multimedia device, a video game application concurrent with the playback of the multimedia content, the video game application providing a video game experience emulating a situation presented in the playback of the multimedia content; and

wherein;

the interactive event information comprises game context information for use by the video game application, the game context information associated with multimedia content presented at the first time point; and

the first user interactive event comprises a gaming action of the video game application based on the game context information.

7. The method of claim 1, further comprising:

presenting, via the multimedia device, an on-screen display that overlies the playback of the multimedia content, the on-screen display presenting scoring information associated with the multimedia content; and

wherein;

the interactive event information comprises updated scoring information associated with the multimedia content presented at the first time point; and

the first user interactive event comprises updating the on-screen display to reflect the updated scoring information.

8. The method of claim 1, further comprising:

during the playback of the multimedia content;

identifying, at the multimedia device, a second user interactive event based on the interactive event information that is associated with a second time point in the playback of the multimedia content;

determining, from the progress of playback of the multimedia content, when playback of the multimedia content is approaching the first time point; and

performing, at the multimedia device, the second user interactive event concurrent with the second time point in the playback of the multimedia content.

9. The method of claim 3, wherein the time stamps of the first set and the second set comprise time stamps synchronized to coordinated universal time.

10. The method of claim 4, wherein the metadata is embedded in the data stream.

11. The method of claim 4, wherein receiving the metadata at the multimedia device comprises accessing the metadata from an external source via a network responsive to initiation of the playback of the multimedia content at the multimedia device.

12. A multimedia device comprising:

a first interface to receive a data stream representing multimedia content, the data stream comprising a first plurality of timing indicators for the multimedia content embedded within and distributed by the data stream, timing indicators of the first plurality of timing indicators providing an indication of playback progress of the multimedia content;

a second interface to receive metadata representing interactive event information associated with the multimedia content, the metadata comprising a second plurality of timing indicators for the interactive event information;

a display controller to process the data stream to provide playback of the multimedia content to a user and to provide progress of the playback of the multimedia content as determined from timing indicators of the first plurality of timing indicators; and

a user interaction controller to identify, during the playback of the multimedia content, a first user interactive event that is associated with a first time point in the playback of the multimedia content based on the interactive event information, the user interaction controller to detect progress of playback of the multimedia content approaching the first time point, the user interaction controller to determine a subset of the interactive event information that is associated with the first time point based on timing indicators of the second plurality of timing indicators in response to determining, based on the first plurality of timing indicators, that the playback of the multimedia content is approaching the first time point, and the user interaction controller to control a performance of the first user interactive event concurrent with the first time point in the playback of the multimedia content,

wherein synchronization of the first user interactive event and the playback of the multimedia content is maintained when a tuning event or a trickplay event occurs.

13. The multimedia device of claim 12, wherein the user interaction controller identifies the first user interactive event from the subset of the interactive event information, and wherein the trickplay event includes a pause, a resume, a fast-forward, a reverse, or a combination thereof.

14. The multimedia device of claim 12, wherein the first plurality of timing indicators comprises a first set of time stamps interspersed substantially equally throughout the data stream and the second plurality of timing indicators comprises a second set of time stamps interspersed within the metadata.

15. The multimedia device of claim 12, further comprising:

a portable storage interface to interface with a portable storage medium, the portable storage medium storing the metadata.

16. The multimedia device of claim 12, further comprising:

a storage device to store executable instructions for a video game application; and

a processor to execute the executable instructions to provide video game experience emulating a situation presented in the playback of the multimedia content concurrent with the playback of the multimedia content; and

wherein;

the interactive event information comprises game context information for use by the video game application, the game context information associated with the multimedia content presented at the first time point; and

the first user interactive event comprises a gaming action of the video game application based on the game context information.

17. The multimedia device of claim 12, further comprising:

a storage device to store executable instructions for an on-screen display application; and

a processor to execute the executable instructions to provide an on-screen display that overlies the playback of the multimedia content, the on-screen display presenting scoring information associated with the multimedia content; and

wherein;

the interactive event information comprises updated scoring information associated with the multimedia content presented at the first time point; and

the first user interactive event comprises updating the on-screen display to reflect the updated scoring information.

18. A video game console comprising:

a first interface to receive a data stream representing multimedia content, the data stream comprising a first plurality of timing indicators for the multimedia content, embedded within and distributed by the data stream;

a second interface to receive metadata representing game context information associated with the multimedia content, the metadata comprising a second plurality of timing indicators for interactive event information;

a display controller to process the data stream to provide a playback of the multimedia content and to provide progress of playback of the multimedia content as determined from timing indicators of the first plurality of timing indicators;

a storage component to store a software program comprising a set of executable instructions representative of a video game application; and

a processor to execute the software program to present a video game experience to a user concurrent with the playback of the multimedia content, the video game experience emulating a situation concurrently presented in the playback of the multimedia content;

the set of executable instructions being configured to manipulate the processor to synchronize gaming actions of the video game experience to corresponding time points in the playback of the multimedia content based on the first plurality of timing indicators and the second plurality of timing indicators,

wherein the processor determines, during the playback of the multimedia content, from the progress of the playback of the multimedia content, when the playback of the multimedia content is approaching a first time point associated with a first gaming action, and

wherein synchronization of the first gaming action and the playback of the multimedia content is maintained when a tuning event or a trickplay event occurs.

19. The video game console of claim 18, wherein:

the metadata is stored on a portable storage medium; and

the second interface comprises a portable storage interface to access the metadata from the portable storage medium.

20. A method comprising:

generating, at a multimedia distribution system, a data stream representing multimedia content, the data stream comprising a first plurality of timing indicators for the multimedia content embedded within and distributed by the data stream, timing indicators of the first plurality of timing indicators providing an indication of progress during playback of the multimedia content;

determining, at the multimedia distribution system, a set of time points in playback of the multimedia content that correspond to identified on-screen events in the multimedia content;

determining, at the multimedia distribution system, interactive event information for each of the identified on-screen events;

generating, at the multimedia distribution system, metadata comprising the interactive event information for the identified events, the metadata further comprising a second plurality of timing indicators associating a time point of an identified on-screen event with corresponding interactive event information;

providing the data stream to a multimedia device; and

providing the metadata to the multimedia device,

wherein the multimedia device determines, during the playback of the multimedia content, from the progress of the playback of the multimedia content, when the playback of the multimedia content is approaching the time point of the identified on-screen event, and

wherein synchronization of the interactive event and the playback of the multimedia content is maintained when a tuning event or a trickplay event occurs.

21. The method of claim 20, wherein the interactive event information comprises score update information for use by an on-screen display overlay at the multimedia device.

22. The method of claim 20, wherein the interactive event information comprises game context information for use by a video game application at the multimedia device.