Synchronization of haptic effect data in a media stream

• US 10,088,903 B2
• Filed: 03/26/2015
• Issued: 10/02/2018
• Est. Priority Date: 10/16/2007
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Ullrich, Christopher J., Rank, Stephen D., Bakircioglu, Munibe M.
• Primary Examiner(s): Small, Naomi J
• Application Number:
  US14/669,551
• Publication Number:
  US 20150199015A1
• Time in Patent Office: 1,286 Days
• Field of Search: 345156
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 5/18 Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 1/72442 for playing music files
• CPC Class Codes: H04M 19/047 Vibrating means for incoming calls
• CPC Class Codes: H04M 19/048 Arrangements providing optical indication of the incoming call, e.g. flasher circuits
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/81 Monomedia components thereof
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Haptic information in a series of frames of a media file is identified and time stamps corresponding thereto are determined in accordance with information embedded in a tag either preceding or appending the media file. The haptic effect information is automatically synchronized to the media file based on the information in one or more frames of the tag, whereby an actuator outputs the defined haptic effects in the tag in a manner synchronized with corresponding audio and/or video content of the media file.

Claims

1. A method, comprising:

receiving a multi-track data stream;

extracting haptic data from the multi-track data stream, wherein the haptic data comprises haptic information in a series of haptic frames; and

synchronizing the haptic data with an associated media data, wherein the synchronizing includes

playing a media frame of the associated media data at a first time;

determining, for each of the haptic frames, a corresponding first time on a master time code at which a respective one of the haptic frames is to be output, wherein the haptic frames are each a self-contained haptic frame independent of a preceding haptic frame and a succeeding haptic frame; and

outputting one or more haptic effects of a first haptic frame of the haptic frames via an actuator in a playback device based on an offset parameter of the first haptic frame, wherein the offset parameter specifies how far after a corresponding first time of the first haptic frame the one or more haptic effects commence, and wherein the corresponding first time of the first haptic frame is different from a start time of the master time code.

2. The method of claim 1, wherein the multi-track data stream comprises audio and video data and wherein the haptic data is streamed independent of the audio and video data.

3. The method of claim 1, wherein the multi-track data stream is a series of data packets which are transmitted from a transmitter and are received by a receiver.

4. The method of claim 1, wherein the media frame comprises a video frame or audio frame.

5. The method of claim 1, wherein each outputted haptic effect is synchronized with a corresponding media frame of the associated media data.

6. An apparatus, comprising:

a reception module configured to receive a multi-track data stream;

an extraction module configured to extract haptic data from the multi-track data stream, wherein the haptic data includes haptic information in a series of haptic frames; and

a synchronization module configured to synchronize the haptic data with an associated media data, wherein the synchronization module is further configured to;

play a media frame of the associated media data at a first time;

determine, for each of the haptic frames, a corresponding first time on a master time code at which a respective one of the haptic frames is to be output, wherein the haptic frames are each a self-contained haptic frame independent of a preceding haptic frame and a succeeding haptic frame; and

output one or more haptic effects of a first haptic frame of the haptic frames via an actuator in a playback device based on an offset parameter of the first haptic frame, wherein the offset parameter specifies how far after a corresponding first time of the first haptic frame the one or more haptic effects commence, and wherein the corresponding first time of the first haptic frame is different from a start time of the master time code.

7. The apparatus of claim 6, wherein the multi-track data stream comprises audio and video data and wherein the haptic data is streamed independent of the audio and video data.

8. The apparatus of claim 6, wherein the multi-track data stream is a series of data packets which are transmitted from a transmitter and are received by a receiver.

9. The apparatus of claim 6, wherein the media frame comprises a video frame or audio frame.

10. The apparatus of claim 6, wherein each outputted haptic effect is synchronized with a corresponding media frame of the associated media data.

11. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a computer, cause the computer to perform a method, the method comprising:

receiving a multi-track data stream;

extracting haptic data from the multi-track data stream, wherein the haptic data comprises haptic information in a series of haptic frames; and

synchronizing the haptic data with an associated media data, wherein the synchronizing includes

playing a media frame of the associated media data at a first time;

determining, for each of the haptic frames, a corresponding first time on a master time code at which a respective one of the haptic frames is to be output, wherein the haptic frames are each a self-contained haptic frame independent of a preceding haptic frame and a succeeding haptic frame; and

outputting one or more haptic effects of a first haptic frame of the haptic frames via an actuator in a playback device based on an offset parameter of the first haptic frame, wherein the offset parameter specifies how far after a corresponding first time of the first haptic frame the one or more haptic effects commence, and wherein the corresponding first time of the first haptic frame is different from a start time of the master time code.

12. The non-transitory computer-readable medium of claim 11, wherein the multi-track data stream comprises audio and video data and wherein the haptic data is streamed independent of the audio and video data.

13. The non-transitory computer-readable medium of claim 11, wherein the multi-track data stream is a series of data packets which are transmitted from a transmitter and are received by a receiver.

14. The non-transitory computer-readable medium of claim 11, wherein the media frame comprises a video frame or audio frame.