Synchronization of haptic effect data in a media transport stream

• US 8,700,791 B2
• Filed: 10/18/2006
• Issued: 04/15/2014
• Est. Priority Date: 10/19/2005
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Lacroix, Robert A., Rabemiarisoa, Andrianaivo, da Costa, Henrique D., Timone, Herve Thu, Rank, Stephen D., Ullrich, Christopher J.
• Primary Examiner(s): Winder, Patrice
• Assistant Examiner(s): TRAN, NAM T
• Application Number:
  US11/583,483
• Publication Number:
  US 20080223627A1
• Time in Patent Office: 2,736 Days
• Field of Search: None
• US Class Current: 709/231
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Haptic information in a series of frames of a media transport stream is identified and time stamps corresponding thereto are determined in accordance with a master time code signal embedded in the media transport stream. Each media transport stream frame containing haptic information is subsequently assigned a time stamp so that it will be used to activate an actuator at a proper time responsive to the time stamp to generate a haptic effect in accordance with the haptic information.

Claims

1. A method for synchronizing haptic effects, comprising:

identifying haptic information in one or more frames of a media transport stream, wherein the media transport stream comprises the haptic information and at least one of;

audio information or video information;

determining one or more time stamps according to a master time code embedded within the media transport stream, wherein the master time code comprises the one or more time stamps, wherein the master time code is used to synchronize the haptic information and at least one of the audio information or the video information;

assigning at least one of the one or more time stamps to each of the one or more frames, wherein the assigned time stamp indicates a time for activating at least one actuator in response to the haptic information; and

defining effect information for each haptic effect in response to the haptic information in the one or more frames.

2. The method of claim 1, further comprising:

activating an actuator in response to the effect information; and

maintaining the haptic effect according to an associated time stamp.

3. The method of claim 1, wherein the identifying haptic information in one or more frames of a media transport stream further includes:

identifying a series of video frames containing imaging information;

identifying a series of audio frames containing sound information; and

identifying a series of haptic frames containing force feedback information.

4. The method of claim 3, further comprising synchronizing the force feedback information in response to the imaging information and sound information.

5. The method of claim 4, further comprising generating a haptic effect including attack level and fade level.

6. The method of claim 1, wherein identifying haptic information in one or more frames further includes identifying the frames in substantially same length of time.

7. The method of claim 1, wherein identifying haptic information in one or more frames of a media transport stream includes analyzing the media transport stream according to the MPEG4 protocol.

8. The method of claim 1 further comprising:

encoding the one or more frames with the haptic information in the media transport stream; and

transporting the media transport stream over a network.

9. The method of claim 8, further comprising:

receiving the media transport stream across the network;

extracting a media component and the haptic component;

playing the media component; and

synchronizing the haptic component with the media component by playing each frame at its assigned time stamp for play.

10. A method for transmitting and playing a haptic signal embedded in a media transport stream, the method comprising:

parsing the haptic signal into successive frames having a substantially fixed length of time, wherein each frame is assigned an initial time at which it should be played according to a master time code for playing the media transport stream, wherein the master time code is embedded within the media transport stream and comprises one or more time stamps, each time stamp indicating an initial time at which a particular frame should be played;

defining effect information for each frame for each haptic effect to be played within a time span of the frame;

encoding the frames as a haptic component in the media transport stream;

transporting the media transport stream;

receiving the media transport stream at a playback device;

extracting a media component and the haptic component;

playing the media component; and

synchronizing the haptic component with the media component by playing each frame at its assigned initial time according to the master time code.

11. The method of claim 10, wherein the effect information includes a start-time, magnitude, frequency, envelope, duration and/or style.

12. The method of claim 10, wherein the media component is an audio component, video component, or both.

13. The method of claim 10, wherein the media transport stream is stored prior to being received at the playback device.

14. The method of claim 10, wherein if no effect information exists for the frame, the frame is not encoded into the media transport stream.

15. The method of claim 10, wherein the substantially fixed length of time is the smallest increment of time according to the master time code.

16. The method of claim 10, wherein the haptic effect information uses a non-parametric encoding.

17. The method of claim 10, wherein the haptic signal is compressed.

18. The method of claim 10, wherein the haptic signal is encrypted.

19. The method of claim 10, wherein the haptic signal is compressed and encrypted.

20. An apparatus for synchronizing haptic effects, comprising:

means for identifying haptic information in one or more frames of a media transport stream, wherein the media transport stream comprises the haptic information and at least one of;

audio information or video information;

means for determining one or more time stamps according to a master time code embedded within the media transport stream, wherein the master time code comprises the one or more time stamps, wherein the master time code is used to synchronize the haptic information and at least one of the audio information or the video information;

means for assigning at least one of the one or more time stamps to each of the one or more frames, wherein the assigned time stamp indicates a time for activating at least one actuator in response to the haptic information; and

means for defining effect information for each haptic effect in response to the haptic information in the one or more frames.

21. The apparatus of claim 20, further comprising:

means for activating an actuator in response to the effect information; and

means for maintaining the haptic effect according to an associated time stamp.

22. The apparatus of claim 20, wherein the means for identifying haptic information in one or more frames of a media transport stream further includes:

means for identifying a series of video frames containing imaging information;

means for identifying a series of audio frames containing sound information; and

means for identifying a series of haptic frames containing force feedback information.

23. The apparatus of claim 20, further comprising means for synchronizing the force feedback information in response to the imaging information and sound information.

24. The apparatus of claim 23, further comprising means for generating a haptic effect including attack level and fade level.

25. The apparatus of claim 20, wherein means for identifying haptic information in one or more frames further includes means for identifying the frames in substantially same length of time.

26. The apparatus of claim 20, wherein means for identifying haptic information in one or more frames of a media transport stream includes means for analyzing the media transport stream according to the MPEG4 protocol.

27. The apparatus of claim 20, further comprising:

means for encoding the one or more frames with the haptic information in the media transport stream; and

means for transporting the media transport stream over a network.

28. The apparatus of claim 20, further comprising:

means for receiving the media transport stream across the network;

means for extracting a media component and the haptic component;

means for playing the media component; and

means for synchronizing the haptic component with the media component by playing each frame at its assigned time stamp for play.

29. An apparatus for embedding a haptic signal in a media transport stream, comprising:

means for parsing the haptic signal into one or more frames having a same fixed length of time;

means for assigning the frames initial times for play associated with the haptic signal and at least one of;

an audio signal or a video signal, the initial times for play associated with a master time code embedded within the media transport stream, wherein the master time code comprises one or more time stamps, each time stamp indicating an initial time for play for a particular frame, wherein the master time code is used to synchronize the haptic signal and at least one of the audio signal or the video signal; and

means for defining effect information for each haptic effect to be played within a time span of each frame.

30. An apparatus for transmitting and playing a haptic signal embedded in a media transport stream, comprising:

means for parsing the haptic signal into successive frames having a fixed length of time, wherein each frame is assigned an initial time at which it should be played according to a master time code embedded within the media transport stream, wherein the master time code comprises one or more time stamps, each time stamp indicating an initial time at which a particular frame should be played, wherein the media transport stream comprises the haptic signal and at least one of;

an audio signal or a video signal, and wherein the master time code is used to synchronize the haptic signal and at least one of the audio signal or the video signal;

means for defining effect information for each frame for each haptic effect to be played within a time span of the frame;

means for encoding the frames as a haptic component in the media transport stream;

means for transporting the transport stream;

means for receiving the media transport stream at a playback device;

means for extracting a media component and the haptic component;

means for playing the media component; and

means for synchronizing the haptic component with the media component by playing each frame at its assigned initial time according to the master time code.