Method and system for providing haptic effects based on information complementary to multimedia content

• US 9,652,945 B2
• Filed: 12/31/2013
• Issued: 05/16/2017
• Est. Priority Date: 09/06/2013
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Levesque, Vincent, Modarres, Ali, Cruz-Hernandez, Juan Manuel, Saboune, Jamal
• Primary Examiner(s): Lau, Hoi
• Application Number:
  US14/145,650
• Publication Number:
  US 20150070150A1
• Time in Patent Office: 1,232 Days
• Field of Search: 340 412, 3404071, 3404072, 340 76, 345156, 345173, 707769, 715702, 386239, 386E09011
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

The present disclosure is generally directed to systems and methods for providing haptic effects based on information complementary to multimedia content. For example, one disclosed method includes the steps of receiving multimedia data comprising multimedia content and complementary data, wherein the complementary data describes the multimedia content, determining a haptic effect based at least in part on the complementary data, and outputting the haptic effect while playing the multimedia content.

Claims

1. A method comprising:

receiving, by a processor, an audio track or a video track associated with multimedia content;

receiving, by the processor, complementary data that describes the multimedia content;

determining, by the processor, a single haptic effect by both (i) analyzing the audio track or the video track and (ii) using the complementary data; and

outputting, by the processor, the haptic effect via a haptic output device while playing the multimedia content.

2. The method of claim 1, wherein the complementary data comprises tag data, the tag data comprising at least one of a title, artist, author, genre, rating, review, a list of songs, or a song type.

3. The method of claim 1, wherein determining the single haptic effect further comprises:

determining a preliminary haptic effect by analyzing the audio track or the video track;

determining a haptic theme using the complementary data; and

determining the single haptic effect by modifying a characteristic of the preliminary haptic effect based on the haptic theme.

10. The method of claim 1, wherein the single haptic effect is determined at least in part by:

analyzing the audio track to determine one or more words associated with the audio track event; and

determining the single haptic effect based at least in part on the one or more words associated with the audio track.

11. The method of claim 10, wherein the one or more words comprises a particular phrase.

12. The method of claim 10, wherein the single haptic effect comprises no haptic effect in response to the one or more words being surrounded by quotation marks.

13. The method of claim 1, wherein the single haptic effect is determined at least in part:

analyzing the video track to determine one or more visual events associated with the video track; and

determining the single haptic effect based at least in part on the one or more visual events associated with the video track.

14. The method of claim 1, wherein the complementary data comprises closed caption data or subtitle data.

15. The method of claim 1, further comprising:

determining context information based on the complementary data; and

determining the single haptic effect by analyzing the audio track or the video track in view of the context information.

16. The method of claim 1, wherein the single haptic effect is a first haptic effect, and further comprising:

determining a second haptic effect; and

determining a priority of the first haptic effect with respect to the second haptic effect.

17. The method of claim 16, further comprising:

inserting the first haptic effect or the second haptic effect into a haptic track based on the priority.

18. The method of claim 1, further comprising:

determining a weight associated with a word in the complementary data, the weight indicating a priority of the word.

19. The method of claim 1, further comprising:

determining a haptic theme using the complementary data; and

determining that the single haptic effect is not to be modified according to a modification associated with the haptic theme.

4. A non-transitory computer readable medium comprising software program code executable by a processor, the software program code configured to cause the processor to:

receive an audio track or a video track associated with multimedia content;

receive complementary data that describes the multimedia content;

determine a single haptic effect by both (i) analyzing the audio track or the video track and (ii) using the complementary data; and

output the haptic effect, via a haptic output device, while playing the multimedia content.

5. The non-transitory computer readable medium of claim 4, wherein the complementary data comprises tag data, the tag data comprising at least one of a title, artist, author, genre, rating, review, a list of songs, or a song type.

6. The non-transitory computer readable medium of claim 4, wherein determining the single haptic effect further comprises:

determining a preliminary haptic effect by analyzing the audio track or the video track;

determining a haptic theme using the complementary data; and

determining the single haptic effect by modifying a characteristic of the preliminary haptic effect based on the haptic theme.

7. A system comprising:

a processor; and

a memory in communication with the processor, the memory comprising computer program code executable by the processor for causing the processor to;

receive an audio track or a video track associated with multimedia content;

receive complementary data that describes the multimedia content;

determine a single haptic effect by both (i) analyzing the audio track or the video track and (ii) using the complementary data; and

output the haptic effect, via a haptic output device, while playing the multimedia content.

8. The system of claim 7, wherein the complementary data comprises tag data, the tag data comprising at least one of a title, artist, author, genre, rating, review, a list of songs, or a song type.

9. The system of claim 7, wherein determining the single haptic effect further comprises:

determining a preliminary haptic effect by analyzing the audio track or the video track;

determining a haptic theme using the complementary data; and

determining the single haptic effect by modifying a characteristic of the preliminary haptic effect based on the haptic theme.