Sound to haptic effect conversion system using mapping

• US 10,339,772 B2
• Filed: 10/20/2017
• Issued: 07/02/2019
• Est. Priority Date: 08/31/2012
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Cruz-Hernandez, Juan Manuel, Ullrich, Christopher J.
• Primary Examiner(s): Foxx, Chico A
• Application Number:
  US15/789,250
• Publication Number:
  US 20180040211A1
• Time in Patent Office: 620 Days
• Field of Search: None
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems

Abstract

A system generates a haptic signal. The system receives an audio signal, pre-processes the audio signal by modifying the audio signal to create a modified audio signal, and maps the modified audio signal to a haptic signal. The system then sends the haptic signal to an actuator to generate one or more haptic effects.

Claims

1. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the operations of:

receiving an audio signal;

pre-processing the audio signal to establish a set of audio regions according to actuator characteristics such that each audio region of the set of audio regions has corresponding actuator characteristics;

separating the audio signal into a plurality of sub-signal sets such that each sub-signal set is associated with a corresponding one of the set of audio regions and includes corresponding sub-signals;

mapping a sub-signal of one of the sub-signal sets of the plurality of sub-signal sets to a haptic signal; and

sending the haptic signal to an actuator having the actuator characteristics corresponding to the audio region associated with the one sub-signal set.

2. The non-transitory computer-readable medium of claim 1, wherein mapping the sub-signal to the haptic signal further comprises mapping each sub-signal of the one sub-signal set to respective haptic signals, and

wherein sending the haptic signal to the actuator further comprises combining the respective haptic signals into a combined haptic signal and sending the combined haptic signal to the actuator.

3. The non-transitory computer-readable medium of claim 1, further comprising dividing each audio region of the set of audio regions into a plurality of audio frequency regions, wherein each audio frequency region of the plurality of audio frequency regions in each audio region is associated with a corresponding haptic frequency region.

4. The non-transitory computer-readable medium of claim 3, further comprising separating sub-signals of each of the plurality of sub-signal sets into corresponding ones of the plurality of audio frequency regions in the audio region corresponding to the sub-signal set.

5. The non-transitory computer-readable medium of claim 3, wherein mapping the sub-signal to the haptic signal further includes mapping respective sub-signals of each of the plurality of sub-signal sets to corresponding haptic signals in the haptic frequency region corresponding to the audio frequency region of the respective sub-signal.

6. The non-transitory computer-readable medium of claim 1, wherein establishing the plurality of audio regions is based on actuator characteristics including at least one of a dynamic range, a frequency response, an amplitude, and a physical location of the actuator.

7. The non-transitory computer-readable medium of claim 3, wherein

frequency ranges of the plurality of audio frequency regions match frequency ranges of the haptic frequency regions corresponding to each audio frequency region.

8. The non-transitory computer-readable medium of claim 3, wherein

frequency ranges of the plurality of audio frequency regions differ from frequency ranges of the haptic frequency regions corresponding to each audio frequency region.

9. The non-transitory computer-readable medium of claim 7, wherein the frequency ranges of the haptic frequency regions differ from the frequency ranges of the plurality of audio frequency regions according to the actuator characteristics of the actuators associated with the plurality of audio regions.

10. A method for generating haptics, comprising:

receiving an audio signal;

pre-processing the audio signal to establish a set of audio regions according to actuator characteristics such that each audio region of the set of audio regions has corresponding actuator characteristics;

separating the audio signal into a plurality of sub-signal sets such that each sub-signal set is associated with a corresponding one of the set of audio regions and includes corresponding sub-signals;

mapping a sub-signal of one of the sub-signal sets of the plurality of sub-signal sets to a haptic signal; and

sending the haptic signal to an actuator having the actuator characteristics corresponding to the audio region associated with the one sub-signal set.

11. The method of claim 10, wherein mapping the sub-signal to the haptic signal further comprises mapping each sub-signal of the one sub-signal set to respective haptic signals, and

wherein sending the haptic signal to the actuator further comprises combining the respective haptic signals into a combined haptic signal and sending the combined haptic signal to the actuator.

12. The method of claim 10, further comprising dividing each audio region of the set of audio regions into a plurality of audio frequency regions, wherein each audio frequency region of the plurality of audio frequency regions in each audio region is associated with a corresponding haptic frequency region.

13. The method of claim 12, further comprising separating the sub-signals of each of the plurality of sub-signal sets into corresponding ones of the plurality of audio frequency regions in the audio region corresponding to the sub-signal set.

14. The method of claim 12, wherein mapping the sub-signal to the haptic signal further includes mapping respective sub-signals of each of the plurality of sub-signal sets to haptic signals in the haptic frequency region corresponding to the audio frequency region of the respective sub-signal.

15. The method of claim 10, wherein establishing the plurality of audio regions is based on actuator characteristics including at least one of a dynamic range, a frequency response, an amplitude, and a physical location of the actuator.

16. The method of claim 12, wherein

frequency ranges of the plurality of audio frequency regions match frequency ranges of the haptic frequency regions corresponding to each audio frequency region.

17. The method of claim 12, wherein

frequency ranges of the plurality of audio frequency regions differ from frequency ranges of the haptic frequency regions corresponding to each audio frequency region.

18. The method of claim 16, wherein the frequency ranges of the haptic frequency regions differ from the frequency ranges of the plurality of audio frequency regions according to the actuator characteristics of the actuators associated with the plurality of audio regions.

19. A haptic signal generation system, comprising:

a memory configured to store a haptic conversion module;

a processor configured to execute instructions of the haptic conversion module; and

an actuator configured to output at least one haptic effect;

wherein the haptic conversion module includes instructions configured for;

receiving an audio signal,

pre-processing the audio signal to establish a set of audio regions according to actuator characteristics such that each audio region of the set of audio regions has corresponding actuator characteristics,

separating the audio signal into a plurality of sub-signal sets such that each sub-signal set is associated with a corresponding one of the set of audio regions and includes corresponding sub-signals,

mapping a sub-signal of one of the sub-signal sets of the plurality of sub-signal sets to a haptic signal, and

sending the haptic signal to an actuator having the actuator characteristics corresponding to the audio region associated with the one sub-signal set.

20. The haptic signal generation system of claim 19, wherein mapping the sub-signal to the haptic signal further comprises mapping each sub-signal of the one sub-signal set to respective haptic signals, and

wherein sending the haptic signal to the actuator further comprises combining the respective haptic signals into a combined haptic signal and sending the combined haptic signal to the actuator.

21. The haptic signal generation system of claim 20, wherein the haptic conversion module instructions are further configured for dividing each audio region of the set of audio regions into a plurality of audio frequency regions, wherein each audio frequency region of the plurality of audio frequency regions in each audio region is associated with a corresponding haptic frequency region.

22. The haptic signal generation system of claim 21, wherein the haptic conversion module instructions are further configured for separating sub-signals of each of the plurality of sub-signal sets into corresponding ones of the plurality of audio frequency regions in the audio region corresponding to the sub-signal set.

23. The haptic signal generation system of claim 21, wherein mapping the sub-signal to the haptic signal further includes mapping respective sub-signals of each of the plurality of sub-signal sets to haptic signals in the haptic frequency region corresponding to the audio frequency region of the respective sub-signal.

24. The haptic signal generation system of claim 19, wherein establishing the plurality of audio regions is based on actuator characteristics including at least one of a dynamic range, a frequency response, an amplitude, and a physical location of the actuator.

25. The haptic signal generation system of claim 21, wherein

frequency ranges of the plurality of audio frequency regions match frequency ranges the haptic frequency regions corresponding to each audio frequency region.

26. The haptic signal generation system of claim 21, wherein

frequency ranges of the plurality of audio frequency regions differ from frequency ranges the haptic frequency regions corresponding to each audio frequency region.

27. The haptic signal generation system of claim 23, wherein the frequency ranges of the haptic frequency regions differ from the frequency ranges of the plurality of audio frequency regions according to the actuator characteristics of the actuators associated with the plurality of audio regions.