Sound to haptic effect conversion system using multiple actuators

• US 10,467,870 B2
• Filed: 09/07/2018
• Issued: 11/05/2019
• Est. Priority Date: 04/04/2012
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Da Costa, Henry, Timone, Herve Thu, Heubel, Robert, Rank, Stephen, Cruz-Hernandez, Juan Manuel, Grant, Danny
• Primary Examiner(s): McClone, Peter D
• Application Number:
  US16/125,041
• Publication Number:
  US 20190051125A1
• Time in Patent Office: 424 Days
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems

Abstract

A haptic conversion system is provided that analyzes an audio signal, generates a plurality of haptic signals based on the analysis of the audio signal, and plays the generated plurality of haptic signals through a plurality of actuators to produce one or more haptic effects. The haptic conversion system maps the generated plurality of haptic signals to the plurality of actuators based on one or more audio characteristics of the audio signal. Each generated haptic signal includes one or more haptic parameters, and is played at its mapped actuator to generate the one or more haptic effects.

Claims

1. A method of producing haptic effects, the method comprising:

outputting a plurality of audio effects by a plurality of actuators, each actuator of the plurality of actuators being located in a different region of a device, wherein a first audio effect of the plurality of audio effects is output from a surface of the device by a first actuator of the plurality of actuators and a second audio effect is output from the surface of the device by a second actuator of the plurality of actuators;

generating a first haptic effect based on the first audio effect by

determining a set of values of audio characteristics of an audio signal corresponding to the first audio effect, the set of values of audio characteristics being generated from an audio data frame and comprising at least one of an amplitude of the audio data frame, a frequency of the audio data frame, and a duration of the audio data frame,

mapping the set of values of audio characteristics to the first actuator,

generating a haptic signal based on the set of values of audio characteristics, wherein the first haptic signal corresponds to the first haptic effect; and

outputting the first haptic effect by the first actuator by

sending the haptic signal to the first actuator based on the mapping of the set of values of audio characteristics to the first actuator, and

playing the haptic signal at the first actuator to generate the first haptic effect.

2. The method of claim 1, wherein the first haptic effect and the first audio effect are output by the first actuator at the same time.

3. The method of claim 2, wherein generating the first haptic effect includes encoding the haptic signal corresponding to the first haptic effect by adding the haptic signal to the audio signal corresponding to the first audio effect.

4. The method of claim 1, further comprising:

generating a second haptic effect based on the second audio effect; and

outputting the second haptic effect by the second actuator.

5. The method of claim 1, wherein the device further includes one or more additional actuators located on a back portion of the device.

6. The method of claim 5, wherein the one or more additional actuators comprise full body actuators configured to provide one or more haptic effects at different frequencies.

7. The method of claim 1,

wherein generating the first haptic effect further comprises

selecting one or more pre-defined haptic signals from a universal haptic layer library based on the set of values of audio characteristics of the first audio signal, and

mapping the one or more pre-defined haptic signals to the first actuator, and

wherein outputting the first haptic effect further comprises

sending the one or more pre-defined haptic signals to the first actuator.

8. The method of claim 1, further comprising:

generating one or more secondary haptic signals based on a stored definition, the stored definition comprising a conditional statement that comprises a condition and a statement, wherein the condition is the occurrence of a generation of the haptic signal and the statement is an instruction to generate the one or more secondary haptic signals,

mapping the one or more secondary haptic signals to a secondary actuator, and

outputting the one or more secondary haptic signals to the secondary actuator.

9. The method of claim 1, wherein generating the first haptic effect further comprises

dividing the haptic signal into haptic signal components, and

mapping one of the haptic signal components to the first actuator, and

wherein outputting the first haptic effect further comprises sending the one of the haptic signal components to the first actuator.

10. The method of claim 1, wherein the first haptic effect and the first audio effect are output at a location of a finger of a user interacting with the surface of the device.

11. The method of claim 1, wherein haptic parameters of the haptic signal comprise at least one of an amplitude parameter, a duration parameter, and a frequency parameter.

12. The method of claim 1, wherein determining the set of values comprises identifying amplitude peaks of the audio signal.

13. The method of claim 1, wherein the mapping is system-defined.

14. The method of claim 1, wherein characteristics of the first actuator comprise an ability to generate the haptic effect with at least one of a specified frequency, a specified amplitude, or a specified duration.

15. A haptic conversion system comprising:

a memory configured to store a haptic conversion module;

a processor configured to execute the haptic conversion module stored on the memory; and

a plurality of actuators, each actuator disposed at a different region of a device,

wherein the haptic conversion module is configured to

output a plurality of audio effects by the plurality of actuators, wherein a first audio effect of the plurality of audio effects is output from a surface of the device by a first actuator of the plurality of actuators and a second audio effect is output from the surface of the device by a second actuator of the plurality of actuators,

generate a first haptic effect based on the first audio effect by

determining a set of values of audio characteristics of an audio signal corresponding to the first audio effect, the set of values of audio characteristics being generated from an audio data frame and comprising at least one of an amplitude of the audio data frame, a frequency of the audio data frame, and a duration of the audio data frame,

mapping the set of values of audio characteristics to the first actuator,

generating a haptic signal based on the set of values of audio characteristics, wherein the haptic signal corresponds to the first haptic effect, and

cause output of the first haptic effect by the first actuator by

sending the haptic signal to the first actuator based on the mapping of the set of values of audio characteristics to the first actuator, and

playing the haptic signal at the first actuator to generate the first haptic effect.

16. The system of claim 15, wherein the processor is further configured to cause the first haptic effect and the first audio effect to be output by the first actuator at the same time.

17. The system of claim 15, wherein the processor is further configured to

generate a second haptic effect based on the second audio effect, and

cause output of the second haptic effect by the second actuator.

18. The system of claim 15, wherein the first haptic effect and the first audio effect are output at a location of a finger of a user interacting with the surface of the device.

19. A method of producing haptic effects, the method comprising:

outputting a plurality of audio effects by a plurality of actuators, each actuator of the plurality of actuators being located in a different region of a device, wherein a first audio effect of the plurality of audio effects is output from a surface of the device by a first actuator of the plurality of actuators and a second audio effect is output from the surface of the device by a second actuator of the plurality of actuators;

generating a first haptic effect based on the first audio effect;

generating one or more secondary haptic signals based on a stored definition, the stored definition comprising a conditional statement that comprises a condition and a statement, wherein the condition is the occurrence of a generation of the haptic signal and the statement is an instruction to generate the one or more secondary haptic signals;

mapping the one or more secondary haptic signals to the second actuator;

outputting the first haptic effect by the first actuator; and

outputting the one or more secondary haptic signals to the second actuator.