Sound to haptic effect conversion system using multiple actuators

• US 10,074,246 B2
• Filed: 06/28/2017
• Issued: 09/11/2018
• Est. Priority Date: 04/04/2012
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Da Costa, Henry, Timone, Herve Thu, Heubel, Robert, Rank, Stephen, Cruz-Hernandez, Juan Manuel, Grant, Danny
• Primary Examiner(s): McLoone, Peter D
• Application Number:
  US15/635,395
• Publication Number:
  US 20170301195A1
• Time in Patent Office: 440 Days
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems

Abstract

A haptic conversion system is provided that analyzes an audio signal, generates a plurality of haptic signals based on the analysis of the audio signal, and plays the generated plurality of haptic signals through a plurality of actuators to produce one or more haptic effects. The haptic conversion system maps the generated plurality of haptic signals to the plurality of actuators based on one or more audio characteristics of the audio signal. Each generated haptic signal includes one or more haptic parameters, and is played at its mapped actuator to generate the one or more haptic effects.

Claims

1. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to execute a method of producing haptic effects, the method comprising:

outputting audio effects by a plurality of actuators, each actuator of the plurality of actuators being located in a different region of a device, wherein at least two of the audio effects are output from different parts of a surface of the device, wherein at least one audio effect is output at a location where a finger of a user interacts with the surface of the device;

generating haptic effects based on the audio effects; and

outputting the haptic effects at the plurality of actuators, wherein at least two of the haptic effects are output from different parts of the surface of the device, wherein the plurality of actuators output the haptic effects and the audio effects at the same time.

2. The non-transitory computer-readable medium of claim 1, wherein the haptic effects are generated based on an encoding scheme.

3. The non-transitory computer-readable medium of claim 2, wherein the encoding scheme comprises a Dolby Digital encoding scheme.

4. The non-transitory computer-readable medium of claim 2, wherein haptic encoding is performed as part of audio encoding by adding low frequency content to audio information.

5. The non-transitory computer-readable medium of claim 1, wherein the device further includes one or more additional actuators located on a back portion of the device.

6. The non-transitory computer-readable medium of claim 5, wherein the one or more additional actuators comprise full body actuators configured to provide one or more haptic effects at different frequencies.

7. The non-transitory computer-readable medium of claim 1, wherein generating the haptic effects comprises:

determining sets of values of audio characteristics of an audio signal, each of the sets of values of audio characteristics comprising at least a frequency, a magnitude, or a duration,

mapping the sets of values of audio characteristics to the actuators, wherein each of the sets of values of audio characteristics is mapped to a different one of the actuators, and

generating haptic signals based on the sets of values of audio characteristics, wherein the haptic signals include a haptic signal generated for each of the sets of values of audio characteristics, and

wherein outputting the haptic effects comprises

sending the haptic signals to the actuators based on the mapping of the sets of values of audio characteristics to the actuators, and

playing the haptic signals at the actuators to generate the haptic effects.

8. The non-transitory computer-readable medium of claim 7, wherein generating the haptic effects further comprises:

selecting one or more pre-defined haptic signals from a universal haptic layer library based on the sets of values of audio characteristics of the audio signal, and

mapping the one or more pre-defined haptic signals to the actuators, wherein each of the one or more pre-defined haptic signals is mapped to a corresponding actuator, and

wherein outputting the haptic effects further comprises;

sending each of the one or more pre-defined haptic signals to the corresponding actuator.

9. The non-transitory computer-readable medium of claim 7, wherein generating the haptic effects further comprises:

generating one or more secondary haptic signals based on a stored definition, wherein the stored definition comprises a conditional statement that comprises a condition and a statement, wherein the condition is the occurrence of a generation of one or more primary haptic signals that are sent to a primary actuator, and the statement is an instruction to generate the one or more secondary haptic signals that are sent to a secondary actuator, and

mapping the one or more secondary haptic signals to the secondary actuator, and

wherein outputting the haptic effects further comprises

sending each of the one or more secondary haptic signals to the secondary actuator based on the stored definition.

10. The non-transitory computer-readable medium of claim 7, wherein generating the haptic effects further comprises:

dividing at least one haptic signal into haptic signal components, and

mapping each of the haptic signal components to corresponding actuators, and

wherein outputting the haptic effects further comprises

sending the haptic signal components to the corresponding actuators.

11. The non-transitory computer-readable medium of claim 7, wherein generating the haptic effects further comprises:

combining at least two of the haptic signals into a combined haptic signal, and

mapping the combined haptic signal to a mapped actuator, and

wherein outputting the haptic effects further comprises

sending the combined haptic signal to the mapped actuator.

12. The non-transitory computer-readable medium of claim 7, wherein haptic parameters of the haptic signals comprise at least one of an amplitude parameter, a duration parameter, or a frequency parameter.

13. The non-transitory computer-readable medium of claim 7, wherein determining sets of values comprises identifying amplitude peaks of the audio signal.

14. The non-transitory computer-readable medium of claim 7, wherein the sets of values of audio characteristics comprise at least one of an amplitude of an audio data frame, a frequency of an audio data frame, and a duration of an audio data frame.

15. The non-transitory computer-readable medium of claim 7, wherein the mapping is at least one of system-defined and user-defined.

16. The non-transitory computer-readable medium of claim 7, wherein characteristics of each actuator comprise an ability to generate a haptic effect with at least one of:

a specified frequency;

a specified amplitude;

or a specified duration.

17. The non-transitory computer-readable medium of claim 7, wherein a user experiences synchronized playback of the haptic effects.

18. The non-transitory computer-readable medium of claim 7,

wherein the audio signal comprises channels;

wherein generating the haptic signals further comprises generating distinct haptic signals for the channels of the audio signal;

wherein sending of the haptic signals to the respective one of the actuators sends a distinct haptic signal to a respective distinct actuator.

19. A computer-implemented method for producing haptic effects, the computer-implemented method comprising:

outputting audio effects by a plurality of actuators, each actuator of the plurality of actuators included in a different region of a device, wherein at least two of the audio effects are output from different parts of a surface of the device, wherein at least one audio effect is output at a location where a finger of a user interacts with the surface of the device;

generating haptic effects based on the audio effects; and

outputting the haptic effects at the plurality of actuators, wherein at least two of the haptic effects are output from different parts of the surface of the device, wherein the plurality of actuators output the haptic effects and the audio effects at the same time.

20. A haptic conversion system comprising:

a memory configured to store a haptic conversion module;

a processor configured to execute the haptic conversion module stored on the memory; and

a plurality of actuators, each actuator included in a different region of a device;

wherein the haptic conversion module is configured to

output audio effects by the plurality of actuators, wherein at least two of the audio effects are output from different parts of a surface of the device, and wherein at least one audio effect is output at a location where a finger of a user interacts with the surface of the device;

generate haptic effects based on the audio effects; and

output the haptic effects at the plurality of actuators, wherein at least two of the haptic effects are output from different parts of the surface of the device, wherein the plurality of actuators output the haptic effects and the audio effects at the same time.