Systems and methods for generating haptic effects associated with transitions in audio signals

• US 10,276,004 B2
• Filed: 06/08/2017
• Issued: 04/30/2019
• Est. Priority Date: 09/06/2013
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Cruz-Hernandez, Juan Manuel, Saboune, Jamal
• Primary Examiner(s): Terrell, Emily C
• Application Number:
  US15/617,172
• Publication Number:
  US 20170270756A1
• Time in Patent Office: 691 Days
• Field of Search: 3404071-4072, 340 76, 704219-233
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G10H 2220/096 using a touch screen touch screen note input, e.g. using a displayed keyboard G10H2220/241; personal digital assistant [PDA] G10H2230/015
• CPC Class Codes: G10H 2220/311 with controlled tactile or haptic feedback effect; output interfaces therefor
• CPC Class Codes: G10H 7/02 in which amplitudes at successive sample points of a tone waveform are stored in one or more memories
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

Systems and methods for generating haptic effects associated with transitions in audio signals are disclosed. One disclosed system for outputting haptic effects includes a processor configured to: receive a signal; determine a haptic effect based in part on the signal; output a haptic signal associated with the haptic effect; an audio output device configured to receive the signal and output an audible effect; and a haptic output device in communication with the processor and coupled to the touch surface, the haptic output device configured to receive the haptic signal and output the haptic effect.

Claims

1. A system for outputting haptic effects comprising:

a processor configured to;

receive a signal;

identify one or more transitions in the signal by;

performing a Fast Fourier Transform on the signal taken at a series of time intervals to generate a transformed signal;

determining a mean value of the transformed signal for each of the time intervals;

normalizing each of the mean values to create a normalized signal;

determining a derivative of the normalized signal to create a derivative signal; and

determining one or more local maximum values within the derivative signal;

apply one or more haptic markers to the one or more transitions;

synch one or more haptic effects to the one or more haptic markers;

transmit a haptic signal to a data store configured to store the haptic effect in a haptic track; and

output the haptic signal associated with the one or more haptic effects to a haptic output device in communication with the processor and configured to receive the haptic signal and output the haptic effect.

2. The system of claim 1, wherein the signal comprises one or more of:

an audio signal, a pressure signal, or an acceleration signal.

3. The system of claim 1, further comprising:

an audio output device configured to receive the signal and output an audible effect.

4. The system of claim 1, wherein the transitions comprise one or more of:

a change in frequency, a change in amplitude, or a repetition of a frequency and amplitude.

5. The system of claim 1, wherein the haptic effect comprises one or more of:

a variation in coefficient of friction, a simulated texture, or a vibration.

6. The system of claim 1, wherein the processor is further configured to filter the normalized signal.

7. The system of claim 1, wherein the one or more local maximum values comprises the one or more transitions.

8. A method for outputting haptic effects comprising:

receiving a signal;

identifying one or more transitions in the signal by;

performing a Fast Fourier Transform on the signal within a series of time intervals to generate a transformed signal;

determining a mean value of the transformed signal for each of the time intervals;

normalizing each of the mean values to create a normalized signal;

determining a derivative of the normalized signal to create a derivative signal; and

determining one or more local maximum values within the derivative signal;

applying one or more haptic markers to the one or more transitions;

synching one or more haptic effects to the one or more haptic markers;

transmitting a haptic signal to a data store configured to store the haptic effect in a haptic track; and

outputting the haptic signal associated with the one or more haptic effects to a haptic output device configured to receive the haptic signal and output the haptic effect.

9. The method of claim 8, wherein the signal comprises one or more of:

an audio signal, a pressure signal, or an acceleration signal.

10. The method of claim 8, further comprising storing the haptic signal in a haptic track.

11. The method of claim 8, wherein the haptic effect comprises one or more of:

a variation in coefficient of friction, a simulated texture, or a vibration.

12. The method of claim 8, wherein the transitions comprise one or more of:

a change in frequency, a change in amplitude, or a repetition of a frequency and amplitude.

13. The method of claim 8, further comprising filtering the normalized signal.

14. The method of claim 8, wherein the one or more local maximum values comprises the one or more transitions.

15. A non-transitory computer readable medium comprising program code, which when executed by a processor, is configured to cause the processor to:

receive a signal;

identify one or more transitions in the signal by;

performing a Fast Fourier Transform on the signal within a series of time intervals to generate a transformed signal;

determining a mean value of the transformed signal for each of the time intervals;

normalizing each of the mean values to create a normalized signal;

determining a derivative of the normalized signal to create a derivative signal; and

determining one or more local maximum values within the derivative signal;

apply one or more haptic markers to the one or more transitions;

synch one or more haptic effects to the one or more haptic markers;

transmit a haptic signal to a data store configured to store the haptic effect in a haptic track; and

output the haptic signal associated with the one or more haptic effects to a haptic output device configured to receive the haptic signal and output the haptic effect.

16. The non-transitory computer readable medium of claim 15, wherein the signal comprises one or more of:

an audio signal, a pressure signal, or an acceleration signal.

17. The non-transitory computer readable medium of claim 15, wherein the haptic effect comprises one or more of:

a variation in coefficient of friction, a simulated texture, or a vibration.

18. The non-transitory computer readable medium of claim 15, wherein the transitions comprise one or more of:

a change in frequency, a change in amplitude, or a repetition of a frequency and amplitude.