Systems and Methods for Generating Haptic Effects Associated With Transitions in Audio Signals

• US 20150070146A1
• Filed: 11/12/2013
• Published: 03/12/2015
• Est. Priority Date: 09/06/2013
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Cruz-Hernandez, Juan Manuel, Saboune, Jamal
• Granted Patent: US 9,711,014 B2
• Time in Patent Office: Days
• US Class Current: 340/407.1
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G10H 2220/096 using a touch screen touch screen note input, e.g. using a displayed keyboard G10H2220/241; personal digital assistant [PDA] G10H2230/015
• CPC Class Codes: G10H 2220/311 with controlled tactile or haptic feedback effect; output interfaces therefor
• CPC Class Codes: G10H 7/02 in which amplitudes at successive sample points of a tone waveform are stored in one or more memories
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

Systems and methods for generating haptic effects associated with transitions in audio signals are disclosed. One disclosed system for outputting haptic effects includes a processor configured to: receive a signal; determine a haptic effect based in part on the signal; output a haptic signal associated with the haptic effect; an audio output device configured to receive the signal and output an audible effect; and a haptic output device in communication with the processor and coupled to the touch surface, the haptic output device configured to receive the haptic signal and output the haptic effect.

Claims

1. A system for outputting haptic effects comprising:

a processor configured to;

receive a signal;

determine a haptic effect based in part on the signal by;

identifying one or more transitions in the signal; and

synching one or more haptic effects to the one or more transitions; and

output a haptic signal associated with the haptic effect.

2. The system of claim 1, wherein the signal comprises one of an:

audio signal, pressure signal, or acceleration signal.

3. The system of claim 1, further comprising:

an audio output device configured to receive the signal and output an audible effect; and

a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.

4. The system of claim 1, further comprising a data store configured to receive the haptic signal and store the haptic effect in a haptic track.

5. The system of claim 1, wherein the transitions comprise one or more of:

a change in frequency, a change in amplitude, or a repetition of a frequency and amplitude.

6. The system of claim 1, wherein the haptic effect comprises one or more of:

a variation in coefficient of friction, a simulated texture, or a vibration.

7. The system of claim 1, wherein identifying one or more transitions comprises:

performing a Fast Fourier Transform on the signal taken at a series of time intervals;

determining a mean value of the transformed signal for each of the time intervals;

normalizing each of the mean values to create a normalized signal;

determining a derivative of the normalized signal to create a derivative signal; and

determining one or more local maximum values within the derivative signal.

8. The system of claim 7, further comprising filtering the normalized signal.

9. The system of claim 7, wherein the one or more local maximum values comprises the one or more transitions.

10. The system of claim 1, wherein identifying one or more transitions comprises:

determining the Power Spectral Density of a group of frequency bands within a first time window and a second time window, wherein the first time window and second time window are consecutive time windows in the signal;

determining a Total Power Spectral Density by summing the Power Spectral Density values for all the frequency bands per time window;

determining a contribution of each frequency band to the Total Power Spectral Density by comparing the value of the Power Spectral Density of each frequency band to the Total Power Spectral Density per time window;

determining a first rate of change of the Power Spectral Density of the frequency bands from the first time window to the second time window;

determining a first distance equal to the sum of the first rate of change weighted by the contribution of each frequency band;

determining a second rate of change of the Power Spectral Density of the frequency bands from the first time window to a third time window, wherein the second time window and the third time window are consecutive time windows in the signal;

determining a second distance equal to the sum of the second rate of change at each frequency band weighted by the contribution of each frequency band;

determining a total distance between the first time window and the second time window by multiplying the first distance by the second distance; and

determining a local maximum.

11. The system of claim 10, wherein the local maximum comprises a local maximum of the first distance, second distance, or total distance.

12. The system of claim 10, wherein the local maximum comprises one or more transitions in the signal.

13. A method for outputting haptic effects comprising:

receiving a signal;

determining a haptic effect based in part on the signal by;

identifying one or more transitions in the signal; and

synching one or more haptic effects to the one or more transitions; and

outputting a haptic signal associated with the haptic effect.

15. The method of claim 13, further comprising:

outputting an audible effect; and

outputting the haptic effect.

16. The method of claim 13, further comprising storing the haptic signal in a haptic track.

17. The method of claim 13, wherein the haptic effect comprises one or more of:

a variation in coefficient of friction, a simulated texture, or a vibration.

18. The method of claim 15, wherein the transitions comprise one or more of:

a change in frequency, a change in amplitude, or a repetition of a frequency and amplitude.

19. The method of claim 15, wherein identifying one or more transitions comprises:

performing a Fast Fourier Transform on the signal within a series of time intervals;

determining a mean value of the transformed signal for each of the time intervals;

normalizing each of the mean values to create a normalized signal;

determining a derivative of the normalized signal to create a derivative signal; and

determining one or more local maximum values within the derivative signal.

20. The method of claim 19, further comprising filtering the normalized signal.

21. The method of claim 19, wherein the one or more local maximum values comprises the one or more transitions.

22. The method of claim 15, wherein identifying one or more transitions comprises:

determining the Power Spectral Density of a group of frequency bands within a first time window and a second time window, wherein the first time window and second time window are consecutive time windows in the signal;

determining a Total Power Spectral Density by summing the Power Spectral Density values for all the frequency bands per time window;

determining a contribution of each frequency band to the Total Power Spectral Density by comparing the value of the Power Spectral Density of each frequency band to the Total Power Spectral Density per time window;

determining a first rate of change of the Power Spectral Density of the frequency bands from the first time window to the second time window;

determining a first distance equal to the sum of the first rate of change weighted by the contribution of each frequency band;

determining a second rate of change of the Power Spectral Density of the frequency bands from the first time window to a third time window, wherein the second time window and the third time window are consecutive time windows in the signal;

determining a second distance equal to the sum of the second rate of change at each frequency band weighted by the contribution of each frequency band;

determining a total distance between the first time window and the second time window by multiplying the first distance by the second distance; and

determining a local maximum.

23. The method of claim 22, wherein the local maximum comprises a local maximum of the first distance, second distance, or total distance.

24. The method of claim 22, wherein the local maximum comprises one or more transitions in the signal.

14. The method of claim 14, wherein the signal comprises one of an:

audio signal, pressure signal, or acceleration signal.

25. A non-transitory computer readable medium comprising program code, which when executed by a processor, is configured to cause the processor to:

receive a signal;

determine a haptic effect based in part on the signal by;

identifying one or more transitions in the signal; and

synching one or more haptic effects to the one or more transitions; and

output a haptic signal associated with the haptic effect.

26. The non-transitory computer readable medium of claim 25, wherein the signal comprises one of an:

audio signal, pressure signal, or acceleration signal.

27. The non-transitory computer readable medium of claim 25, further comprising program code, which when executed by a processor is configured to cause the processor to:

output an audible effect; and

output the haptic effect.

28. The non-transitory computer readable medium of claim 25, further comprising program code, which when executed by a processor is configured to cause the processor to store the haptic signal in a haptic track.

29. The non-transitory computer readable medium of claim 25, wherein the haptic effect comprises one or more of:

a variation in coefficient of friction, a simulated texture, or a vibration.

30. The non-transitory computer readable medium of claim 27, wherein the transitions comprise one or more of:

a change in frequency, a change in amplitude, or a repetition of a frequency and amplitude.

31. The non-transitory computer readable medium of claim 27, wherein identifying one or more transitions comprises:

performing a Fast Fourier Transform on the signal within a series of time intervals;

determining a mean value of the transformed signal for each of the time intervals;

normalizing each of the mean values to create a normalized signal;

determining a derivative of the normalized signal to create a derivative signal; and

determining one or more local maximum values within the derivative signal.

32. The non-transitory computer readable medium of claim 31, wherein the one or more local maximum values comprises the one or more transitions.

33. The non-transitory computer readable medium of claim 31, further comprising filtering the normalized signal.

34. The non-transitory computer readable medium of claim 27, wherein identifying one or more transitions comprises:

determining the Power Spectral Density of a group of frequency bands within a first time window and a second time window, wherein the first time window and second time window are consecutive time windows in the signal;

determining a Total Power Spectral Density by summing the Power Spectral Density values for all the frequency bands per time window;

determining a contribution of each frequency band to the Total Power Spectral Density by comparing the value of the Power Spectral Density of each frequency band to the Total Power Spectral Density per time window;

determining a first rate of change of the Power Spectral Density of the frequency bands from the first time window to the second time window;

determining a first distance equal to the sum of the first rate of change weighted by the contribution of each frequency band;

determining a second rate of change of the Power Spectral Density of the frequency bands from the first time window to a third time window, wherein the second time window and the third time window are consecutive time windows in the signal;

determining a second distance equal to the sum of the second rate of change at each frequency band weighted by the contribution of each frequency band;

determining a total distance between the first time window and the second time window by multiplying the first distance by the second distance; and

determining a local maximum.

35. The non-transitory computer readable medium of claim 34, wherein the local maximum comprises one or more transitions in the signal.

36. The non-transitory computer readable medium of claim 34, wherein the local maximum comprises a local maximum of the first distance, second distance, or total distance.