Sound to haptic effect conversion system using waveform

• US 8,717,152 B2
• Filed: 02/03/2012
• Issued: 05/06/2014
• Est. Priority Date: 02/11/2011
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Ullrich, Christopher J., Cruz-Hernandez, Juan Manuel, Bhatia, Satvir Singh, Gandhi, Kanav, Timone, Herve Thu, Lau, Jason Hoi Fun
• Primary Examiner(s): Tweel, Jr., John A
• Application Number:
  US13/366,010
• Publication Number:
  US 20120206247A1
• Time in Patent Office: 823 Days
• Field of Search: 340/407.1, 345/156, 463/30, 715/702
• US Class Current: 340/407.1
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04

Abstract

A haptic conversion system is provided that intercepts frames of audio data, such as a digital audio signal, converts the frames into a haptic signal, and plays the created haptic signal through an actuator to produce haptic effects. The haptic signal is based on a maximum value of each audio data frame, which defines a magnitude of the haptic signal. The haptic signal is applied to the actuator configured to receive the haptic signal, where the actuator utilizes the haptic signal to generate the one or more haptic effects.

Claims

1. A computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to convert an audio signal into one or more haptic effects, the converting comprising:

receiving a data frame of the audio signal;

generating a maximum amplitude value of the data frame;

transforming the data frame;

generating at least one sine carrier waveform based on a maximum amplitude value of the transformed data frame by calculating a sine value of the transformed data frame based on a frequency of the transformed data frame and a phase of the transformed data frame, and multiplying the calculated sine value by the maximum amplitude value of the transformed data frame;

mixing the transformed data frame with the at least one sine carrier waveform to generate a modulated data frame;

generating a haptic signal based on the maximum amplitude value of the data frame and the modulated data frame, wherein the maximum amplitude value of the data frame determines whether the haptic signal is generated, and wherein the modulated data frame defines a waveform of the haptic signal; and

sending the haptic signal to an actuator to generate the one or more haptic effects.

2. The computer-readable medium of claim 1, wherein the haptic signal is generated only when the maximum amplitude value of the data frame is greater than a specified threshold.

3. The computer-readable medium of claim 1, wherein the haptic signal comprises a waveform effect.

4. The computer-readable medium of claim 1, wherein transforming the data frame comprises band-pass filtering the data frame.

5. The computer-readable medium of claim 1, wherein transforming the data frame comprises low-pass filtering the data frame.

6. The computer-readable medium of claim 1, the converting further comprising resampling the data frame before the data frame is transformed.

7. The computer-readable medium of claim 1, the converting further comprising boosting the audio signal contained within the data frame after the data frame is transformed.

8. The computer-readable medium of claim 1, wherein the transformed data frame is mixed with the at least one sine carrier waveform according to a formula where the mix of the transformed data frame and the at least one sine carrier waveform is normalized.

9. The computer-readable medium of claim 1, the converting further comprising extracting a frequency content from the data frame using a fast fourier transformation of the data frame.

10. The computer-readable medium of claim 1, the converting further comprising extracting a frequency content from the data frame using a band-pass filter.

11. The computer-readable medium of claim 1, the converting further comprising:

dividing the data frame into low frequency content, middle frequency content, and high frequency content; and

extracting the middle frequency content and high frequency content.

12. A computer-implemented method for converting an audio signal into one or more haptic effects, the computer-implemented method comprising:

receiving a data frame of the audio signal;

generating a maximum amplitude value of the data frame;

transforming the data frame;

generating at least one sine carrier waveform based on a maximum amplitude value of the transformed data frame by calculating a sine value of the transformed data frame based on a frequency of the transformed data frame and a phase of the transformed data frame, and multiplying the calculated sine value by the maximum amplitude value of the transformed data frame;

mixing the transformed data frame with the at least one sine carrier waveform to generate a modulated data frame;

generating a haptic signal based on the maximum amplitude value of the data frame and the modulated data frame, wherein the maximum amplitude value of the data frame determines whether the haptic signal is generated, and wherein the modulated data frame defines a waveform of the haptic signal; and

sending the haptic signal to an actuator to generate the one or more haptic effects.

13. The computer-implemented method of claim 12, wherein the haptic signal is generated only when the maximum amplitude value of the data frame is greater than a specified threshold.

14. The computer-implemented method of claim 12, wherein the haptic signal comprises a waveform effect.

15. The computer-implemented method of claim 12, wherein transforming the data frame comprises band-pass filtering the data frame.

16. The computer-implemented method of claim 12, further comprising extracting a frequency content from the data frame using a fast fourier transformation of the data frame.

17. The computer-implemented method of claim 12, further comprising extracting a frequency content from the data frame using a band-pass filter.

18. The computer-implemented method of claim 12, further comprising:

dividing the data frame into low frequency content, middle frequency content, and high frequency content; and

extracting the middle frequency content and high frequency content.

19. A haptic conversion system comprising:

a memory configured to store a haptic conversion module, wherein the haptic conversion module comprises a reverb module, a boombox module, and a transformation module;

a processor configured to execute the haptic conversion module stored on the memory; and

an actuator configured to output one or more haptic effects;

wherein the reverb module is configured to receive a data frame of the audio signal;

wherein the boombox module is configured to generate a maximum amplitude value of the data frame;

wherein the transformation module is configured to transform the data frame;

wherein the boombox module is further configured to generate at least one sine carrier waveform based on a maximum amplitude value of the transformed data frame by calculating a sine value of the transformed data frame based on a frequency of the transformed data frame and a phase of the transformed data frame, and multiplying the calculated sine value by the maximum amplitude value of the transformed data frame;

wherein the boombox module is further configured to mix the transformed data frame with the at least one sine carrier waveform to generate a modulated data frame;

wherein the reverb module is further configured to generate a haptic signal based on the maximum amplitude value of the data frame and the modulated data frame, wherein the maximum amplitude value of the data frame determines whether the haptic signal is generated, and wherein the modulated data frame defines a waveform of the haptic signal; and

wherein the reverb module is further configured to send the haptic signal to an actuator to generate the one or more haptic effects.

20. The haptic conversion system of claim 19, wherein the reverb module is further configured to generate the haptic signal only when the maximum amplitude value of the data frame is greater than a specified threshold.

21. The haptic conversion system of claim 19, wherein the haptic signal comprises a waveform effect.

22. The haptic conversion system of claim 19, wherein the transformation module further comprises a band-pass filter configured to band-pass filter the data frame.

23. The haptic conversion system of claim 19, wherein the boombox module is further configured to extract a frequency content from the data frame using a fast fourier transformation of the data frame.

24. The haptic conversion system of claim 19, wherein the boombox module is further configured to extract a frequency content from the data frame using a band-pass filter.

25. The haptic conversion system of claim 19, wherein the boombox module is further configured to:

divide the data frame into low frequency content, middle frequency content, and high frequency content; and

extract the middle frequency content and high frequency content.