Sound to haptic effect conversion system using waveform

• US 10,055,950 B2
• Filed: 02/15/2017
• Issued: 08/21/2018
• Est. Priority Date: 02/11/2011
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Bhatia, Satvir Singh, Gandhi, Kanav, Ullrich, Christopher J., Cruz-Hernandez, Juan Manuel, Timone, Herve Thu, Lau, Jason Hoi Fun
• Primary Examiner(s): Tweel, Jr., John A
• Application Number:
  US15/433,488
• Publication Number:
  US 20170213430A1
• Time in Patent Office: 552 Days
• Field of Search: 3404071
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04

Abstract

A haptic conversion system is provided that intercepts frames of audio data, such as a digital audio signal, converts the frames into a haptic signal, and plays the created haptic signal through an actuator to produce haptic effects. The haptic signal is based on a maximum value of each audio data frame, which defines a magnitude of the haptic signal. The haptic signal is applied to the actuator configured to receive the haptic signal, where the actuator utilizes the haptic signal to generate the one or more haptic effects.

Claims

1. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to convert an audio signal into a haptic effect, the converting comprising:

receiving an audio signal comprising a first data frame and a second data frame;

determining a maximum amplitude value of the first data frame;

generating a haptic signal based on the maximum amplitude value of the first data frame; and

sending the haptic signal to an actuator to generate the haptic effect.

2. The non-transitory computer-readable medium of claim 1, the converting further comprising:

decoding the audio signal into the first data frame and the second data frame.

3. The non-transitory computer-readable medium of claim 2, the converting further comprising determining a maximum amplitude value of the second data frame.

4. The non-transitory computer-readable medium of claim 1, the converting further comprising transforming the first data frame to generate a transformed first data frame.

5. The non-transitory computer-readable medium of claim 4, the converting further comprising modulating the transformed first data frame.

6. The non-transitory computer-readable medium of claim 1, further comprising transforming the first data frame by band-pass filtering the first data frame.

7. The non-transitory computer-readable medium of claim 1, further comprising transforming the first data frame by low-pass filtering the first data frame.

8. The non-transitory computer-readable medium of claim 1, further comprising boosting the audio signal contained within the first data frame after a transforming of the first data frame.

9. The non-transitory computer-readable medium of claim 1, wherein the haptic signal is generated only when the maximum amplitude value of the first data frame is greater than a specified threshold.

10. A computer-implemented method for converting an audio signal into a haptic effect, the computer-implemented method comprising:

receiving an audio signal;

decoding the audio signal into a first data frame and second data frame;

determining a maximum amplitude value of the first data frame;

generating a haptic signal based on the maximum amplitude value of the first data frame; and

sending the haptic signal to an actuator to generate the haptic effect.

11. The computer-implemented method of claim 10, further comprising:

determining a maximum amplitude value of the second data frame and generating a haptic signal based on the maximum amplitude value of the second data frame.

12. The computer-implemented method of claim 10, further comprising transforming the first data frame to generate a transformed first data frame.

13. The computer-implemented method of claim 12, further comprising modulating the transformed first data frame.

14. A haptic conversion system comprising:

a device configured to receive an audio signal comprising a set of samples; and

an actuator configured to output a haptic effect,

wherein the device, upon receiving the set of samples of the audio signal, is configured to

resample the audio signal to generate a resampled audio signal,

transform a set of samples of the resampled audio signal to generate a transformed set of samples, wherein the transforming comprises boosting the set of samples of the resampled audio signal, and

generate a haptic signal based on the transformed set of samples, and

send the haptic signal to the actuator to output the haptic effect.

15. The haptic conversion system of claim 14, wherein the device is further configured to modulate the transformed set of samples to generate a modulated set of samples, wherein the haptic signal is generated based on the modulated set of samples.

16. The haptic conversion system of claim 14, wherein the device is further configured to determine a maximum amplitude value of the set of samples of the audio signal, wherein the haptic signal is generated only when the maximum amplitude value of the set of samples is greater than a specified threshold.

17. The haptic conversion system of claim 14, wherein the set of samples of the audio signal is part of a data frame, wherein the device includes at least a communication device configured to receive the audio signal and a processor configured to generate the haptic signal, and wherein the haptic conversion system is part of a mobile device.

18. A computer-implemented method for converting an audio signal into one or more haptic effects, the computer-implemented method comprising:

receiving a set of samples of the audio signal;

transforming the set of samples to generate a transformed set of samples;

generating a haptic signal by mixing the transformed set of samples with a periodic waveform; and

sending the haptic signal to an actuator to generate the one or more haptic effects.

19. The computer-implemented method of claim 18,

wherein transforming the set of samples comprises filtering the set of samples to generate a filtered set of samples,

wherein the periodic waveform has an amplitude value that is based on a maximum amplitude value of the filtered set of samples, and

wherein the mixing comprises generating a weighted sum of the transformed set of samples and the periodic waveform.

20. The computer-implemented method of claim 19, wherein the weighted sum is generated as a normalized value that is a sum of a first multiple and a second multiple, wherein the first multiple is a first value multiplied by the transformed set of samples, and the second multiple is a second value multiplied by the periodic waveform, wherein the second value is equal to 1 minus the first value, the first value being a decimal value greater than 0 and less than 1.