Sound data output and manipulation using haptic feedback

• US 7,208,671 B2
• Filed: 02/20/2004
• Issued: 04/24/2007
• Est. Priority Date: 10/10/2001
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Chu, Lonny L.
• Primary Examiner(s): DONELS, JEFFREY
• Application Number:
  US10/781,918
• Publication Number:
  US 20040161118A1
• Time in Patent Office: 1,159 Days
• Field of Search: 318/568.11, 318/568.25, 345/156, 345/157, 345/161, 345/163, 345/167, 345/146, 345/326, 345/333, 744/71, 463/30, 463/37, 463/38, 434/45, 395/99, 244/233, 846/09
• US Class Current: 84/609
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/36 Accompaniment arrangements
• CPC Class Codes: G10H 2210/101 Music Composition or musical creation; Tools or processes therefor
• CPC Class Codes: G10H 2210/241 Scratch effects, i.e. emulating playback velocity or pitch manipulation effects normally obtained by a disc-jockey manually rotating a LP record forward and backward
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/311 with controlled tactile or haptic feedback effect; output interfaces therefor
• CPC Class Codes: G10H 2240/031 File merging MIDI, i.e. merging or mixing a MIDI-like file or stream with a non-MIDI file or stream, e.g. audio or video
• CPC Class Codes: H01H 2215/03 Sound
• CPC Class Codes: H01H 25/002 having an operating member rectilinearly slidable in different directions
• CPC Class Codes: H01H 25/008 Operating part movable both angularly and rectilinearly, the rectilinear movement being perpendicular to the axis of angular movement
• CPC Class Codes: H04H 60/04 Studio equipment; Interconnection of studios
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers

Abstract

Sound data output and manipulation with haptic feedback. Haptic sensations are associated with sound data to assist in navigating through and editing the sound data. The sound data is loaded into computer memory and played such that sound is output from an audio device. The sound playing is controlled by user input for navigation through the sound data. Haptic commands are generated based on the sound data and are used to output haptic sensations to the user by a haptic feedback device manipulated by the user. The haptic sensations correspond to one or more characteristics of the sound data to assist the user in discerning features of the sound data during the navigation through and editing of the sound data.

Claims

1. A method, comprising:

generating an audio signal based on sound data, the audio signal configured to produce sound from an audio device;

generating a haptic command based on the sound data, the haptic command configured to cause a haptic feedback device to output a haptic sensation, the haptic sensation being associated with at least one characteristic of the sound data; and

receiving a navigation command associated with the sound data and based, at least in part, on the haptic sensation.

2. The method of claim 1, further comprising:

receiving a speed command associated with the sound data, the audio signal being further based on the speed command.

3. The method of claim 1, further comprising:

receiving a direction command associated with the sound data, the audio signal being further based on the direction command, the direction command having a value associated with one of a forward direction and a reverse direction.

4. The method of claim 1, the haptic command being from a plurality of haptic commands, the haptic sensation being from a plurality of haptic sensations, wherein:

the plurality of haptic commands is configured to output the plurality of haptic sensations continuously when the sound is produced by the audio device in response to the audio signal. each haptic sensation from the plurality of haptic sensations has a magnitude associated with an amplitude of the sound data.

5. The method of claim 1, the haptic command being from a plurality of haptic commands, the haptic sensation being from a plurality of haptic sensations, wherein:

the plurality of haptic commands is configured to output the plurality of haptic sensations continuously when the sound is produced by the audio device in response to the audio signal, each haptic sensation from the plurality of haptic sensations has a magnitude directly proportional to an amplitude of the sound data.

6. The method of claim 1, the haptic command being from a plurality of haptic commands, the haptic sensation being from a plurality of haptic sensations, wherein:

the plurality of haptic commands is configured to output the plurality of haptic sensations continuously when the sound is produced by the audio device in response to the audio signal, each haptic sensation from the plurality of haptic sensations has a magnitude inversely proportional to an amplitude of the sound data.

7. The method of claim 1, wherein the haptic command is configured to output the haptic sensation only when the sound having a predetermined characteristic is produced by the audio device in response to the audio signal.

8. The method of claim 1, wherein the haptic command is configured to output the haptic sensation only when the sound having a predetermined characteristic is produced by the audio device in response to the audio signal, the predetermined characteristic including a rise in amplitude of the sound data over a predetermined threshold.

9. The method of claim 1, wherein:

the haptic command is configured to output the haptic sensation only when the sound is produced by the audio device in response to the audio signal, the sound has a plurality of predetermined characteristics, the plurality of predetermined characteristics including a first predetermined characteristic and a second predetermined characteristic, the first predetermined characteristic is a rise in amplitude of the sound data over a predetermined threshold, the second predetermined characteristic is a drop in amplitude of the sound data, the second predetermined characteristic being after the first predetermined characteristic.

10. The method of claim 1, wherein:

the haptic command is configured to output the haptic sensation only when the sound is produced by the audio device in response to the audio signal, the sound has a predetermined characteristic, the predetermined characteristic includes a minimum time interval between a first amplitude peak of the sound and a second amplitude peak of the sound, the haptic command configured to output the haptic sensation substantially during the second amplitude peak of the sound.

11. The method of claim 1, the audio signal configured to produce sound having a first set of frequencies and a second set of frequencies different from the first set of frequencies. the method further comprising:

filtering the sound data such that the first set of frequencies of the sound produced in response to the audio signal is removed and the second set of frequencies of the sound produced in response to the audio signal remains.

12. The method of claim 1, the audio signal configured to produce sound having a first set of frequencies and a second set of frequencies different from the first set of frequencies, the method further comprising:

filtering the sound data such that the first set of frequencies of the sound produced in response to the audio signal is removed and the second set of frequencies of the sound produced in response to the audio signal remains, the second set of frequencies of the sound produced in response to the audio signal is associated with a particular type of haptic sensation.

13. The method of claim 1, wherein the haptic command is configured to output the haptic sensation if a marker associated with the sound data is reached during the sound being produced in response to the audio signal.

14. The method of claim 1, further comprising:

storing a portion of the sound data in a secondary buffer; and

processing the portion of the sound data to identify at least one characteristic of the sound data, the processing being performed in substantially real-time during the sound being produced by the audio device in response to the audio signal.

15. The method of claim 1, further comprising:

sending a display signal associated with a visual representation of the sound produced from the audio device in response to the audio signal; and

sending a cursor signal associated with the visual representation, the cursor signal associated with a part of the sound being produced from the audio device in response to the audio signal.

16. A method, comprising:

identifying a feature of a sound data, the feature having at least one predetermined characteristic, the sound data being associated with a sound waveform;

generating a marker associated with a location of the feature in the sound data based on the identifying, the location of the feature being associated with haptic feedback; and

sending a signal when the location of sound being output based on the sound data corresponds to the location associated with the marker, the signal being associated with the haptic feedback.

17. The method of claim 16, wherein the at least one predetermined characteristic includes a rise in amplitude of the sound waveform associated with the sound data the rise in amplitude being over a predetermined threshold amplitude.

18. The method of claim 16, wherein the at least one predetermined characteristic includes a drop in amplitude of the sound waveform of a predetermined amount after a rise in amplitude of the sound waveform.

19. The method of claim 16, wherein the at least one predetermined characteristic includes a minimum time interval between a first amplitude peak of the sound waveform and a second amplitude peak of the sound waveform, the haptic feedback is based on the sound data and associated with the second amplitude peak of the sound waveform.

20. The method of claim 16, the sound data having a first set of frequencies and a second set of frequencies different from the first set of frequencies, wherein the identifying includes filtering the sound data such that the first set of frequencies of the sound data is removed.

21. The method of claim 16, wherein a type of the haptic feedback is based on the at least one predetermined characteristic of the feature to which the associated marker refers.

22. A processor readable medium comprising code representing instructions to cause a processor to perform a method, the method comprising:

generating an audio signal based on a sound data, the audio signal configured to produce sound from an audio device;

generating a haptic command based on the sound data, the haptic command configured to cause a haptic feedback device to output a haptic sensation, the haptic sensation being associated with at least one characteristic of the sound data; and

receiving a navigation command associated with the sound data and based, at least in part, on the haptic sensation.

23. The method of claim 22, the haptic command being from a plurality of haptic commands, the haptic sensation being from a plurality of haptic sensations, wherein:

the plurality of haptic commands being configured to output the plurality of haptic sensations continuously when the sound is produced by the audio device in response to the audio signal, each haptic sensation from the plurality of haptic sensations has a magnitude associated with an amplitude of the sound data.

24. The method of claim 22, the audio signal configured to produce sound having a first set of frequencies and a second set of frequencies different from the first set of frequencies, the method further comprising:

filtering the sound data such that the first set of frequencies of the sound produced in response to the audio signal is removed and the second set of frequencies of the sound produced in response to the audio signal remains, the second set of frequencies of the sound produced in response to the audio signal is associated with a particular type of haptic sensation.

25. The method of claim 22, wherein the haptic command is configured to output the haptic sensation if a marker associated with the sound data is reached during the sound being produced in response to the audio signal.

26. A processor readable medium comprising code representing instructions to cause a processor to perform a method, the method comprising:

identifying a feature of a sound data, the feature having at least one predetermined characteristic, the sound data being associated with a sound waveform;

generating a marker associated with a location of the feature in the sound data based on the identifying, the location of the feature being associated with at least one haptic feedback; and

sending a signal when the location of sound being output based on the sound data corresponds to the location associated with the marker, the signal being associated with the at least one haptic feedback.

27. The method of claim 26, the sound data having a first set of frequencies and a second set of frequencies different from the first set of frequencies, wherein the method further comprises:

filtering the sound data such that the first set of frequencies of the sound data is removed.

28. The method of claim 26, wherein a type of the at least one haptic feedback is based on the at least one predetermined characteristic of the feature to which the associated marker refers.