Sound data output and manipulation using haptic feedback

• US 20030068053A1
• Filed: 10/10/2001
• Published: 04/10/2003
• Est. Priority Date: 10/10/2001
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Chu, Lonny L.
• Granted Patent: US 6,703,550 B2
• Time in Patent Office: Days
• US Class Current: 381/118
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/36 Accompaniment arrangements
• CPC Class Codes: G10H 2210/101 Music Composition or musical creation; Tools or processes therefor
• CPC Class Codes: G10H 2210/241 Scratch effects, i.e. emulating playback velocity or pitch manipulation effects normally obtained by a disc-jockey manually rotating a LP record forward and backward
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/311 with controlled tactile or haptic feedback effect; output interfaces therefor
• CPC Class Codes: G10H 2240/031 File merging MIDI, i.e. merging or mixing a MIDI-like file or stream with a non-MIDI file or stream, e.g. audio or video
• CPC Class Codes: H01H 2215/03 Sound
• CPC Class Codes: H01H 25/002 having an operating member rectilinearly slidable in different directions
• CPC Class Codes: H01H 25/008 Operating part movable both angularly and rectilinearly, the rectilinear movement being perpendicular to the axis of angular movement
• CPC Class Codes: H04H 60/04 Studio equipment; Interconnection of studios
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers

Abstract

Sound data output and manipulation with haptic feedback. Haptic sensations are associated with sound data to assist in navigating through and editing the sound data. The sound data is loaded into computer memory and played such that sound is output from an audio device. The sound playing is controlled by user input for navigation through the sound data. Haptic commands are generated based on the sound data and are used to output haptic sensations to the user by a haptic feedback device manipulated by the user. The haptic sensations correspond to one or more characteristics of the sound data to assist the user in discerning features of the sound data during the navigation through and editing of the sound data.

Claims

1. A method for associating haptic sensations with sound data to assist in navigating through and editing said sound data, the method comprising:

loading at least a portion of said sound data into a memory of a computer;

playing said sound data such that an audio signal is generated and used for outputting sound from an audio device, wherein said playing of said sound is controlled by user input received by said computer from a user for navigation through said sound data; and

generating haptic commands based on said sound data, said haptic commands used to output haptic sensations to said user by a haptic feedback device manipulated by said user and in communication with said computer, said haptic sensations corresponding to one or more characteristics of said sound data to assist said user in discerning features of said sound data during said navigation through and editing of said sound data.

2. A method as recited in claim 1 wherein said user can control a speed of said playing of said sound data.

3. A method as recited in claim 1 wherein said user can control a direction of said playing of said sound data, said direction including forward and reverse.

4. A method as recited in claim 1 wherein said haptic sensations are continuously output during said playing of said sound data, and wherein an output haptic sensation has a magnitude based on an amplitude of said sound data currently being played.

5. A method as recited in claim 4 wherein said haptic sensations have a magnitude directly proportional to said amplitude of said played sound data.

6. A method as recited in claim 4 wherein said haptic sensations have a magnitude inversely proportional to said amplitude of said played sound data.

7. A method as recited in claim 1 wherein said haptic sensations are output only when features of said sound data having predetermined characteristics are played.

8. A method as recited in claim 7 wherein said predetermined characteristics include a rise in amplitude of said sound data over a predetermined threshold.

9. A method as recited in claim 8 wherein said predetermined characteristics include a drop in amplitude of a predetermined amount following said rise in amplitude.

10. A method as recited in claim 7 wherein said predetermined characteristics include a minimum time interval that must occur between two peaks in amplitude to allow a haptic sensation to be output when the second of said peaks is played.

11. A method as recited in claim 1 further comprising filtering said sound data such that undesired frequencies of sound generated by said sound data are removed and a desired range of frequencies remain.

12. A method as recited in claim 11 wherein said desired range of frequencies are associated with a particular type of haptic sensation.

13. A method as recited in claim 1 wherein one of said haptic commands is generated and a haptic sensation output if a marker that was previously stored for a particular location in said sound data is reached during said playing of said sound.

14. A method as recited in claim 1 further comprising storing a portion of said sound data in a secondary buffer and processing said portion of said sound data to find said one or more characteristics in said sound data real-time during said playing of said sound data.

15. A method as recited in claim 1 further comprising displaying a visual representation of said sound data and a moving cursor to indicate a current part of said sound data that is being played.

16. A method for pre-processing sound data to allow haptic sensations associated with said sound data to be output when said sound data is played to a user, the method comprising:

loading at least a portion of said sound data into a memory of a computer, said sound data describing a sound waveform;

processing said sound data to find a sound feature having one or more predetermined characteristics; and

when said sound feature is found, storing a marker in a list of markers, wherein said marker indicates a location of said associated sound feature in said sound data, said location to be associated with at least one haptic sensation such that when said sound data is played, said associated at least one haptic sensation is output to a user when said marker is reached during said playing of said sound data.

17. A method as recited in claim 16 wherein said one or more predetermined characteristics include a rise in amplitude of said sound waveform described by said sound data, said rise in amplitude being over a predetermined threshold amplitude.

18. A method as recited in claim 16 wherein said one or more predetermined characteristics include a drop in amplitude of a predetermined amount following said rise in amplitude.

19. A method as recited in claim 16 wherein said one or more predetermined characteristics include a minimum time interval that must occur between two peaks in amplitude of said sound waveform to allow a haptic sensation to be output when the second of said peaks is played.

20. A method as recited in claim 16 wherein said processing of said sound data includes filtering said sound data such that undesired frequencies of said sound data are removed.

21. A method as recited in claim 16 further comprising storing an indication of one or more particular haptic sensations that are to be associated with a particular one of said markers.

22. A method as recited in claim 21 wherein a type of said one more particular haptic sensations is based on said one or more predetermined characteristics of said sound feature to which said particular marker refers.

23. A computer readable medium including program instructions for performing steps that associate haptic sensations with sound data to assist in navigating through and editing said sound data, the steps comprising:

loading at least a portion of said sound data into a memory of a computer;

playing said sound data such that an audio signal is generated and used for outputting audio sound from an audio device, wherein said playing of said sound is controlled by user input received by said computer from a user for navigation of said sound data; and

generating haptic commands based on said sound data, said haptic commands used to output haptic sensations to said user by a haptic feedback device manipulated by said user and in communication with said computer, said haptic sensations corresponding to one or more characteristics of said sound data to assist said user in discerning features of said sound data during said navigation through and editing of said sound data.

24. A method as recited in claim 23 wherein said user can control a speed of said playing of said sound data.

25. A method as recited in claim 23 wherein said user can control a direction of said playing of said sound data, said direction including forward and reverse.

26. A method as recited in claim 23 wherein said haptic sensations are continuously output during said playing of said sound data, and wherein an output haptic sensation has a magnitude based on an amplitude of said sound data currently being played.

27. A method as recited in claim 23 wherein said haptic sensations are output only when features of said sound data having predetermined characteristics are played.

28. A method as recited in claim 23 wherein one of said haptic commands is generated and a haptic sensation output if a marker that was previously stored for a particular location in said sound data is reached during said playing of said sound.

29. A method as recited in claim 23 further comprising storing a portion of said sound data in a secondary buffer and processing said portion of said sound data to find said one or more characteristics in said sound data real-time during said playing of said sound data.