Interactive multi-channel data sonification to accompany data visualization with partitioned timbre spaces using modulation of timbre as sonification information carriers

• US 8,440,902 B2
• Filed: 04/18/2012
• Issued: 05/14/2013
• Est. Priority Date: 06/17/2010
• Status: Expired due to Fees

• Current Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Original Assignee: Lester F. Ludwig
• Inventors: Ludwig, Lester F.
• Primary Examiner(s): Warren, David S.
• Application Number:
  US13/450,350
• Publication Number:
  US 20120260790A1
• Time in Patent Office: 391 Days
• Field of Search: 84600-602, 84622-624, 84659-661
• US Class Current: 84/622
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 1/06 Circuits for establishing the harmonic content of tones , or other arrangements for changing the tone colour
• CPC Class Codes: G10H 2210/301 Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
• CPC Class Codes: G10H 2220/351 Environmental parameters, e.g. temperature, ambient light, atmospheric pressure, humidity, used as input for musical purposes
• CPC Class Codes: G10H 2250/471 General musical sound synthesis principles, i.e. sound category-independent synthesis methods

Abstract

Interactive data sonification for representing multidimensional numerical information with a plurality of variable-timbre channels for use complementing data visualization is described. The method includes generation of a plurality of variable-timbre audio waveforms, each having an audio frequency parameter and at least one timbre modulation parameter having an adjustable value that affects the timbre of the audio waveform. The method includes associating aspects of multidimensional numerical data with the timbre modulation parameter of each audio frequency waveform using a mapping element. The mapping element varies values of timbre modulation parameters responsive to selected values from the multidimensional numerical data. Each audio frequency waveform can be positioned within a sonically-rendered sound field, associating information with positions within the sound field. Mapping elements can vary the positions responsive to selected values from the multidimensional numerical data. In an implementation, a cursor moves over visualized data as associated multidimensional numerical data values are sonified.

Claims

1. A data sonification method for representing a plurality of channels of numerical information via a plurality of corresponding discernable variations of timbre of at least one audio-frequency waveform, the audio-frequency waveform perceived by a user as an audio tone with a plurality of corresponding discernable timbre attributes, further employing spatial sound rendering, the method comprising:

receiving multidimensional numerical data;

generating plurality of audio-frequency waveforms, each audio-frequency waveform comprising an associated fixed audio-frequency and at least one adjustable timbre attribute responsive to an associated timbre control parameter, each timbre control parameter having an associated adjustable value which can be discernibly varied within a timbre space occupied by the plurality of audio-frequency waveforms;

associating aspects of multidimensional numerical data with the at least one timbre control parameter of at least one of the audio-frequency waveforms from the plurality of audio-frequency waveforms; and

adjusting the value of each timbre control parameter responsive to an associated first value of the multidimensional numerical data according to a mapping function, the mapping function transforming first values of the multidimensional numerical data to values of associated timbre control parameters,

wherein the timbre and the spatial location of the at least one audio-frequency waveform carries information responsive to values of the multidimensional numerical data for presentation as data sonification to a user, and

wherein at least a portion of the multidimensional numerical data is also used by data visualization software.

2. The method of claim 1 wherein the at least one audio-frequency waveform is sonically rendered within a stereo sound field, and the spatial location of the sonic rendering of the at least one audio-frequency waveform is responsive to second values of the multidimensional numerical data.

3. The method of claim 2 wherein the spatial location of the sonic rendering of the at least one audio-frequency waveform is made according to another associated mapping function.

4. The method of claim 1 wherein the at least one audio-frequency waveform is sonically rendered within an at least two-dimensional spatial sound field, and the spatial location of the sonic rendering of the at least one audio-frequency waveform is responsive to second values of the multidimensional numerical data.

5. The method of claim 1 wherein the values of the multidimensional numerical data are sequentially selected responsive to a position on a trajectory specified by the user through a subset of the multidimensional numerical data, the trajectory rendered in the data visualization.

6. The method of claim 1 wherein the values of the multidimensional numerical data are sequentially selected responsive to a position of a cursor within the rendered data visualization.

7. The method of claim 6 wherein the position of the cursor and the values of the multidimensional numerical data used in the data sonification are sequentially selected according to a trajectory selected through a subset of the multidimensional numerical data.

8. The method of claim 1 wherein the multidimensional numerical data is retrieved from storage.

9. The method of claim 1 wherein the multidimensional numerical data is provided by a live real-time data stream.

10. The method of claim 1 wherein the multidimensional numerical data is created from live sensor data.

11. The method of claim 1 wherein the mapping comprises pre-sonification operations involving the imposing of indexing on selected data.

12. The method of claim 1 wherein the mapping comprises pre-sonification operations involving data suppression.

13. The method of claim 1 wherein the mapping comprises pre-sonification operations involving data value normalization.

14. The method of claim 1 wherein the mapping comprises pre-sonification operations involving linear transformations.

15. The method of claim 1 wherein the mapping comprises pre-sonification operations involving affine transformations.

16. The method of claim 1 wherein the mapping comprises pre-sonification signal processing filtering operations on selected data.

17. The method of claim 1 wherein the mapping comprises pre-sonification operations involving nonlinear transformations.

18. The method of claim 1 wherein the mapping comprises pre-sonification operations involving linear predictor operations.

19. The method of claim 1 wherein the at least one timbre control parameter controls the width of a pulse waveform.

20. The method of claim 1 wherein at least one audio-frequency waveform comprises a plurality of timbre control parameters, each timbre control parameter affecting the timbre of the audio-frequency waveform;

and

wherein the mapping element adjusts the value of each of the plurality of timbre control parameters responsive to values of the multidimensional numerical data.