Multi-channel data sonification employing data-modulated sound timbre classes

• US 10,037,186 B2
• Filed: 05/03/2017
• Issued: 07/31/2018
• Est. Priority Date: 06/17/2010
• Status: Active Grant

• Current Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Original Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Inventors: Ludwig, Lester F.
• Primary Examiner(s): Warren, David
• Application Number:
  US15/585,702
• Publication Number:
  US 20170235548A1
• Time in Patent Office: 454 Days
• Field of Search: 84600-602, 84609, 84649, 84622, 84623, 84659
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 1/06 Circuits for establishing the harmonic content of tones , or other arrangements for changing the tone colour
• CPC Class Codes: G10H 2210/301 Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
• CPC Class Codes: G10H 2220/351 Environmental parameters, e.g. temperature, ambient light, atmospheric pressure, humidity, used as input for musical purposes
• CPC Class Codes: G10H 2250/471 General musical sound synthesis principles, i.e. sound category-independent synthesis methods

Abstract

Multichannel data sonification arrangements for providing parallel perceptual channels for representing complex numerical data to a user seeking to identify correlations within the data are presented. In an implementation, several varying data quantities are represented by time-varying sound and presented to a user to observe both variations and correlations among sonic events or trends. The data sonification can include multiple data-modulated sound timbre classes, for example each separately rendered within a stereo sound field. The sound timbre classes can be arranged to convey a sonic metaphor.

Claims

1. A data sonification system for representing a plurality of channels of numerical information via a plurality of corresponding discernable variations of timbre of at least one of a plurality of audio-frequency waveforms, the at least one of the plurality of the audio-frequency waveforms being perceivable by a user as comprising a plurality of audio tones having at least one corresponding discernable timbre attribute, the data sonification system comprising:

a plurality of audio-frequency waveform generators, each of the plurality of audio-frequency waveform generators generating an associated one of the plurality of the audio-frequency waveforms, wherein each of the associated ones of the plurality of the audio-frequency waveforms comprises a fixed audio-frequency and at least one adjustable timbre control parameter, each of the at least one of the adjustable timbre control parameter having an associated adjustable value which can be discernibly varied within a timbre space occupied by the plurality of the audio-frequency waveforms; and

a mapping element for mapping aspects of multidimensional numerical data with the at least one of the adjustable timbre control parameters of each of the associated ones of the plurality of the audio-frequency waveforms,

wherein for the associated ones of the plurality of the audio-frequency waveforms the mapping element adjusts a value of the at least one of the adjustable timbre control parameters responsive to values of the multidimensional numerical data,

wherein for the plurality of the audio-frequency waveforms the timbre of the associated ones of the audio-frequency waveform carries information responsive to values of underlying data for presentation to a user, and

wherein the associated ones of the plurality of the audio-frequency waveforms and their respective adjustable timbres are arranged to provide a plurality of data-modulated sound timbre classes within the timbre space.

2. The data sonification system of claim 1 wherein the at least one of the plurality of the audio-frequency waveforms comprises a plurality of timbre control parameters, each of the plurality of the timbre control parameters affecting the timbre of the at least one of the plurality of the audio-frequency waveforms;

and wherein the mapping element adjusts the value of each of the plurality of the timbre control parameters responsive to values of the multidimensional numerical data.

3. The data sonification system of claim 1, wherein the sound timbre classes can be arranged to convey a sonic metaphor.

4. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected by the user from the multidimensional numerical data, the values sequentially selected over an interval of time.

5. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected according to a trajectory selected through a subset of the multidimensional numerical data.

6. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected responsive to a position on a trajectory specified by the user through a subset of the multidimensional numerical data, the trajectory rendered in a data visualization.

7. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected responsive to a position of a cursor within a rendered data visualization.

8. The data sonification system of claim 1, wherein the multidimensional numerical data is retrieved from storage.

9. The data sonification system of claim 1, wherein the multidimensional numerical data is provided by a live real-time data stream.

10. The data sonification system of claim 1, wherein the multidimensional numerical data is created from live sensor data.

11. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving the imposing of indexing on selected data.

12. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving data suppression.

13. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving data value normalization.

14. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving linear transformations.

15. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving affine transformations.

16. The data sonification system of claim 1, wherein the mapping comprises pre-sonification signal processing filtering operations on selected data.

17. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving nonlinear transformations.

18. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving linear predictor operations.

19. The data sonification system of claim 1, wherein the at least one timbre control parameter controls a width of a pulse waveform.

20. The data sonification system of claim 1, wherein at least one of the plurality of the audio-frequency waveforms comprises a plurality of the adjustable timbre control parameters, each of the plurality of the adjustable timbre control parameter affecting the timbre of the audio-frequency waveform;

and wherein the mapping adjusts the value of each of the plurality of the adjustable timbre control parameters responsive to the values of the multidimensional numerical data.