Joint and coordinated visual-sonic metaphors for interactive multi-channel data sonification to accompany data visualization

• US 9,646,589 B2
• Filed: 02/07/2014
• Issued: 05/09/2017
• Est. Priority Date: 06/17/2010
• Status: Active Grant

• Current Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Original Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Inventors: Ludwig, Lester F.
• Primary Examiner(s): WARREN, DAVID S
• Application Number:
  US14/176,000
• Publication Number:
  US 20140150629A1
• Time in Patent Office: 1,187 Days
• Field of Search: 84600-602, 84609, 84649, 84622-623, 84659
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 1/06 Circuits for establishing the harmonic content of tones , or other arrangements for changing the tone colour
• CPC Class Codes: G10H 2210/301 Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
• CPC Class Codes: G10H 2220/351 Environmental parameters, e.g. temperature, ambient light, atmospheric pressure, humidity, used as input for musical purposes
• CPC Class Codes: G10H 2250/471 General musical sound synthesis principles, i.e. sound category-independent synthesis methods

Abstract

Data sonification arrangements for use with data visualization so as to provide parallel perceptual channels for representing complex numerical data to a user seeking to identify correlations within the data are presented. In an implementation, several varying data quantities are represented by time-varying graphics while several other varying data quantities are represented by time-varying sound, both presented to a user to observe correlations between sonic and visual events or trends. Sonification can be used to offload some information-carrying information capacity from a visualization system, while other information can be rendered via both sonification and visualization to provide affirming or orienting redundancy. In an implementation joint and coordinated visual-sonic metaphors are used for this or other purposes. For example, data sonification can include multiple data-modulated sound timbre classes, each rendered within a stereo sound field according to a spatial metaphor that is shared with the visualization.

Claims

1. A joint data sonification and visualization method for representing a first plurality of channels of numerical information via data visualization and a second plurality of channels of numerical information via at least one audio-frequency waveform, the at least one audio-frequency waveform perceived by a user as an audio tone with a plurality of corresponding discernable timbre attributes, further employing spatial sonic rendering, the method comprising:

receiving multidimensional numerical data;

generating a plurality of audio-frequency waveforms, each of the plurality of audio-frequency waveforms comprising an associated fixed audio-frequency and at least one adjustable timbre attribute responsive to an associated timbre control parameter, each timbre control parameter having an associated adjustable value which can be discernibly varied within a timbre space occupied by the plurality of audio-frequency waveforms;

associating aspects of multidimensional numerical data with the at least one timbre control parameter of at least one of the plurality of audio-frequency waveforms from the plurality of audio-frequency waveforms; and

adjusting the adjustable value of each timbre control parameter responsive to an associated first value of the multidimensional numerical data according to a mapping function, the mapping function transforming first values of the multidimensional numerical data to values of associated timbre control parameters,

wherein timbre and spatial location of the at least one of the plurality of audio-frequency waveforms carries information responsive to the first values of the multidimensional numerical data for presentation as data sonification to the user, and

wherein at least a portion of the multidimensional numerical data is provided to data visualization software.

2. The method of claim 1 wherein the at least one of the plurality of audio-frequency waveforms is sonically rendered within a stereo sound field, and the spatial location of the sonic rendering of the at least one of the plurality of audio-frequency waveforms is responsive to second values of the multidimensional numerical data.

3. The method of claim 2 wherein the spatial location of the sonic rendering of the at least one of the plurality of audio-frequency waveforms is made according to another associated mapping function.

4. The method of claim 1 wherein the at least one of the plurality of audio-frequency waveforms is sonically rendered within an at least two-dimensional spatial sound field, and the spatial location of the sonic rendering of the at least one of the plurality of audio-frequency waveforms is responsive to second values of the multidimensional numerical data.

5. The method of claim 1 wherein the values of the multidimensional numerical data are sequentially selected responsive to a position on a trajectory specified by the user through a subset of the multidimensional numerical data, the trajectory rendered in the data visualization.

6. The method of claim 1 wherein the values of the multidimensional numerical data are sequentially selected responsive to a position of a cursor within a rendered data visualization.

7. The method of claim 6 wherein the position of the cursor and the values of the multidimensional numerical data used in the data sonification are sequentially selected according to a trajectory selected through a subset of the multidimensional numerical data.

8. The method of claim 1 wherein the multidimensional numerical data is retrieved from storage.

9. The method of claim 1 wherein the multidimensional numerical data is provided by a live real-time data stream.

10. The method of claim 1 wherein the multidimensional numerical data is created from live sensor data.

11. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving the imposing of indexing on selected data.

12. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving data suppression.

13. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving data value normalization.

14. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving linear transformations.

15. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving affine transformations.

16. The method of claim 1 wherein the mapping function comprises pre-sonification signal processing filtering operations on selected data.

17. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving nonlinear transformations.

18. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving linear predictor operations.

19. The method of claim 1 wherein the mapping function comprises pre-sonification operations involving interpolation operations.

20. The method of claim 1 wherein at least one of the plurality of audio-frequency waveforms comprises a plurality of timbre control parameters, each of the plurality of timbre control parameters affecting the timbre of the at least one of the plurality of audio-frequency waveforms;

and

wherein a mapping element adjusts the adjustable value of each of the plurality of timbre control parameters responsive to the first values of the multidimensional numerical data.

21. The method of claim 1 wherein a selected metaphor is used to automatically generate parameter assignments and graphics rendering operations.

22. The method of claim 1 wherein the first plurality of channels of numerical information and second plurality of channels of numerical information both comprise at least one channel of numerical information in common.

23. The method of claim 1 wherein the data sonification and the data visualization both employ a spatial metaphor.

24. The method of claim 1 wherein the data sonification and the data visualization share at least one data preprocessing operation.

25. The method of claim 1 wherein the data sonification and the data visualization and share a common data preprocessing environment.