Multi-channel data sonification system with partitioned timbre spaces including periodic modulation techniques

• US 10,365,890 B2
• Filed: 07/30/2018
• Issued: 07/30/2019
• Est. Priority Date: 06/17/2010
• Status: Active Grant

• Current Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Original Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Inventors: Ludwig, Lester F.
• Primary Examiner(s): Warren, David S
• Application Number:
  US16/049,573
• Publication Number:
  US 20180336012A1
• Time in Patent Office: 365 Days
• Field of Search: 84622, 84659
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 1/06 Circuits for establishing the harmonic content of tones , or other arrangements for changing the tone colour
• CPC Class Codes: G10H 2210/301 Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
• CPC Class Codes: G10H 2220/351 Environmental parameters, e.g. temperature, ambient light, atmospheric pressure, humidity, used as input for musical purposes
• CPC Class Codes: G10H 2250/471 General musical sound synthesis principles, i.e. sound category-independent synthesis methods details of musical waveform synthesis G10H2250/541; special instrument [spint] G10H2230/045; sound category-specific synthesis G10H2250/315

Abstract

A data sonification system for representing a plurality of channels of numerical information is described wherein the rate of the periodic modulation is itself used as a vehicle for carrying information of at least one component of the multidimensional numerical data. The data sonification system includes a plurality of audio waveform generator elements. Each of the audio waveform generator elements generates an associated audio frequency waveform. Each audio frequency waveform has an audio frequency parameter and at least one timbre modulation parameter having a settable value. The timbre modulation parameter affects the timbre of the audio waveform. The data sonification system includes a mapping element for associating aspects of multidimensional numerical data with the timbre modulation parameter of each audio frequency waveform. The mapping element sets the value of the timbre modulation parameter in response to multidimensional numerical data and/or periodic modulation.

Claims

1. A data sonification system for representing a plurality of channels of numerical information via a plurality of corresponding discernable variations of at least one of a plurality of audio-frequency waveforms, the at least one of the plurality of the audio-frequency waveforms being perceivable by a user as comprising a plurality of audio tones having at least one corresponding discernable timbre attribute, the data sonification system comprising:

a plurality of audio-frequency waveform generators, each of the plurality of audio-frequency waveform generators generating an associated one of the plurality of the audio-frequency waveforms, wherein each of the associated ones of the plurality of the audio-frequency waveforms comprises an audio-frequency and at least one adjustable timbre control parameter, each of the at least one of the adjustable timbre control parameter having an associated adjustable value which can be discernibly varied within a timbre space occupied by the plurality of the audio-frequency waveforms; and

a mapping element for mapping aspects of multidimensional numerical data with the at least one of the adjustable timbre control parameters of each of the associated ones of the plurality of audio-frequency waveforms, wherein for the associated ones of the plurality of the audio-frequency waveforms the mapping element adjusts a value of a periodic modulation of the at least one of the adjustable timbre control parameters responsive to values of the multidimensional numerical data,

wherein the rate of the periodic modulation is itself used as a vehicle for carrying information of at least one component of the multidimensional numerical data,

wherein the timbre of each of the other the audio-frequency waveform carries information responsive to values of multidimensional numerical data for presentation to a user, and

wherein the associated ones of the plurality of the audio-frequency waveforms and their respective adjustable timbres are arranged to provide a plurality of data-modulated sound timbre classes within the timbre space.

2. The data sonification system of claim 1 wherein the at least one of the plurality of the audio-frequency waveforms comprises a plurality of timbre control parameters, each of the plurality of the timbre control parameters affecting the timbre of the at least one of the plurality of the audio-frequency waveforms;

and wherein the mapping element adjusts the value of each of the plurality of the timbre control parameters responsive to values of the multidimensional numerical data.

3. The data sonification system of claim 1, wherein the sound timbre classes can be arranged to convey a sonic metaphor.

4. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected by the user from the multidimensional numerical data, the values sequentially selected over an interval of time.

5. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected according to a trajectory selected through a subset of the multidimensional numerical data.

6. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected responsive to a position on a trajectory specified by the user through a subset of the multidimensional numerical data, the trajectory rendered in a data visualization.

7. The data sonification system of claim 1, wherein the values of the multidimensional numerical data are sequentially selected responsive to a position of a cursor within a rendered data visualization.

8. The data sonification system of claim 1, wherein the multidimensional numerical data is retrieved from storage.

9. The data sonification system of claim 1, wherein the multidimensional numerical data is provided by a live real-time data stream.

10. The data sonification system of claim 1, wherein the multidimensional numerical data is created from live sensor data.

11. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving the imposing of indexing on selected data.

12. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving data suppression.

13. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving data value normalization.

14. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving linear transformations.

15. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving affine transformations.

16. The data sonification system of claim 1, wherein the mapping comprises pre-sonification signal processing filtering operations on selected data.

17. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving nonlinear transformations.

18. The data sonification system of claim 1, wherein the mapping comprises pre-sonification operations involving linear predictor operations.

19. The data sonification system of claim 1, wherein the at least one timbre control parameter controls a width of a pulse waveform.

20. A data sonification system for representing a plurality of channels of numerical information via a plurality of corresponding discernable variations of at least one of a plurality of audio-frequency waveforms, the at least one of the plurality of the audio-frequency waveforms being perceivable by a user as comprising a plurality of audio tones having at least one corresponding discernable timbre attribute, the data sonification system comprising:

a plurality of audio-frequency waveform generators, each of the plurality of audio-frequency waveform generators generating an associated one of the plurality of the audio-frequency waveforms, wherein each of the associated ones of the plurality of the audio-frequency waveforms comprises an audio-frequency and at least one adjustable timbre control parameter, each of the at least one of the adjustable timbre control parameter having an associated adjustable value which can be discernibly varied within a timbre space occupied by the plurality of the audio-frequency waveforms; and

a mapping element for mapping aspects of multidimensional numerical data with the at least one of the adjustable timbre control parameters of each of the associated ones of the plurality of the audio-frequency waveforms,

wherein for the associated ones of the plurality of the audio-frequency waveforms the mapping element adjusts a value of the at least one of the adjustable timbre control parameters responsive to values of the multidimensional numerical data,

wherein for the plurality of the audio-frequency waveforms the timbre of the associated ones of the audio-frequency waveform carries information responsive to values of underlying data for presentation to a user,

wherein the associated ones of the plurality of the audio-frequency waveforms and their respective adjustable timbres are arranged to provide a plurality of data-modulated sound timbre classes within the timbre space,

wherein the pitch of at least one of the audio-frequency waveforms is varied periodically, and

wherein the rate of the periodic variation in pitch is itself used as a vehicle for carrying information of at least one component of the multidimensional numerical data.