Virtual musical interface in a haptic virtual environment

• US 20110191674A1
• Filed: 08/04/2005
• Published: 08/04/2011
• Est. Priority Date: 08/06/2004
• Status: Abandoned Application

• Current Assignee: Dentsable Incorporated
• Original Assignee: SensAble Technologies, Inc. (3D Systems Corporation)
• Inventors: Gupta, Vivek, Sharko, John, Grinstein, Georges, Bezzati, Abdulrahmane, Bubert, Paul, Victorine, Jon, Rawley, Curt, Bauman, Alex
• Application Number:
  US11/659,650
• Publication Number:
  US 20110191674A1
• Time in Patent Office: Days
• US Class Current: 715/702
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G10H 1/0025 Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece automatically producing a series of tones G10H1/26
• CPC Class Codes: G10H 2220/101 for graphical creation, edition or control of musical data or parameters
• CPC Class Codes: G10H 2220/311 with controlled tactile or haptic feedback effect; output interfaces therefor
• CPC Class Codes: G10H 2220/401 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing. movement pattern or gesture sensing G10H2220/201; geolocation sensing G10H2220/355, 3D sensing with accelerometer G10H2220/395

Abstract

The invention relates to a virtual musical interface for composing, editing, and/or performing audio works. In certain embodiments, the invention relates to a virtual musical interface rendered as a virtual object in a haptic environment, where one or more parameters of the virtual object are associated with one or more audio attributes. Thus, methods and systems of the invention provide audio, visual, and/or haptic feedback in response to a user interaction.

Claims

1. A method of haptically rendering a virtual musical interface, the method comprising the steps of:

(a) rendering a virtual object in a haptic virtual environment; and

(b) associating at least one parameter of the virtual object with an audio attribute to create a musical interface whereby a user is provided audio and haptic feedback in response to a user interaction.

2-61. -61. (canceled)

62. A method for producing audio output using haptically rendered virtual objects, the method comprising the steps of:

haptically rendering at least one virtual object in a haptic virtual environment, wherein each of the at least one virtual object is mapped to at least one audio parameter;

rendering a volume through which the at least one virtual object travels;

traversing the at least one virtual object; and

producing audio output corresponding to movement of the at least one virtual object through the volume.

63. The method of claim 62 wherein a virtual pointer denotes the virtual object being currently traversed.

64. The method of claim 62 wherein the audio output is modified by modifying the volume through which the at least one virtual object travels.

65. A method of modifying audio attributes of a musical composition, the method comprising:

accessing audio attributes associated with a musical composition;

associating at least one of the audio attributes with a graphical representation of the musical composition; and

modifying the associated audio attribute in response to an interaction directed at the graphical representation of the musical composition.

66. The method of claim 65, wherein the audio attributes include at least one of a tone, volume, pitch, attack rate, decay rate, sustain rate, vibrato, and sostenuto.

67. The method of claim 65, wherein the audio attributes are accessed from a MIDI file.

69. The method of claim 65, wherein the musical composition includes at least one of a voice, a monody, and a polyphony.

70. The method of claim 65, wherein the graphical representation of the musical composition includes at least one of a three-dimensional graphical surface and a three-dimensional volume.

71. The method of claim 70, wherein the graphical representation represents a range of modification of the at least one audio attribute.

72. The method of claim 70, wherein the at least one audio attribute is further associated with at least one of a texture, a coordinate location, a color, a depth, and a height of the graphical representation.

73. The method of claim 65, wherein the interaction directed at the graphical representation of the musical composition is received via an input device communicatively coupled to a digital data processing device.

74. The method of claim 73, wherein the input device is at least one of a mouse, a spaceball, a trackball, a stylus, a sensory glove, a voice-responsive device, a haptic interface device, a game controller, a remote control, and a transceiver.

75. The method of claim 65, wherein the interaction directed at the graphical representation of the musical composition corresponds to a haptic interaction associated with at least one of a force, a direction, a velocity, a position, an acceleration, and a moment.

76. The method of claim 65, wherein the interaction directed at the graphical representation of the musical composition corresponds to a tactile interaction associated with a texture.

77. The method of claim 65, wherein the modified audio attribute reverts to an unmodified state after a predetermined period of time.

78. The method of claim 65, wherein the modified audio attribute reverts to an unmodified state upon occurrence of an event.

79. The method of claim 78, wherein the event corresponds to at least one of a cessation of a haptic interaction, a cessation of a tactile interaction, a recordation of the modified audio attribute, a recordation of the musical composition, and a performance of the musical composition.

80. The method of claim 65, further comprising:

associating the at least one audio attribute with a haptic parameter, a value of the haptic parameter being manipulated in response to the interaction directed at the graphical representation of the musical composition, wherein the modified audio attribute is based at least partly on the value of the haptic parameter.

81. The method of claim 80, further comprising:

calculating the modified audio attribute based on a mathematical relationship with the haptic parameter.

82. The method of claim 65, further comprising:

performing the musical composition incorporating the modified audio attribute.

83. The method of claim 82, further comprising:

storing the musical composition with the modified audio attribute in a memory communicatively coupled to a digital data processing device.

84. The method of claim 83, wherein the stored musical composition is stored in a MIDI file format.

85. The method of claim 83, further comprising:

transmitting the stored musical composition to another digital data processing device via a network.

86. The method of claim 65, further comprising:

associating the graphical representation of the musical composition with a different musical composition; and

performing the different musical composition in response to another interaction directed at the graphical representation.

87. The method of claim 65, further comprising:

associating the graphical representation of the musical composition with at least one other musical composition; and

concurrently performing the musical composition and the at least one other musical composition in response to another interaction directed at the graphical representation.

88. The method of claim 65, further comprising:

displaying the graphical representation of the musical composition;

displaying another graphical representation of a different musical composition; and

performing the musical compositions based at least partly on another interaction directed at a space between the graphical representations.

89. The method of claim 65, further comprising:

performing the musical composition with the modified audio attribute, wherein the musical composition corresponds to an object within at least one of a game, a learning exercise, a simulation, a music production, and a multimedia production.

68. (canceled)

90-102. -102. (canceled)

103. A system for modifying audio attributes of a musical composition, the system comprising:

a plurality of audio attributes associated with a musical composition;

a plurality of graphical parameters associated with a graphical surface;

a mapping data structure associating at least some of the audio attributes with at least some of the graphical parameters; and

a calculation software process calculating modified values of the associated audio attributes in response to interactions directed at the graphical surface.

104. The system of claim 103, wherein the audio attributes include at least one of a tone, volume, pitch, attack rate, decay rate, sustain rate, vibrato, sostenuto, and base.

105. The system of claim 103, wherein the audio attributes are accessed from a MIDI file.

106. The system of claim 103, wherein the audio attributes are accessed from a non-MIDI file.

107. The system of claim 103, wherein the musical composition includes at least one of a voice, a monody, and a polyphony.

108. The system of claim 103, wherein the graphical surface represents a range of modification of the associated audio attributes.

109. The system of claim 103, wherein the graphical parameters correspond to at least one of a texture, a coordinate location, a color, a depth, and a height.

110. The system of claim 103, further comprising:

an input device directing the interactions at the graphical surface, wherein the input device is at least one of a mouse, a spaceball, a trackball, a stylus, a sensory glove, a voice-responsive device, a haptic interface device, a game controller, a remote control, and a transceiver.

111. The system of claim 103, wherein the interaction directed at the graphical surface corresponds to a haptic interaction associated with at least one of a force, a direction, a velocity, a position, an acceleration, and a moment.

112. The system of claim 103, wherein the interaction directed at the graphical surface corresponds to a tactile interaction associated with a texture.

113. The system of claim 103, wherein the mapping data structure relates haptic parameters with the associated audio attributes and graphical parameters, and wherein the calculation software process calculates the modified values of the associated audio attributes based at least partly on values of the haptic parameters.

114. The system of claim 103, further comprising:

an audio element forming an audible rendition of the musical composition, the musical composition incorporating the modified values of the associated audio attributes.

115. The system of claim 103, further comprising:

a rendering software process rendering the graphical surface on a display of a digital data processing device.

116. The system of claim 103, further comprising:

a MIDI file storing the modified values of the associated audio attributes.

117. The system of claim 103, wherein the modified values of the associated audio attributes are performed in at least one of a game, a learning exercise, a simulation, a music production, and a multimedia production.