Cybernetic 3D music visualizer

• US 20060181537A1
• Filed: 01/25/2006
• Published: 08/17/2006
• Est. Priority Date: 01/25/2005
• Status: Abandoned Application

• Current Assignee: Rik Henderson, Srini Vasan, Vladimir Bulatov
• Original Assignee: Rik Henderson, Srini Vasan, Vladimir Bulatov
• Inventors: Henderson, Rik, Vasan, Srini, Bulatov, Vladimir
• Application Number:
  US11/339,740
• Publication Number:
  US 20060181537A1
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data
• CPC Class Codes: G10H 2240/311 MIDI transmission G10H2240/056 takes precedence

Abstract

3D music visualization process employing a novel method of real-time reconfigurable control of 3D geometry and texture, employing blended control combinations of software oscillators, computer keyboard and mouse, audio spectrum, control recordings and MIDI protocol. The method includes a programmable visual attack, decay, sustain and release (V-ADSR) transfer function applicable to all degrees of freedom of 3D output parameters, enhancing even binary control inputs with continuous and aesthetic spatio-temporal symmetries of behavior. A “Scene Nodes Graph” for authoring content acts as a hierarchical, object-oriented graphical interpreter for defining 3D models and their textures, as well as flexibly defining how the control source blend(s) are connected or “Routed” to those objects. An “Auto-Builder” simplifies Scene construction by auto-inserting and auto-routing Scene Objects. The Scene Nodes Graph also includes means for real-time modification of the control scheme structure itself, and supports direct real-time keyboard/mouse adjustment to all parameters of all input control sources and all output objects. Dynamic control schemes are also supported such as control sources modifying the Routing and parameters of other control sources. Auto-scene-creator feature allows automatic scene creation by exploiting the maximum threshold of visualizer set of variables to create a nearly infinite set of scenes. A Realtime-Network-Updater feature allows multiple local and/or remote users to simultaneously co-create scenes in real-time and effect the changes in a networked community environment where in universal variables are interactively updated in real-time thus enabling scene co-creation in a global environment. In terms of the human subjective perception, the method creates, enhances and amplifies multiple forms of both passive and interactive synesthesia. The method utilizes transfer functions providing multiple forms of applied symmetry in the control feedback process yielding an increased level of perceived visual harmony and beauty. The method enables a substantially increased number of both passive and human-interactive interpenetrating control/feedback processes that may be simultaneously employed within the same audio-visual perceptual space, while maintaining distinct recognition of each, and reducing the threshold of human ergonomic effort required to distinguish them even when so coexistent. Taken together, these novel features of the invention can be employed (by means of considered Scene content construction) to realize an increased density of “orthogonal features” in cybernetic multimedia content. This furthermore increases the maximum number of human players who can simultaneously participate in shared interactive music visualization content while each still retaining relatively clear perception of their own control/feedback parameters.

Claims

1. A visualization method for real-time modulation of visual object parameters of an 3D computer graphics animation, the method comprising:

a. A real-time software runtime interpreter having one or more visualizer 3D ‘

scenes’

comprised of a matrix of input-output control transfer functions loaded into RAM prior to runtime from an external non-volatile data store;

b. Loading of a plurality of 3D resources from an external data store prior to runtime into RAM data utilized by the interpreter and applying and modulating such resources during runtime in the output 3D visual space;

c. Production and output of 3D animation modulations and effects that are precisely synchronized with simultaneously presented musical content;

d. Allowing of simultaneous real-time control inputs from a plurality of control sources;

e. Allowing of simultaneous modulation of a plurality of 3D objects and their parameters including 3D spatial geometry of models, 3D applied surface textures, 3D particles and video effects;

f. Production in real-time of visualizer outputs on a primary display device or window of either a 2D (CRT or other panel) or 3D (stereoscopic or volumetric) type;

g. Input of streaming digital video resource in real-time into the interpreter and applying and modulating such resources at runtime in the output 3D visual space;

2. The system of claim 1, wherein simultaneous to the 3D scene output display, an secondary display or window is provided for the software interpreter'"'"'s representation of a visualizer scene that is graphically presented to the user in terms of a hierarchical “

nodes graph”

, comprising;

a. A default new scene provides initial required objects;

b. Additional scene objects may be inserted by simple menu selections and keyboard quick-key commands;

c. Node graph objects may be reordered by drag-and-drop editing;

d. Once input control sources and output objects are inserted into the scene nodes graph, an auto-routing feature of the interpreter'"'"'s GUI assists in completing the transfer function definition by auto-inserting an appropriate “

route.”

This is especially productive and useful when applied to particle engine effects;

e. Hierarchical nodes in the nodes graph can be expanded or collapsed for editing convenience (where children beneath a given node may be hidden or revealed);

f. Additional detailed parameter settings for objects in the nodes graph window are accessed by double-clicking on the object name or icon in the nodes graph to reveal their detail windows;

g. For objects in a given scene, any and all object parameters in their corresponding detail windows can optionally be manipulated in real-time by mouse increment/decrement drags, and/or numeric ASCII keys, and the results in the 3D space are immediately and in real-time displayed in the primary scene display.

3. The system of claim 1, wherein the resources loaded into interpreter RAM includes a plurality of 3D Actors, 3D Models, 2D images and/or 2D movies (including AVI file type) and whereas all such resources are available for modulation(s) in the 3D visual output space and in real-time.

4. The system of claim 1, wherein the plurality of real-time control inputs includes any combination of previously user-created control recordings;

internal oscillators;

computer keyboard and mouse actions;

the audio spectrum; and

/or MIDI protocol messages from any MIDI device, software or instrument, and furthermore comprising;

a. Means whereby any simultaneous weighted combination of control inputs to comprise a ‘

control blend’

used to modulate 3D visual objects and their parameters;

b. Means whereby a plurality of such control blends to simultaneously modulate 3D visual objects and their parameters;

c. Means whereby a sufficiently broad scope and richness in output modulation parameters and scene setup topologies such that a given scene'"'"'s matrix of transfer functions may be designed with considerably distinct (perceptually orthogonal) feature spaces thereby enhancing simultaneous multi-player distinction of feedback, as well as enhancing perception of simultaneous feature modulations on a given object (such as shape and texture and color modulation on a single object, such modulations derived from simultaneous control sources.)

d. Means by which imultaneous players including one (local) ASCII keyboard and mouse player (if any) together with an unlimited number of MIDI device players.

e. Means by which layers can be local or remote via MIDI over TCP/IP.

5. The system of claim 1, wherein a plurality of simultaneous control blends may be allocated by considered scene design and of their transfer functions to reside in adjacent control ranges both within each control type, and in correlation between different control types, such a system comprising means whereby:

a. Various audio spectrum frequency “

bins”

whether adjacent in frequency or not, may each be allocated to different transfer functions of any provided modulation means and may be applied to any output 3D scene object(s) or parameter(s);

b. Various computer keyboard keys, may each be allocated to different transfer functions of any provided visual modulation means and may be applied to any output 3D scene object(s) or parameter(s);

c. Various MIDI instrument keys and controls, may each be allocated to different transfer functions of any provided visual modulation means and may be applied to any output 3D scene object(s) or parameter(s);

d. A plurality of such adjacent control ranges may be correlated between different control types, such that for example a first control input range for each of keyboard, audio spectrum and MIDI device have the identical or similar modulation effect in one aspect (object(s) and/or parameter(s)) of the 3D output visual space;

a second control input range for each of keyboard, audio spectrum and MIDI device have identical or similar modulation effect in a second and distinct aspect (object(s) and/or parameter(s)) of the 3D output visual space, and so forth similarly for any number of such adjacent control ranges and for any number of output modulation effect(s).

6. The system of claim 5, wherein the control input-output topology of transfer functions (routing) exhibits substantially flexible programmability in scene design, such a system comprising means whereby:

a. Routing may exhibit a one-to-many topology of one control blend to (n) parameters modulation;

b. Routing may exhibit a many-to-one topology of (n) control blends to one output parameter modulation;

c. Routing may exhibit a many-to-many topology of (n) control blends to (n) output parameters modulation;

d. Routing may exhibit a one-to-one topology of one control blend to one output parameter modulation;

e. In a given scene a plurality of such transfer function routings may co-reside in any combination of such routing topologies, for any number of routes, for any number of control blends, and for any number of output parameters (numeric limits being imposed only by the capacity of RAM memory of the interpreter).

7. The system of claim 1, wherein the software algorithmic and data structures approach to implementation of the 3D visualizer interpreter results is highly efficient rendering resulting in high frame rates for a true real-time 3D visualization.

8. The system of claim 1, wherein the number of available types of real-time modulation objects includes at least fourteen different object families including for:

background, camera, 3D transform, object, switch, touch-sensor, 3D model, 2D texture (applicable to 3D surfaces), 3D animator, 3D light, route, interpolator, slider, and keyboard sensor; and

furthermore comprising means whereby;

a. Families of object types each may include from 1 to 19 or more individual objects (for example in the 3D Model case such as plane, sphere, torus, shell, box, cone, hedron, isohedron, etc.);

b. All individual object types for all object families when taken together comprise on the order of 74 or more unique and fully real-time modulation objects;

c. Individual objects include on the order of from 2 to 45 different modulation parameters and typically average a dozen or more each (for example in the case of camera 14 parameters including X, Y and Z position;

X, Y and Z orientation;

angle;

field of view;

speed;

spin speed;

tilt;

height;

drop opacity; and

navigation);

d. All individual parameters for all individual object types for all object families when taken together comprise on the order of 784 unique, real-time modulation parameters;

each and any of these may be utilized by the interpreter in a single and/or a plurality of instances of that parameter in any given scene;

9. The system of claim 8, wherein the interpreter'"'"'s secondary GUI windows, specifically within any object detail (parameters) window, provides an automated means for the user to quickly auto-increment through a large number of parameter combinations for a given object, in order to set defaults for the object in that particular scene, as well as to easily find and set aesthetic limits for that object'"'"'s parameter modulations. This Auto-Scene-Creator feature allows automatic scene creation by exploiting the maximum threshold of visualizer variables to create nearly an infinite set of visualizer scenes.

10. The system of claim 1, wherein any and all routed transfer function(s) between any control input source(s) and any output modulated parameter(s), may exhibit a response curve with four distinct time segments (vs. amplitude) namely arrack, decay, sustain and release, such a system comprising means whereby;

a. When applied, such Visual-ADSR or V-ADSR provides an aesthetic character to any and all of the interpreter'"'"'s visual modulations, being similar in result (but in visual terms) to the well-known aesthetic character of such response curves when applied in the audio domain of a musical note or event;

b. Visual-ADSR brings a smooth, continuous character to animation effects when applied in the visual domain, even in the presence of such as binary MIDI or ASCII keyboard triggers as the control source, i.e. input triggers having no variable velocity;

c. V-ADSR represents an application of symmetry to an input trigger;

d. When velocity is present in the input control source, that is taken into account in the V-ADSR response;

e. V-ADSR may optionally be applied to transfer functions (animators) for ASCII Keyboard, MIDI, and/or Audio. It operates identically as to the nature of the response curves applied, even when used for effects in totally different feature spaces (i.e. texture shifting as contrasted with geometric shape morphing.)

f. V-ADSR settings may be individually applied and independently adjusted for each and every transfer function (animator) it is applied to;

(i.e. it is not a global setting.)

11. The system of claim 5, wherein the setup of MIDI transfer functions (MIDI animators) may be setup, the various different supported MIDI message types may be setup to exhibit certain general types of spatio-temporal response “

styles”

of behavior; and

comprising means to implement such “

styles”

including;

a. Disable;

no animation effect active (available with all supported message types);

b. Smooth;

smoothly ramps from the minimum value to the maximum value, then smoothly ramps back to minimum value;

(available with Note On/Off, Polyphonic Aftertouch, Control Change, and Pitch Bend);

c. Jump;

suddenly jumps from the minimum to the maximum value, then suddenly jumps back to minimum value;

(available only with Note On/Off;

);

d. Smooth Up Jump Back;

smoothly ramps from the minimum value to the maximum, then jumps back to the minimum value;

(available with Note On/Off, Control Change and Pitch Bend).

e. Multi-Jump;

Smoothly ramps from minimum to maximum value, jumps back to the minimum value, and repeats the cycle;

(available only with Polyphonic Aftertouch).

12. The system of claim 1, wherein also a Real-time-Network-Updater functionality allows multiple users to simultaneously co-create and run visualizer scenes in real-time and effect the changes in a networked community environment, where in visualizer variables are interactively updated in real-time thus enabling scene co-creation and co-play in a global environment.