Real-time multimedia visual programming system

• US 6,331,864 B1
• Filed: 09/23/1997
• Issued: 12/18/2001
• Est. Priority Date: 09/23/1997
• Status: Expired due to Term

• Current Assignee: Wiremed Tech LLC (IP Edge LLC)
• Original Assignee: Onadime Incorporated
• Inventors: Coco, Geoffrey P., Reay, C. Phillip
• Primary Examiner(s): dela Torre, Crescelle N.
• Application Number:
  US08/935,618
• Time in Patent Office: 1,547 Days
• Field of Search: 345/339, 345/348, 345/349, 345/326, 345/333, 345/334, 345/335, 345/967, 364/188, 364/191, 364/578, 395/500, 395/701, 395/710, 395/500.34, 395/500.43, 700/83, 700/86, 702/123, 703/13, 703/22, 717/1, 717/10
• US Class Current: 715/763
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: Y10S 715/967 Visual or iconic programming

Abstract

A user friendly visual programming system with a visual programming interface that allows a user to visually create multimedia programs in real-time. In this system, input and output functions are represented as graphical input and receive interface leads, respectively. A user with no programming skill easily links desired transmit interface leads to desired receive interface leads. In real-time, the output devices perform the functions associated with receive interface leads and any changes to the receive interface leads that occur as a result of any links to transmit interface leads. The user may create complex programs by encapsulating input and receive interface leads in an endless number encapsulation layers.

Claims

1. A method for controlling output sources using input sources, said method comprising:

(a) receiving a first raw data value based on an input function performed on an input source;

(b) calculating a first normalized value for the received first raw data value;

(c) changing a second normalized value based on the calculated first normalized value;

(d) calculating a second raw data value for an output source based on the changed second normalized value; and

(e) controlling the output source with an output function provided with the second raw data value, and wherein (a), (b), (c), (d) and (e) are performed in real-time.

2. A computer-readable medium with a computer program stored thereon for controlling data output sources using data input sources, said computer program comprising:

a receiving component for receiving a first raw data vlaue based on an input function performed on an input source;

a first calculating component for calculating a first normalized value for the receivced first raw data value;

a changing component for changing a second normalized value based on the calculated first normalized value;

a second calculating component for calculating a second raw data value for an output source based on the changed second normalized value; and

a controlling component for controlling the output source with an output function provided with the second raw data value, wherein the receiving component, first calculating component, changing component, second calculating component and controlling component operate to control the output source in real-time.

3. A method performed on a computer for interoperating data inputs received from an input source and data outputs sent to an output source, said method comprising:

defining input functions based on the data inputs;

defining output functions based on the data outputs;

linking at least one input function to at least one output function to indicate that the at least one output function will change as the linked at least one input function changes, and without determining if the type of data input upon which the at least one input function is based is compatible with the type of data output upon which the at least one output is based; and

dynamically controlling the at least one output function by changing the at least one input function, wherein changing the linked at least one input function and controlling the at least one output function occur simultaneously.

4. The method of claim 3, wherein defining the input functions and the output functions comprises:

defining a normalized data range for data values assigned to the data inputs and the data outputs;

defining an input function raw data range for data values generated by an input function of a data input;

converting the data value generated by the input function into a data value within the normalized data range based on the input function raw data range;

defining an output function raw data range for data values received by an output function of a data output; and

converting the data value assigned to the data output into a data value within the output function raw data range based on the normalized data range.

5. The method of claim 4, further comprising:

defining a relationship between the defined normalized data values of the data input and the data output; and

setting the normalized data value of the data output based on the normalized data value of the data input and the defined relationship between the defined normalized data values of the data input and the data output.

6. The method of claim 5, wherein controlling the at least one output function comprises:

setting the normalized data value for the data input based on the normalized data value for the data output, the defined relationship between the defined normalized data values of the data input and the data output, and the linked data input; and

determining the data value for the data output based on the set normalized data value for the data input.

7. The method of claim 6, further comprising storing the data output value.

8. The method of claim 5, wherein the data input comprises at least one of an aural data input or a visual data input.

9. The method of claim 8, wherein the data output comprises at least one of an aural data output or a visual data output.

10. The method of claim 5, wherein the data input is generated by an external device.

11. The method of claim 5, wherein the data input is generated internally.

12. The method of claim 5, wherein the data input comprises at least one signal data input.

13. The method of claim 3, wherein the data inputs are dynamic inputs, and whereby the data outputs are dynamic data outputs.

14. The method of claim 3, wherein said at least one input function is linked to a plurality of output functions.

15. A computer program stored on a computer-readable medium for programming the interoperability of input and output devices, said computer program comprising:

a first defining component for defining input functions of the input devices as transmit controls;

a second defining component for defining output functions of the output devices as receive controls; and

a first linking component for linking at least one transmit control to at least one receive control without determining if the data processed by the input devices is compatible with the data processed by the output devices, such that the at least one output function defined by the at least one receive control is dynamically controlled in real-time by manipulating the input function defined by the at least one transmit control linked to the at least one receive control.

16. The computer program of claim 15, further comprising:

a normalized data range defining component for defining a normalized data range for data values assigned to the transmit and receive controls;

wherein the first defining component comprises;

an input function raw data range defining component for defining a input function raw data range for data values generated by an input function of an input device;

a first data value converting component for converting the data value generated by the input function into a data value within the normalized data range based on the input function raw data range and the normalized data range;

wherein the second defining component comprises;

an output function raw data range defining component for defining an output function raw data range for data values received by an output function of an output device;

a second data converting component for converting the data value assigned to the receive control into a data value defined in the output function raw data range based on the normalized data range and the output function raw data range.

17. The computer program of claim 16, wherein said first linking component comprises:

a first linking sub-component for linking said at least one transmit control to a transform receive control; and

a second linking sub-component for linking a transform transmit control to said at least one receive control, wherein said transform transmit control has a defined relationship to said transform receive control, and wherein said at least one output function of the output device defined by the at least one receive control is dynamicaly controlled by manipulating the input function defined by said at least one transmit control linked to said at least one receive control and by said defined relationship between said transform transmit control and said transform receive control.

18. The computer program of claim 17, further comprising:

an input control form component that includes at least one transmit control associated with an input device; and

an output control form component that includes at least one receive control associatied with an output device.

19. The computer program of claim 18, further comprising:

a linking control form component that includes at least one pair of a transform transmit control and an associated transform receive control.

20. The computer program of claim 19, further comprising:

an encapsulated control form component that includes one or more of at least one input control form, output control form and transform control form.

21. The computer program of claim 15, wherein said first linking component links said at least one transmit control to a plurality of receive controls.

22. A computer-based system for programming the interoperability of input and output devices, said computer-based system comprising:

at least one input device with one or more input functions;

at least one output device with one or more output functions;

a display device;

a processor with a computer program stored therein, said computer program comprising;

a first defining component for defining the one or more input functions of the at least one input device as graphical transmit controls displayed on the display device;

a second defining component for defining the one or more output functions of the at least one output device as graphical receive controls displayed on the display device;

a first linking component for linking at least one graphical transmit control which defines at least one input function to at least one graphical receive control which defines at least one output function, wherein the at least one graphical transmit control is linked to the at least one graphical receive control without determining if the data processed by the at least one input device is compatible with the data processed by the at least one output device; and

a first controlling component for dynamically controlling the one or more output functions defined by the at least one graphical receive control in real-time by manipulating the one or more linked input functions that are defined by the graphical transmit control linked to the at least one graphical receive control.

23. The system of claim 22, wherein said computer program further comprises:

a normalized data range defining component for defining a normalized data range for data values assigned to the graphical transmit and receive controls;

wherein the first defining component comprises;

an input function raw data range defining component for defining a input function raw data range for data values generated by an input function of an input device;

a first data value converting component for converting the data value generated by the input function into a data value within the normalized data range based on the input function raw data range and the normalized data range;

wherein the second defining component comprises;

an output function raw data range defining component for defining an output function raw data range for data values received by an output function of an output device;

a second data converting component for converting the data value assigned to the graphical receive control into a data value defined in the output function raw data range based on the normalized data range and the output function raw data range.

24. The system of claim 23, wherein the transmit controls and the receive controls are graphical user slide bars.

25. The system of claim 23, wherein said first linking component within the computer program comprises:

a first linking sub-component for linking said at least one graphical transmit control to a graphical transform receive control; and

a second linking sub-component for linking a graphical transform transmit control to said at least one receive control, wherein said graphical transform transmit control has a defined relationship to said graphical transform receive control; and

wherein said first controlling component within the computer program comprises;

a first controlling sub-component for controlling said at least one output function of the output device defined by the at least one graphical receive control by manipulating the input device with the input function defined by said at least one graphical transmit control linked to said at least one receive control and by said defined relationship between said graphical transform transmit control and said graphical transform receive control.

26. The system of claim 25, wherein said computer program further comprises:

an input control form component that includes at least one graphical transmit control associated with an input device; and

an output control form component that includes at least one graphical receive control associated with an output device.

27. The system of claim 26, wherein said computer program further comprises:

a linking control form component that includes at least one pair of a graphical transform transmit control and an associated graphical transform receive control.

28. The system of claim 27, wherein said computer program further comprises:

an encapsulated control form component that includes one or more of at least one input control form, output control form and transform control form.

29. The computer-based system of claim 22, wherein said first linking component links said at least one graphical transmit control to a plurality of graphical receive controls.

30. A method for generating a multimedia computer program using a visual programming system, said method comprising:

providing a plurality of input functions that produce data;

providing a plurality of output functions that react to data;

providing a mechanism for a user to define a relationship between any of the input functions and any of the output functions without determining if the type of data produced by the input function is compatible with the type of data to which the output functions reacts, wherein the mechanism causes the output function to react in real-time to the data produced by the input function with which the output function has a relationship; and

generating a program in response to a user defining at least one relationship between at least one input function and at least one output function.

31. The method of claim 30, further comprising providing a plurality of linking functions, wherein a linking function manipulates data.

32. The method of claim 31, wherein at least one relationship between at least one input function and at least one output function includes at least one linking function between the at least one input function and the at least one output function.

33. the method of claim 32, wherein at least one relationship between any of the input functions and any of the output functions is a relationship between at least one input function and a plurality of output functions.

34. The method of claim 30, wherein providing a plurality of input functions that produce data comprises:

for each input function;

determining a raw data range for the input function;

creating a translation calculation for translating a raw data value in the raw data range into a normalized data value;

creating a transmit interface lead for accessing the translation calculation; and

providing a user access to the transmit interface lead.

35. The method of claim 34, further comprising, creating a form comprising at least one transmit interface lead, in response to a user request to create a form.

36. The method of claim 30, wherein providing a plurality of output functions that react to data comprises:

for each output function;

determining a raw data range for the output function;

creating a translation calculation for translating a raw data value in the raw data range into a normalized data value;

creating a receive interface lead for accessing the translation calculation; and

providing a user access to the receive interface lead.

37. The method of claim 36, further comprising creating a form comprising at least one receive interface lead, in response to a user request to create a form.

38. A system for providing a visual programming interface for creating real-time multimedia experiences comprising:

a storage medium containing a graphical user interface component for receiving a plurality of user inputs, the user inputs identifying at least one data input device and at least one data output device; and

a processor coupled to the storage medium for converting raw data received by the graphical user interface component from the at least one data input device into raw data to be output to the at least one data output device in real-time, wherein the processor converts the raw data received by the graphical user interface by;

determining a received raw data range for the raw data received by the graphical user interface component;

converting the raw data received by the graphical user interface component into normalized data within a normalized data range that is based upon the received raw data range;

determining an output raw data range for the raw data to be output by the at least one data output device; and

converting the normalized data into raw data within the output raw data range, wherein the raw data is to be output by the at least one data output device.

39. The system of claim 38, wherein each data input device comprises at least one input function.

40. The system of claim 39, wherein the input function comprises a plurality of input sub-functions.

41. The system of claim 39, wherein each input function is graphically represented in the graphical user interface component by a transmit interface lead.

42. The system of claim 41, wherein each data output device comprises at least one output function.

43. The system of claim 42, wherein the output function comprises a plurality of output sub-functions.

44. The system of claim 42, wherein each output function is graphically represented in the graphical user interface component by a receive interface lead.

45. The system of claim 44, wherein the graphical user interface component further comprises a mechanism for linking the transmit interface lead to the receive interface lead.

46. The system of claim 45, wherein the graphical user interface component further comprises a display component for displaying a graphical indication of a link created by the linking mechanism.

47. The system of claim 46, wherein the graphical user interface component further comprises an encapsulation component, wherein the transmit interface lead is encapsulated in an encapsulation layer in response to a user request to encapsulate the transmit interface lead.

48. The system of claim 47, wherein the graphical user interface component further comprises an encapsulation component, wherein the receive interface lead is encapsulated in an encapsulation layer in response to a user request to encapsulate the receive interface lead.

49. The system of claim 48, wherein the number of encapsulation layers is limited only by the storage medium and the processor speed.

50. The system of claim 39, wherein the at least one input function is predetermined.

51. The system of claim 50, wherein the at least one input function is capable of being altered by a user via the graphical user interface component.

52. The system of claim 39, wherein the at least one output function is predetermined.

53. The system of claim 52, wherein the at least one output function is capable of being altered by a user via the graphical user interface component.

54. A multimedia program embodied on a computer-readable medium, wherein the multimedia program is created by a method comprising:

providing a plurality of input functions that produce data;

providing a plurality of output functions that react to data;

providing a mechanism for a user to define a relationship between any of the input functions and any of the output functions, without determining if the type of data produced by the input functions is compatible with the type of data to which the output functions react, wherein the mechanism allows the output function to be dynamically controlled by the input function in real-time; and

generating the multimedia program in response to a user defining at least one relationship between at least one input function and at least one output function.

55. The multimedia program of claim 54, wherein the at least one output function generates visual data.

56. The multimedia program of claim 54, wherein the at least one output function generates audio data.