Extensible simulation system and graphical programming method

• US 5,566,295 A
• Filed: 01/25/1994
• Issued: 10/15/1996
• Est. Priority Date: 01/25/1994
• Status: Expired due to Term

• Current Assignee: Apple Computer Incorporated (Apple Inc.)
• Original Assignee: Apple Computer Incorporated (Apple Inc.)
• Inventors: Cypher, Allen, Smith, David C., Spohrer, James C.
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): dela Torre, Crescelle N.
• Application Number:
  US08/187,682
• Time in Patent Office: 994 Days
• Field of Search: 395/155, 395/161, 395/159, 395/152, 395/157, 395/160, 364/578
• US Class Current: 715/763
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06F 8/34 Graphical or visual programming

Abstract

An extensible simulation system and graphical programming method enable a simulation user to program the behaviors of objects in a simulation while requiring no knowledge of computer programming concepts or languages. The simulation user defines each object'"'"'s behaviors by creating Graphical Rewrite Rules through programming by demonstration. The user can selectively abstract a given Graphical Rewrite Rule (GRR) such that it applies to states within the simulation according to hierarchical object types and object property conditions. The extensible simulation system comprises a Central Processing Unit (CPU), an input device, an output device, an external storage device, predetermined amounts of Random Access Memory (RAM) and Read-Only Memory (ROM), and an extensible simulator. The extensible simulator comprises an object source, a simulation viewer, a GRR editor, an action recorder, an abstractor, a drawing editor, an object property editor, an object rule viewer, a subroutine editor, and a simulation execution controller. Each element of the system has an input and an output coupled to a common system bus. Preferably, each element of the extensible simulator is a computer program step sequence stored in RAM. The graphical programming method comprises the steps of: selecting an object for which a GRR is to be defined; establishing a simulation context corresponding to the GRR; defining for the GRR a set of actions to be performed by or upon one or more objects within the simulation context, each action defined through programming by demonstration; automatically recording a computer program step sequence for each action defined; and allowing the GRR to be abstracted.

Claims

1. A computerized extensible simulation system providing an easy to use and interactive environment for creating and executing a computer simulation, said system comprising:

an output device for displaying data and graphical images, the output device having an input;

an input device for sending data and commands into the system, the input device having an output;

a memory having inputs and outputs for storing data and commands, the memory having an object source comprised of program instructions for generating a plurality of graphic objects, an abstractor comprised of program instructions for generating a graphical rewrite rule in response to a graphical manipulation of a graphic object upon the output device via the input device wherein the graphical rewrite rule comprises program instructions defining a manner in which the graphic object is to be selectively manipulated upon the output device, a simulation viewer comprised of program instructions for controlling the display of the graphic objects and their actions on the output device, a graphical rewrite rule editor comprised of program instructions for allowing the graphical definition of properties and graphical rewrite rules for the graphic objects, a simulation controller comprised of program instructions for executing the graphical rewrite rule to produce the graphic object'"'"'s actions; and

a processing unit having an input and output for processing data and generating images on the output device under the control of the memory, the output of the processing unit coupled to the input of the output device, the input of the processing unit coupled to the output of the input device, and the input and output of the processing unit coupled to the memory.

2. The computerized extensible simulation system of claim 1, wherein at least one of the plurality of graphic objects has an associated graphical representation, a type, a unique name, a set of properties, and a set of graphical rewrite rules.

3. The computerized extensible simulation system of claim 1, wherein the simulation viewer maintains a simulation as a set of simulation grid elements having predetermined graphical characteristics, and the simulation viewer controls the processing unit to display scenes of the simulation as one or more grid elements in a simulation viewer window along with graphical representations of graphic objects in the set of simulation grid elements,

wherein a simulation grid element comprises a displayable portion of the simulation.

4. The computerized extensible simulation system of claim 3, wherein the simulation viewer also controls the outputting of a graphic object property display, the graphic object property display showing on the output device current properties of a graphic object and positioned at a predefined position with respect to the graphic object.

5. The computerized extensible simulation system of claim 3, wherein the graphical rewrite rule editor controls the processing unit to display an editor window, a graphic object, and a simulation context on the output device, and includes an action recorder for receiving actions input by a user of the graphic object in the simulation context displayed in the editor window and for translating the actions into a graphical rewrite rule executable by the simulation controller,

wherein the simulation context is a subset of simulation grid elements containing a selected graphic object.

6. The computerized extensible simulation system of claim 1, wherein the memory further comprises program instructions for generating:

a graphic object source for storing the plurality of graphic objects, for maintaining at least one type for each graphic object, for maintaining a hierarchical relationship between graphic object types, and for maintaining a set of properties for at least one graphic object; and

a graphic object abstractor for modifying the applicability of a graphical rewrite rule having a before portion including associated graphic objects, the graphic object abstractor changing the type of at least one of the associated graphic objects and adding property match conditions for use by the simulation controller during execution of the rule.

7. The computerized extensible simulation system of claim 1, wherein the simulation controller sequentially tests the graphical rewrite rules in at least one graphic object'"'"'s graphical rewrite rule set for a match with a simulation context and then for the first match detected executes a Graphical Re-write Rule program corresponding to the graphical rewrite rule which matched the simulation context to produce the actions input by the user during creation of the graphical rewrite rule,

wherein the simulation context is a subset of simulation grid elements containing a selected graphic object, and

wherein a simulation grid element comprises a displayable portion of the simulation.

8. The computerized extensible simulation system of claim 2, wherein the memory further comprises a drawing editor comprised of program instructions for controlling the display of a drawing editor window, the drawing editor providing for modification of the graphical representation of the graphic object in the simulation and for the creation of graphical representations of the graphic object transitioning between a before and an after portion of a graphical rewrite rule.

9. The computerized extensible simulation system of claim 2, wherein the memory further comprises a graphic object property editor comprised of program instructions for controlling the display of a graphic object property editor window on the output device, and for accepting inputs via the processing unit for adding or changing the properties associated with a graphic object.

10. The computerized extensible simulation system of claim 1, wherein the memory further comprises a graphic object rule viewer comprised of program instructions for controlling the display of a graphic object rule viewer window on the output device, the graphic object rule viewer displaying a before portion and an after portion for at least one graphical rewrite rule associated with a selected graphic object.

11. A computer implemented method for executing a simulation having a plurality of graphic objects in an extensible simulation system, the method comprising the steps of:

determining a current simulation context comprising a graphical portion of a display generated upon a display device;

selecting a graphic object within the current simulation context;

determining whether there is a graphical rewrite rule having a before portion and an after portion associated with the selected graphic object, wherein the graphical rewrite rule defines a manner in which a first simulation context, matching the before portion, is to be graphically replaced with a second simulation context, matching the after portion;

selecting a graphical rewrite rule if there is a graphical rewrite rule associated with the selected object;

determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object; and

executing a program corresponding to the selected graphical rewrite rule if the before portion of the selected graphical rewrite rule matches the current simulation context.

12. The computer implemented method of claim 11, further comprising the step of repeating the steps of selecting a graphical rewrite rule, determining whether the before portion of the selected graphical rewrite rule matches, and executing a program, for each graphical rewrite rule associated with the selected graphic object.

13. The computer implemented method of claim 11, further comprising the step of repeating the steps of selecting a graphic object, determining whether there is a graphical rewrite rule, selecting a graphical rewrite rule, determining whether the before portion of the selected graphical rewrite rule matches, and executing a program, for each graphic object in the simulation having an associated graphical rewrite rule.

14. The computer implemented method of claim 11, further comprising the steps of:

maintaining a simulation timer; and

incrementing the simulation timer before the first selecting step and the second determining step are performed for the graphic objects in the simulation.

15. The computer implemented method of claim 11, wherein the step of determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object is performed by comparing graphic object types and graphic object properties of the before portion of the selected graphical rewrite rule to graphic object types and graphic object properties in the current simulation context.

16. A computer implemented method for executing a simulation having a plurality of graphic objects in an extensible simulation system, the method comprising the steps of:

determining a current simulation context comprising a graphical portion of a display generated upon a display device;

selecting a graphic object within the current simulation context;

determining whether there is a graphical rewrite rule having a before portion and an after portion associated with the selected graphic object, wherein the graphical rewrite rule defines a manner in which a first simulation context, matching the before portion, is to be graphically replaced with a second simulation context, matching the after portion;

selecting a graphical rewrite rule if there is a graphical rewrite rule associated with the selected graphic object;

determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object; and

executing a program corresponding to the selected graphical rewrite rule if the before portion of the selected graphical rewrite rule matches the current simulation context,

wherein the step of determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object further comprises the steps of;

comparing the relative positions of graphic objects in the before portion with the relative positions of graphic objects in the current simulation context;

comparing the number, relative position and graphic object occupancy of simulation grid elements in the before portion to the number, relative position and graphic object occupancy of simulation grid elements in the current simulation context wherein a simulation grid element comprises a displayable portion of the simulation;

comparing graphic object types according to a set of hierarchical classifications of the before portion to graphic object types of the current simulation context; and

determining whether the property conditions defined for the graphic objects in the before portion are satisfied.

17. A computer implemented method for creating a graphical rewrite rule for a graphic object in an extensible simulation system, the method for creating a graphical rewrite rule comprising the steps of:

selecting the graphic object for which an action is to be defined in response to a selection from an input device;

determining a simulation context for the selected graphic object, wherein the simulation context is a subset of simulation grid elements containing the selected graphic object and wherein a simulation grid element comprises a displayable portion of the simulation;

displaying the simulation context in a graphical rewrite rule editor window; and

generating the graphical rewrite rule in response to a graphical manipulation of the graphic object via the input device,

wherein the graphical rewrite rule defines a manner in which a first simulation context, matching a before portion, is to be graphically replaced with a second simulation context, matching an after portion.

18. The method of claim 17, further comprising the steps of:

determining whether a user has updated the simulation context;

determining the updated simulation context for the selected graphic object; and

displaying the updated simulation context in the before portion and the after portion of the graphical rewrite rule editor window.

19. The method of claim 17, wherein the step of creating a new program, further comprises the steps of:

selecting a graphic object within a defined simulation context in response to a selection from the input device;

defining an action corresponding to the selected graphic object in response to a selection from the input device;

recording the defined action and translating the defined action into a program step sequence;

creating a symbolic program step corresponding to the defined action; and

storing the before portion and the after portion of the graphical rewrite rule corresponding to the program step sequences recorded.

20. The method of claim 19, wherein the step of defining an action corresponding to the selected graphic object is accomplished by selecting one from a group consisting of:

moving the selected graphic object from one simulation grid element to another within the simulation context in response to a selection from the input device;

defining a set of operations to be performed upon a graphic object property in response to a selection from the input device; and

creating an animated transition that alters a portion of a graphic object'"'"'s graphical representation within the simulation context in response to a selection from the input device.

21. The method of claim 17, further comprising the steps of:

determining whether the graphical rewrite rule is to be abstracted, wherein abstraction comprises the steps of analyzing the action, generalizing the action, and creating a corresponding generalized computer program step sequence; and

performing abstraction on the graphical rewrite rule if it is determined that the graphical rewrite rule is to be abstracted.

22. A computer implemented method for creating a graphical rewrite rule for a graphic object in an extensible simulation system, the method for creating a graphical rewrite rule comprising the steps of:

selecting the graphic object for which an action is to be defined;

determining a simulation context for the selected graphic object, wherein the simulation context is a subset of simulation grid elements containing the selected graphic object and wherein a simulation grid element comprises a displayable portion of the simulation;

displaying the simulation context in a graphical rewrite rule editor window;

generating the graphical rewrite rule in response to a graphical manipulation of the graphic object via an input device,

wherein the graphical rewrite rule defines a manner in which a first simulation context, matching a before portion, is to be graphically replaced with a second simulation context, matching an after portion;

determining whether the graphical rewrite rule is to be abstracted, wherein abstraction comprises the steps of analyzing the action, generalizing the action, and creating a corresponding generalized computer program step sequence; and

performing abstraction on the graphical rewrite rule if it is determined that the graphical rewrite rule is to be abstracted,

wherein the step of performing abstraction further comprises the steps of;

determining whether picture or property abstraction is to be performed;

if picture abstraction is to be performed, then

determining the graphic object the user has selected;

determining a hierarchical classification of graphic object types for the selected graphic object;

displaying a hierarchical classification of graphic object types for the selected graphic object in a hierarchy menu; and

determining the classification for the graphic object type selected and storing the selected classification as part of the graphical rewrite rule; and

if property abstraction is to be performed, then

displaying a property abstraction window;

identifying a graphic object that was selected;

displaying the properties for the selected graphic object;

determining a property, a condition and an expression input; and

storing a property abstraction definition including the property, condition and expression input as part of the graphical rewrite rule.

23. A computer implemented method for editing a graphical rewrite rule for a graphic object having a type and a property in an extensible simulation system, the method for editing a graphical rewrite rule comprising the steps of:

determining whether a graphical rewrite rule has been selected for editing, wherein the graphical rewrite rule, having a before portion and an after portion, defines a manner for graphically replacing, on an output device, first simulation context, matching the before portion, with a second simulation context, matching the after portion, wherein the simulation context is a subset of simulation grid elements containing a selected graphic object, and wherein a simulation grid element comprises a displayable portion of the simulation;

identifying the graphical rewrite rule that has been selected for editing;

displaying the graphical rewrite rule showing the before portion and the after portion of the graphical rewrite rule in a graphical rewrite rule editor window;

determining whether the graphical rewrite rule is to be abstracted, wherein abstraction comprises the steps of analyzing the graphical rewrite rule, generalizing the graphical rewrite rule, and creating a corresponding generalized computer program step sequence; and

performing abstraction on the graphical rewrite rule if it is determined the graphical rewrite rule is to be abstracted.

24. A computer implemented method for editing a graphical rewrite rule for a graphic object having a type and a property in an extensible simulation system, the method for editing a graphical rewrite rule comprising the steps of:

determining whether a graphical rewrite rule has been selected for editing, wherein the graphical rewrite rule, having a before portion and an after portion, defines a manner for graphically replacing, on an output device, a first simulation context, matching the before portion, with a second simulation context, matching the after portion, wherein the simulation context is a subset of simulation grid elements containing a selected graphic object, and wherein a simulation grid element comprises a displayable portion of the simulation;

identifying the graphical rewrite rule that has been selected for editing;

displaying the graphical rewrite rule showing the before portion and the after portion of the graphical rewrite rule in a graphical rewrite rule editor window;

determining whether the graphical rewrite rule is to be abstracted, wherein abstraction comprises the steps of analyzing the graphical rewrite rule, generalizing the graphical rewrite rule, and creating a corresponding generalized computer program step sequence; and

performing abstraction on the graphical rewrite rule if it is determined the graphical rewrite rule is to be abstracted,

wherein the step of performing abstraction further comprises the steps of;

determining the kind of abstraction to be performed;

if picture abstraction has been selected, then

determining the graphic object a user has selected;

determining a hierarchical classification of graphic object types for the selected graphic object;

displaying a hierarchical classification of graphic object types for the selected graphic object in a hierarchy menu; and

determining the classification for the graphic object type selected and storing the classification as part of the graphical rewrite rule; and

if property abstraction has been selected, then

displaying a property abstraction window;

identifying a graphic object that was selected;

displaying the properties for the selected graphic object;

determining a property, a condition and an expression input; and

storing a property abstraction definition including the property, condition and expression input as part of the graphical rewrite rule.

25. A computerized extensible simulation system providing an easy to use and interactive environment for creating and executing a computer simulation, said system comprising:

a graphic object source means for storing a plurality of simulation graphic objects, each simulation graphic object having an associated graphical representation, a type, a unique name, a set of properties and a set of graphical rewrite rules, the graphic object source means controlling the display of a graphic object source menu for creating graphic objects and including the graphic objects in the simulation;

a simulation viewing means for viewing the simulation, the simulation viewing means controlling a display of a simulation viewer window having simulation grid elements and graphic objects displayed within the simulation viewer window;

a graphical rewrite rule editing means for creating and modifying a graphical rewrite rule and properties for graphic objects in response to a graphical manipulation of a graphic object upon the simulation viewer window via an input device wherein the graphical rewrite rule comprises program instructions defining a manner in which the graphic object is to be selectively manipulated upon the simulation viewer window;

wherein the graphical rewrite rule editing means controls the display of a graphical rewrite rule editor window having a before portion window and an after portion window, and the graphical rewrite rule editing means modifying the after portion window for actions input by a user;

an action recording means for generating a computer program step sequence for each action the user demonstrates for a graphical rewrite rule using the graphical rewrite rule editing means; and

a simulation execution means for maintaining and modifying the simulation context by testing the graphical rewrite rules of each graphic object in the simulation for a match with the simulation context and performing the computer program step sequence associated with a graphical rewrite rule if the simulation context matches the before portion of the graphical rewrite rule.

26. The extensible simulation system of claim 25, wherein the simulation viewing means also generates a time-out region and a remove icon as part of the simulation viewer window, the time-out region forming an area in which a graphic object may be positioned to remove the graphic object from the simulation while retaining the rules and properties associated with the graphic object, and the remove icon enabling the selected graphic object to be deleted from the simulation viewer window.

27. The extensible simulation system of claim 25, further comprising a drawing editor means for modifying the graphical representation associated with a graphic object, for creating an animated transition corresponding to a graphic object action, and for producing a drawing editor window.

28. The extensible simulation system of claim 25, further comprising a graphic object property editing means for creating or modifying a property associated with a graphic object, for controlling the display of graphic object properties, and for generating a graphic object property editor window.

29. The extensible simulation system of claim 25, further comprising a graphic object rule viewing means for displaying the before portion and the after portion of each graphical rewrite rule in a graphic object rule viewer window.

30. The extensible simulation system of claim 25, further comprising a subroutine editing means for graphically combining graphical rewrite rules to form a subroutine, and for generating a subroutine editor window.

31. The extensible simulation system of claim 25, wherein:

the action recording means produces a symbolic program step for each action recorded; and

the graphical rewrite rule editing means controls the display of a set of symbolic program steps corresponding to the graphical rewrite rule.

32. The extensible simulation system of claim 25, further comprising an abstraction means for generating a type hierarchy menu for modifying a hierarchical classification of a graphic object, and for generating a property abstraction window for creating a set of property conditions for a graphical rewrite rule.

33. The extensible simulation system of claim 32, wherein the simulation execution means tests the set of property conditions and uses the hierarchical classification of a graphic object when testing the graphical rewrite rules of each graphic object in the simulation for a match with the simulation context.

34. The extensible simulation system of claim 25, further comprising a means for generating a graphic object property display, the graphic object property display showing the properties associated with a graphic object shown in the simulation viewer window.

35. The extensible simulation system of claim 25, wherein the simulation execution means generates an execution control menu having a simulation clock window, a stop button, a single step forward button, a repeated step forward button, a single step reverse button, and a repeated step reverse button for controlling the execution of simulation.

36. The extensible simulation system of claim 25, wherein the graphical rewrite rule editing means maintains a simulation context indicator and cooperates with the simulation viewing means to show the simulation context indicator in the simulation viewer window;

and

wherein the simulation context indicator is an outlined box surrounding one or more of the grid elements.

37. A computer useable medium having computer readable program code embodied therein, the program code causing a computer to execute a simulation having a plurality of graphic objects in an extensible simulation system by performing steps comprising:

determining a current simulation context comprising a graphical portion of a display generated upon a display device;

selecting a graphic object within the current simulation context;

determining whether there is a graphical rewrite rule having a before portion and an after portion associated with the selected graphic object, wherein the graphical rewrite rule defines a manner in which a first simulation context, matching the before portion, is to be graphically replaced with a second simulation context, matching the after portion;

selecting a graphical rewrite rule if there is a graphical rewrite rule associated with the selected graphic object;

determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object; and

executing a program corresponding to the selected graphical rewrite rule if the before portion of the selected graphical rewrite rule matches the current simulation context.

38. The computer useable medium of claim 37 wherein the program code for performing the step of determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object further comprises program code for comparing graphic object types and graphic object properties of the before portion of the selected graphical rewrite rule to graphic object types and properties in the current simulation context.

39. The computer useable medium of claim 37 wherein the program code for performing the step of determining whether the before portion of the selected graphical rewrite rule matches the current simulation context for the selected graphic object further comprises program code for:

comparing the relative positions of graphic objects in the before portion with the relative positions of graphic objects in the current simulation context;

comparing the number, relative position and graphic object occupancy of simulation grid elements in the before portion to the number, relative position and graphic object occupancy of simulation grid elements in the current simulation context wherein a simulation grid element comprises a displayable portion of the simulation; and

comparing graphic object types according to a set of hierarchical classifications of the before portion to graphic object types of the current simulation context; and

determining whether the property conditions defined for the graphic objects in the before portion are satisfied.

40. A computer useable medium having computer readable program code embodied therein;

the program code causing a computer to create a graphical rewrite rule for a graphic object in an extensible simulation system by performing steps comprising;

selecting the graphic object for which an action is to be defined in response to a selection from an input device;

determining a simulation context for the selected graphic object, wherein the simulation context is a subset of simulation grid elements containing the selected graphic object and wherein a simulation grid element comprises a displayable portion of the simulation;

displaying the simulation context in a graphical rewrite rule editor window; and

generating the graphical rewrite rule in response to a graphical manipulation of the graphic object via the input device,

wherein the graphical rewrite rule defines a manner in which a first simulation context, matching a before portion, is to be graphically replaced with a second simulation context, matching an after portion.

41. The computer useable medium of claim 40 wherein the program code for performing the step of creating a new program further comprises program code for:

selecting a graphic object within a defined simulation context in response to a selection from the input device;

defining an action corresponding to the selected graphic object in response to a selection from the input device;

recording the defined action and translating the defined action into a program step sequence;

creating a symbolic program step corresponding to the defined action; and

storing the before portion and the after portion of the graphical rewrite rule corresponding to the program step sequences recorded.

42. The computer useable medium of claim 41 wherein the program code for performing the step of defining an action corresponding to the selected graphic object further comprises program code selected from a group of program code consisting of:

moving the selected graphic object from one stimulation grid element to another within the simulation context in response to a selection from the input device;

defining a set of operations to be performed upon a graphic object property in response to a selection from the input device; and

creating an animated transition that alters a portion of a graphic object'"'"'s graphical representation within the simulation context in response to a selection from the input device.

43. The computer useable medium of claim 40 further comprising program code for:

determining whether the graphical rewrite rule is to be abstracted, wherein abstraction comprises the steps of analyzing the action, generalizing the action, and creating a corresponding generalized computer program step sequence; and

performing abstraction on the graphical rewrite rule if it is determined that the graphical rewrite rule is to be abstracted.

44. The computer useable medium of claim 43 wherein the program code for performing the step of performing abstraction further comprises program code for:

determining whether picture or property abstraction is to be performed;

if picture abstraction is to be performed, then

determining the graphic object the user has selected;

determining a hierarchical classification of graphic object types for the selected graphic object;

displaying a hierarchical classification of graphic object types for the selected graphic object in a hierarchy menu; and

determining the classification for the graphic object type selected and storing the selected classification as part of the graphical rewrite rule; and

if property abstraction is to be performed, then

displaying a property abstraction window;

identifying a graphic object that was selected;

displaying the properties for the selected graphic object;

determining a property, a condition and an expression input; and

storing a property abstraction definition including the property, condition and expression input as part of the graphical rewrite rule.