System and method for automatically generating a graphical program to perform an image processing algorithm

• US 6,763,515 B1
• Filed: 06/05/2000
• Issued: 07/13/2004
• Est. Priority Date: 06/05/2000
• Status: Expired due to Term

• Current Assignee: National Instruments Corporation (Emerson Electric Company)
• Original Assignee: National Instruments Corporation (Emerson Electric Company)
• Inventors: Vazquez, Nicolas, Kodosky, Jeffrey L., Caltagirone, Christophe, Kudukoli, Ram, Nair, Dinesh, Schultz, Kevin L.
• Primary Examiner(s): Zhen, Wei
• Assistant Examiner(s): STEELMAN, MARY J
• Application Number:
  US09/587,682
• Time in Patent Office: 1,499 Days
• Field of Search: 717/106-109, 717/113, 717/110, 717/115, 345/967
• US Class Current: 717/109
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/36 Software reuse
• CPC Class Codes: Y10S 715/967 Visual or iconic programming

Abstract

System and method for automatically generating a graphical program to perform an image processing algorithm. A user may develop an image processing algorithm in an image prototyping environment. The image prototyping environment enables the user to easily apply various image processing functions to an image and immediately see the results, in order to develop the desired algorithm. As the user applies each image processing function to an image, the function may be recorded as a step in a script. Once the user has developed an algorithm, the user may request the image prototyping environment to automatically generate a program implementing the image processing algorithm. In various embodiments, the prototyping environment may be operable to generate different types of programs, including text-based and graphical programs. For example, a text-based program may include function calls corresponding to each step in the script recorded while developing the image processing algorithm, while a graphical program may include nodes corresponding to the steps of the script. The generated program may be operable to display a user interface comprising controls for interactively changing various parameters used by the image processing functions. An application programming interface (API) provided by a graphical programming environment that supports the dynamic creation/editing of a graphical program is described. The image prototyping environment may interface with a graphical programming environment through such an API in order to generate a graphical program that implements an algorithm specified by a script.

Claims

1. A memory medium comprising program instructions executable to:

perform a plurality of image processing functions on an image in response to user input, wherein the plurality of image processing functions form an image processing algorithm;

record the plurality of image processing functions; and

programmatically generate a graphical program based on the recorded plurality of image processing functions, wherein the graphical program implements the image processing algorithm comprising the plurality of image processing functions, wherein said programmatically generating the graphical program comprises programmatically generating nodes corresponding to the image processing functions and links among the nodes in the graphical program without user input selecting the nodes and specifying the links.

2. The memory medium of claim 1,

wherein each of the image processing functions is recorded after the user input indicating performance of the respective image processing function.

3. The memory medium of claim 1, further comprising program instructions executable to:

execute the graphical program to perform the image processing algorithm.

4. The memory medium of claim 1,

wherein the graphical program comprises a plurality of interconnected nodes which visually indicate functionality of the graphical program.

5. The memory medium of claim 4,

wherein the graphical program is a graphical data flow program;

wherein the links represent data flow among the nodes.

6. The memory medium of claim 1,

wherein the graphical program includes a block diagram portion and a front panel portion.

7. The memory medium of claim 1,

wherein said programmatically generating the graphical program comprises programmatically displaying the nodes corresponding to the plurality of image processing functions and the links between the nodes in the graphical program.

8. The memory medium of claim 1, wherein the memory medium further comprising program instructions executable to:

create an association between the recorded plurality of image processing functions and the graphical program;

modify the recorded plurality of image processing functions to create a new recorded plurality of image processing functions in response to user input after said creating the association; and

modify the graphical program according to the new recorded plurality of image processing functions to create a new graphical program.

9. The memory medium of claim 8,

wherein said modifying the graphical program according to the new recorded plurality of image processing functions uses the association between the recorded plurality of image processing functions and the graphical program;

wherein the association remains between the new recorded plurality of image processing functions and the new graphical program.

10. The memory medium of claim 8, further comprising program instructions executable to:

receive user input indicating a desire to change the graphical program;

display information regarding the recorded plurality of image processing functions;

modify the information in response to user input to create modified information; and

modify the graphical program according to said modified information.

11. The memory medium of claim 1, further comprising program instructions executable to:

create an association between the recorded plurality of image processing functions and the graphical program;

lock the association between the recorded plurality of image processing functions and the graphical program, wherein said locking prevents user editing of the graphical program.

12. The memory medium of claim 11, further comprising program instructions executable to:

unlock the association between the recorded plurality of image processing functions and the graphical program in response to user input after said locking;

directly change the graphical program in response to user input after said unlocking.

13. The memory medium of claim 12,

wherein said unlocking removes the association between the recorded plurality of image processing functions and the graphical program.

14. The memory medium of claim 1, further comprising program instructions executable to:

create an association between the recorded plurality of image processing functions and the graphical program;

modify the graphical program in response to user input after said generating the graphical program and after said creating the association between the recorded plurality of image processing functions and the graphical program;

determine if an association exists between the recorded plurality of image processing functions and the graphical program in response to said modifying the graphical program; and

remove the association between the recorded plurality of image processing functions and the graphical program in response to said modifying.

15. The memory medium of claim 1, further comprising program instructions executable to:

receive user input specifying code generation information;

wherein said programmatically generating the graphical program utilizes the code generation information in said programmatically generating.

16. The memory medium of claim 15,

wherein the code generation information specifies a type of program to create in response to the recorded plurality of image processing functions;

wherein the type of type of program specified is a graphical program type;

wherein the graphical program is created in accordance with the specified graphical program type.

17. The memory medium of claim 15,

wherein the code generation information specifies an image source for the program to be generated;

wherein the generated graphical program is operable to obtain an image from the specified image source and perform the image processing algorithm on the image.

18. The memory medium of claim 17,

wherein the image source is one of the following:

an image file;

or

a hardware device.

19. The memory medium of claim 15,

wherein a plurality of parameters are associated with the image processing functions, wherein each parameter is an input parameter which provides input to an image processing function or an output parameter which accepts output from an image processing function;

wherein the code generation information specifies one or more of the input parameters which are desired to be interactively changeable or one or more of the output parameters which are desired to be interactively viewable;

wherein said programmatically generating the graphical program comprises enabling the graphical program to receive user input during program operation, wherein the user input specifies values for the specified one or more input parameters;

wherein said programmatically generating the graphical program comprises enabling the graphical program to display output during program operation, wherein the output indicates values for the specified one or more output parameters.

20. The memory medium of claim 19,

wherein the code generation information specifies one or more of the input parameters which are desired to be interactively changeable;

wherein said programmatically generating the graphical program comprises programmatically creating a user interface associated with the graphical program;

wherein the graphical program is operable to display the user interface during program operation;

wherein the user interface includes a user interface control for each specified input parameter;

wherein the user interface control associated with a particular input parameter is operable to accept user input specifying a value and provide the value to the image processing function with which the parameter is associated.

21. The memory medium of claim 19,

wherein the code generation information specifies one or more of the output parameters which are desired to be interactively viewable;

wherein said programmatically generating the graphical program comprises creating a user interface associated with the graphical program;

wherein the graphical program is operable to display the user interface during program operation;

wherein the user interface includes a user interface indicator for each specified output parameter;

wherein the user interface indicator associated with a particular output parameter is operable to accept program output specifying a value for the parameter and display the value on the user interface.

22. The memory medium of claim 1,

wherein said programmatically generating the graphical program comprises:

generating portions of graphical code, wherein each portion of graphical code implements one of the image processing functions;

linking the portions of graphical code together.

23. The memory medium of claim 22,

wherein each portion of graphical code includes one or more graphical program nodes, wherein each node has one or more inputs or outputs;

wherein generating each portion of graphical code comprises connecting the node inputs and outputs together in order to implement the image processing function with which the portion of graphical code is associated.

24. The memory medium of claim 23,

wherein linking a first portion of graphical code to a second portion of graphical code comprises connecting an output of a node in the first portion of graphical code to an input of a node in the second portion of graphical code.

25. The memory medium of claim 24,

wherein at least one of the image processing functions has an associated output parameter;

wherein each portion of code that implements an image processing function that has an associated output parameter includes a node that has an output for providing a value for the output parameter;

wherein each portion of code that implements an image processing function that has an associated output parameter includes a leaf node that has an input for receiving a value for the output parameter;

wherein the leaf node input for receiving the parameter value is connected to the node output for providing the parameter value.

26. The memory medium of claim 23,

wherein at least one of the image processing functions has an associated input parameter;

wherein each portion of code that implements an image processing function that has an associated input parameter includes a node that has an input for receiving a value for the input parameter;

wherein each portion of code that implements an image processing function that has an associated input parameter includes a leaf node that has an output for providing a value for the input parameter;

wherein the leaf node output for providing the parameter value is connected to the node input for receiving the parameter value.

27. The memory medium of claim 1, further comprising:

for each of at least a subset of the image processing functions, retrieving information associated with the image processing function from a database;

wherein programmatically generating the nodes corresponding to the image processing functions utilizes the database information retrieved for the particular image processing function.

28. The memory medium of claim 1,

wherein said programmatically generating the graphical program comprises:

generating a portion of graphical code for at least one of the image processing functions,

wherein the portion of graphical code comprises a first plurality of nodes connected by links which perform the at least one image processing function.

29. The memory medium of claim 1,

wherein said programmatically generating nodes corresponding to the image processing functions comprises programmatically generating a first plurality of nodes and links between the first plurality of nodes for each of at least a subset of the image processing functions.

30. A method of creating a graphical program to perform an image processing algorithm, the method comprising:

performing a plurality of image processing functions on an image in response to user input;

storing information regarding the plurality of image processing functions and an ordering of the image processing functions as performed by the user; and

programmatically generating the graphical program based on the stored information, wherein the graphical program is executable to implement the plurality of image processing functions according to said ordering, wherein said programmatically generating the graphical program comprises programmatically generating nodes corresponding to the image processing functions and links among the nodes in the graphical program without user input selecting the nodes and specifying the links.

31. The method of claim 30, further comprising:

executing the graphical program to perform the image processing algorithm.

32. The method of claim 30,

wherein the graphical program is a graphical data flow program.

33. The method of claim 30,

wherein said programmatically generating comprises:

programmatically displaying the nodes on a display, wherein each of the nodes corresponds to one of the plurality of image processing functions; and

programmatically displaying the links between each of the nodes, wherein the links are programmatically displayed according to said ordering.

34. The method of claim 30, wherein the recorded plurality of image processing functions comprise a script, the method further comprising:

creating an association between the script and the graphical program;

modifying the script to create a new script in response to user input after said creating the association; and

modifying the graphical program according to the new script to create a new graphical program.

35. The method of claim 34,

wherein said modifying the graphical program according to the new script uses the association between the script and the graphical program;

wherein the association remains between the new script and the new graphical program.

36. The method of claim 30,

wherein the recorded plurality of image processing functions comprise a script, the method further comprising:

receiving user input indicating a desire to change the graphical program;

displaying script information of the script;

modifying the script information in response to user input; and

modifying the graphical program after said modifying the script information.

37. The method of claim 30,

wherein the recorded plurality of image processing functions comprise a script, the method further comprising:

creating an association between the script and the graphical program;

locking the association between the script and the graphical program, wherein said locking prevents user editing of the graphical program.

38. The method of claim 37, further comprising:

unlocking the association between the script and the graphical program in response to user input after said locking;

directly changing the graphical program in response to user input after said unlocking.

39. The method of claim 38,

wherein said unlocking removes the association between the script and the graphical program.

40. The method of claim 30, further comprising:

modifying the graphical program in response to user input after said generating the graphical program and after said creating the association between the script and the graphical program;

determining if an association exists between the script and the graphical program in response to said modifying the graphical program; and

removing the association between the script and the graphical program in response to said modifying.

41. The method of claim 30, further comprising:

receiving user input specifying code generation information;

wherein said programmatically generating the graphical program utilizes the code generation information.

42. The method of claim 30,

wherein a plurality of parameters are associated with the image processing functions, wherein each parameter is one of an input parameter which provides input to an image processing function or an output parameter which accepts output from an image processing function;

wherein the code generation information specifies one or more of the input parameters which are desired to be interactively changeable and/or one or more of the output parameters which are desired to be interactively viewable;

wherein said programmatically generating the graphical program comprises enabling the graphical program to perform at least one of;

receive user input during program operation, wherein the user input specifies values for the specified one or more input parameters, and/or display output during program operation, wherein the output indicates values for the specified one or more output parameters.

43. The method of claim 30,

wherein said programmatically generating the graphical program comprises:

generating portions of graphical code, wherein each portion of graphical code implements one of the image processing functions;

linking the portions of graphical code together.

44. The method of claim 43,

wherein each portion of graphical code includes one or more graphical program nodes, wherein each node has one or more inputs or outputs;

wherein generating each portion of graphical code comprises connecting the node inputs and outputs together in order to implement the image processing function with which the portion of graphical code is associated.

45. The method of claim 44,

wherein linking a first portion of graphical code to a second portion of graphical code comprises connecting an output of a node in the first portion of graphical code to an input of a node in the second portion of graphical code.

46. A system for creating a graphical program to perform an image processing algorithm, the system comprising:

a processor;

a memory coupled to the processor which stores an image, wherein the memory also stores an image prototyping program and a graphical program creation program;

a user input device which receives user input;

wherein the image prototyping program is executable in response to the user input to perform a plurality of image processing functions on the image;

wherein the memory stores the plurality of image processing functions;

wherein the graphical program creation program is executable to programmatically generate a graphical program based on the stored plurality of image processing functions, wherein the graphical program implements the plurality of image processing functions, and wherein, in programmatically generating the graphical program, the graphical program creation program is executable to programmatically generate nodes corresponding to the image processing functions and links among the nodes in the graphical program without user input selecting the nodes and specifying the links.

47. The system of claim 46,

wherein the image prototyping program is executable to call the graphical program creation program;

wherein the graphical program creation program is executable to programmatically generate the graphical program in response to said image prototyping program calling the graphical program creation program.

48. The system of claim 46,

wherein the graphical program is a graphical data flow program.

49. A method of creating a graphical program to perform a machine vision algorithm, the method comprising:

performing one or more machine vision functions on an image in response to user input;

recording the one or more machine vision functions; and

programmatically generating the graphical program based on the recorded one or more machine vision functions, wherein the graphical program implements a machine vision algorithm comprising the one or more machine vision functions, wherein said programmatically generating the graphical program comprises programmatically generating nodes corresponding to the machine vision functions and links among the nodes in the graphical program without user input selecting the nodes and specifying the links.

50. A memory medium comprising program instructions executable to:

display an object on a display;

perform a plurality of functions on the object in response to user input;

record the plurality of functions; and

programmatically generate a graphical program based on the recorded plurality of functions, wherein the graphical program implements the plurality of functions, and wherein said programmatically generating the graphical program comprises programmatically generating nodes corresponding to the functions and links among the nodes in the graphical program without user input selecting the nodes and specifying the links.

51. The memory medium of claim 50,

wherein each of the image processing functions is recorded after the user input indicating performance of the respective image processing function.

52. The memory medium of claim 50, further comprising program instructions executable to:

execute the graphical program to perform the plurality of functions.

53. The memory medium of claim 50,

wherein the graphical program comprises a plurality of interconnected nodes which visually indicate functionality of the graphical program.

54. The memory medium of claim 53,

wherein the graphical program is a graphical data flow program.

55. The memory medium of claim 50,

wherein the graphical program includes a block diagram portion and a front panel portion.

56. The memory medium of claim 50,

wherein said programmatically generating the graphical program comprises displaying the nodes corresponding to the plurality of functions and the links between the nodes in the graphical program.

57. The memory medium of claim 50, wherein the memory medium further comprising program instructions executable to:

create an association between the recorded plurality of image processing functions and the graphical program;

modify the recorded plurality of image processing functions to create a new recorded plurality of image processing functions in response to user input after said creating the association; and

modify the graphical program according to the new recorded plurality of image processing functions to create a new graphical program.

58. A method of creating a graphical program to perform an algorithm, the method comprising:

performing one or more functions in response to user input;

recording the one or more functions; and

programmatically generating the graphical program based on recorded one or more functions, wherein the graphical program implements an algorithm comprising the one or more functions, and wherein said programmatically generating the graphical program comprises programmatically generating nodes corresponding to the image processing functions and links among the nodes in the graphical program without user input selecting the nodes and specifying the links.