Intelligent controller with neural network and reinforcement learning

• US 5,448,681 A
• Filed: 03/27/1992
• Issued: 09/05/1995
• Est. Priority Date: 03/27/1992
• Status: Expired due to Term

• Current Assignee: National Semiconductor Corporation (Texas Instruments, Inc.)
• Original Assignee: National Semiconductor Corporation (Texas Instruments, Inc.)
• Inventors: Khan, Emdadur R.
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): HAFIZ, TARIQ R
• Application Number:
  US07/859,328
• Time in Patent Office: 1,257 Days
• Field of Search: 395/11, 395/22-24, 395/27, 395/906, 395/907, 364/152
• US Class Current: 706/23
• CPC Class Codes: G05B 13/027 using neural networks only
• CPC Class Codes: G06N 3/043 based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS]
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: Y10S 706/906 Process plant
• CPC Class Codes: Y10S 706/907 Power plant

Abstract

A plant controller using reinforcement learning for controlling a plant includes action and critic networks with enhanced learning for generating a plant control signal. Learning is enhanced within the action network by using a neural network configured to operate according to unsupervised learning techniques based upon a Kohonen Feature Map. Learning is enhanced within the critic network by using a distance parameter which represents the difference between the actual and desired states of the quantitative performance, or output, of the plant when generating the reinforcement signal for the action network.

Claims

1. A plant controller using reinforcement learning for controlling a plant by generating a control signal therefor, the plant controller comprising:

an action network for coupling to a plant and receiving therefrom a plant state signal, receiving a reinforcement signal and in accordance therewith generating a plant control signal, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant, the reinforcement signal represents a desired state of a performance control parameter for the action network, and the plant control signal represents a desired state of the quantitative plant performance parameter, and further wherein the plant control signal is generated in further accordance with unsupervised learning; and

a critic network, coupled to the action network, for coupling to the plant and receiving therefrom the plant state signal and a plant performance signal and in accordance therewith outputting the reinforcement signal to the action network, wherein the plant performance signal represents a qualitative plant performance parameter for the plant.

2. A plant controller as recited in claim 1, wherein the action network generates a continuous plant control signal.

3. A plant controller as recited in claim 1, wherein the action network comprises an unsupervised learning based neural network.

4. A plant controller as recited in claim 3, wherein the unsupervised learning based neural network comprises first, second and third neuron layers with first and second pluralities of inter-neuron weights data stored in memory and associated therewith, and wherein the first and second pluralities of inter-neuron weights data are selectively modified in accordance with the reinforcement signal.

5. A plant controller as recited in claim 1, wherein the critic network comprises a neural network.

6. A plant controller as recited in claim 5, wherein the neural network comprises first, second and third neuron layers with first, second and third pluralities of inter-neuron weights data stored in memory and associated therewith, and wherein the first, second and third pluralities of inter-neuron weights data are selectively modified in accordance with the plant performance signal.

7. A plant controller using reinforcement learning for controlling a plant by generating a control signal therefor, the plant controller comprising:

an action network for coupling to a plant and receiving therefrom a plant state signal, receiving a reinforcement signal and in accordance therewith generating a plant control signal, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant, the reinforcement signal represents a desired state of a performance control parameter for the action network, and the plant control signal represents a desired state of the quantitative plant performance parameter; and

a critic network, coupled to the action network, for coupling to the plant and receiving therefrom the plant state signal and a plant performance signal and in accordance therewith outputting the reinforcement signal to the action network, wherein the plant performance signal represents a qualitative plant performance parameter for the plant, and further wherein the reinforcement signal is outputted in further accordance with a distance parameter which represents a difference between the actual and desired states of the quantitative plant performance parameter.

8. A plant controller as recited in claim 7, wherein the action network generates a continuous plant control signal.

9. A plant controller as recited in claim 7, wherein the action network comprises a neural network.

10. A plant controller as recited in claim 9, wherein the neural network comprises first, second and third neuron layers with first and second pluralities of inter-neuron weights data stored in memory and associated therewith, and wherein the first and second pluralities of inter-neuron weights data are selectively modified in accordance with the reinforcement signal.

11. A plant controller as recited in claim 9, wherein the critic network comprises a neural network.

12. A plant controller as recited in claim 11, wherein the neural network comprises first, second and third neuron layers with first, second and third pluralities of inter-neuron weights data stored in memory and associated therewith, and wherein the first, second and third pluralities of inter-neuron weights data are selectively modified in accordance with the plant performance signal.

13. A plant control method using reinforcement learning for controlling a plant by generating a control signal therefor, comprising the steps of:

receiving a plant state signal and a plant performance signal from a plant, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant and the plant performance signal represents a qualitative plant performance parameter for the plant;

generating a reinforcement signal in accordance with the plant state signal and the plant performance signal, wherein the reinforcement signal represents a desired state of a performance control parameter;

receiving the reinforcement signal; and

generating a plant control signal in accordance with the plant state signal and the reinforcement signal and in further accordance with unsupervised learning, wherein the plant control signal represents a desired state of the quantitative plant performance parameter.

14. A plant control method as recited in claim 13, wherein the step of generating the plant control signal in accordance with the plant state signal and the reinforcement signal comprises generating a continuous plant control signal.

15. A plant control method as recited in claim 13, wherein the step of generating the plant control signal in accordance with the plant state signal and the reinforcement signal comprises generating the plant control signal with an unsupervised learning based neural network which includes first and second pluralities of inter-neuron weights data associated therewith.

16. A plant control method as recited in claim 15, wherein the step of generating the plant control signal with an unsupervised learning based neural network comprises selectively modifying the first and second pluralities of inter-neuron weights data in accordance with the reinforcement signal.

17. A plant control method as recited in claim 13, wherein the step of generating the reinforcement signal in accordance with the plant state signal and the plant performance signal comprises generating the reinforcement signal with a neural network.

18. A plant control method as recited in claim 17, wherein the step of generating the reinforcement signal with a neural network comprises selectively modifying first, second and third pluralities of inter-neuron weights data in accordance with the plant performance signal.

19. A plant control method using reinforcement learning for controlling a plant by generating a control signal therefor, comprising the steps of:

receiving a plant state signal and a plant performance signal from a plant, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant and the plant performance signal represents a qualitative plant performance parameter for the plant,

generating a reinforcement signal in accordance with the plant state signal and the plant performance signal and in further accordance with a distance parameter which represents a difference between the actual state and a desired state of the quantitative plant performance parameter, wherein the reinforcement signal represents a desired state of a performance control parameter;

receiving the reinforcement signal; and

generating a plant control signal in accordance with the plant state signal and the reinforcement signal, wherein the plant control signal represents the desired state of the quantitative plant performance parameter.

20. A plant control method as recited in claim 19, wherein the step of generating the plant control signal in accordance with the plant state signal and the reinforcement signal comprises generating a continuous plant control signal.

21. A plant control method as recited in claim 19, wherein the step of generating the plant control signal in accordance with the plant state signal and the reinforcement signal comprises generating the plant control signal with a neural network which includes first and second pluralities of inter-neuron weights data associated therewith.

22. A plant control method as recited in claim 21, wherein the step of generating the plant control signal with a neural network comprises selectively modifying the first and second pluralities of inter-neuron weights data in accordance with the reinforcement signal.

23. A plant control method as recited in claim 19, wherein the step of generating the reinforcement signal in accordance with the plant state signal and the plant performance signal comprises generating the reinforcement signal with a neural network.

24. A plant control method as recited in claim 23, wherein the step of generating the reinforcement signal with a neural network comprises selectively modifying first, second and third pluralities of inter-neuron weights data in accordance with the plant performance signal.

25. A plant controller using reinforcement learning for controlling a plant by generating a control signal therefor, the plant controller comprising:

an action network for coupling to a plant and receiving therefrom a plant state signal, receiving a reinforcement signal and in accordance therewith generating a plant control signal, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant, the reinforcement signal represents a desired state of a performance control parameter for the action network, and the plant control signal represents a desired state of the quantitative plant performance parameter, and wherein the plant control signal is generated in further accordance with unsupervised learning, and further wherein the action network comprises an unsupervised learning based neural network, and still further wherein the unsupervised learning based neural network is configured to perform Kohonen feature mapping; and

a critic network, coupled to the action network, for coupling to the plant and receiving therefrom the plant state signal and a plant performance signal and in accordance therewith outputting the reinforcement signal to the action network, wherein the plant performance signal represents a qualitative plant performance parameter for the plant.

26. A plant controller using reinforcement learning for controlling a plant by generating a control signal therefor, the plant controller comprising:

an action network for coupling to a plant and receiving therefrom a plant state signal, receiving a reinforcement signal and in accordance therewith generating a plant control signal, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant, the reinforcement signal represents a desired state of a performance control parameter for the action network, and the plant control signal represents a desired state of the quantitative plant performance parameter, and wherein the plant control signal is generated in further accordance with unsupervised learning, and further wherein the action network generates the plant control signal in still further accordance with a plurality of fuzzy logic rules; and

a critic network, coupled to the action network, for coupling to the plant and receiving therefrom the plant state signal and a plant performance signal and in accordance therewith outputting the reinforcement signal to the action network, wherein the plant performance signal represents a qualitative plant performance parameter for the plant.

27. A plant controller using reinforcement learning for controlling a plant by generating a control signal therefor, the plant controller comprising:

an action network for coupling to a plant and receiving therefrom a plant state signal, receiving a reinforcement signal and in accordance therewith generating a plant control signal, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant, the reinforcement signal represents a desired state of a performance control parameter for the action network, and the plant control signal represents a desired state of the quantitative plant performance parameter, and wherein the action network comprises a neural network, and further wherein the neural network is configured to perform Kohonen feature mapping; and

a critic network, coupled to the action network, for coupling to the plant and receiving therefrom the plant state signal and a plant performance signal and in accordance therewith outputting the reinforcement signal to the action network, wherein, the plant performance signal represents a qualitative plant performance parameter for the plant, and further wherein the reinforcement signal is outputted in further accordance with a distance parameter which represents a difference between the actual and desired states of the quantitative plant performance parameter.

28. A plant controller using reinforcement learning for controlling a plant by generating a control signal therefor, the plant controller comprising:

an action network for coupling to a plant and receiving therefrom a plant state signal, receiving a reinforcement signal and in accordance therewith generating a plant control signal, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant, the reinforcement signal represents a desired state of a performance control parameter for the action network, and the plant control signal represents a desired state of the quantitative plant performance parameter, and wherein the action network generates the plant control signal in still further accordance with a plurality of fuzzy logic rules; and

a critic network, coupled to the action network, for coupling to the plant and receiving therefrom the plant state signal and a plant performance signal and in accordance therewith outputting the reinforcement signal to the action network, wherein the plant performance signal represents a qualitative plant performance parameter for the plant, and further wherein the reinforcement signal is outputted in further accordance with a distance parameter which represents a difference between the actual and desired states of the quantitative plant performance parameter.

29. A plant control method using reinforcement learning for controlling a plant by generating a control signal therefor, comprising the steps of:

receiving a plant state signal and a plant performance signal from a plant, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant and the plant performance signal represents a qualitative plant performance parameter for the plant;

generating a reinforcement signal in accordance with the plant state signal and the plant performance signal, wherein the reinforcement signal represents a desired state of a performance control parameter;

receiving the reinforcement signal; and

generating a plant control signal in accordance with the plant state signal and the reinforcement signal and in further accordance with unsupervised learning and Kohonen feature mapping with an unsupervised learning based neural network which includes first and second pluralities of inter-neuron weights data associated therewith, wherein the plant control signal represents a desired state of the quantitative plant performance parameter.

30. A plant control method using reinforcement learning for controlling a plant by generating a control signal therefor, comprising the steps of:

receiving a plant state signal and a plant performance signal from a plant, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant and the plant performance signal represents a qualitative plant performance parameter for the plant;

generating a reinforcement signal in accordance with the plant state signal and the plant performance signal, wherein the reinforcement signal represents a desired state of a performance control parameter;

receiving the reinforcement signal; and

generating a plant control signal in accordance with the plant state signal and the reinforcement signal and in further accordance with unsupervised learning and in still further accordance with a plurality of fuzzy logic rules, wherein the plant control signal represents a desired state of the quantitative plant performance parameter.

31. A plant control method using reinforcement learning for controlling a plant by generating a control signal therefor, comprising the steps of:

receiving a plant state signal and a plant performance signal from a plant, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant and the plant performance signal represents a qualitative plant performance parameter for the plant,

generating a reinforcement signal in accordance with the plant state signal and the plant performance signal and in further accordance with a distance parameter which represents a difference between the actual state and a desired state of the quantitative plant performance parameter, wherein the reinforcement signal represents a desired state of a performance control parameter;

receiving the reinforcement signal; and

generating a plant control signal in accordance with the plant state signal, the reinforcement signal and Kohonen feature mapping with a neural network which includes first and second pluralities of inter-neuron weights data associated therewith, wherein the plant control signal represents the desired state of the quantitative plant performance parameter.

32. A plant control method using reinforcement learning for controlling a plant by generating a control signal therefor, comprising the steps of:

receiving a plant state signal and a plant performance signal from a plant, wherein the plant state signal represents an actual state of a quantitative plant performance parameter for the plant and the plant performance signal represents a qualitative plant performance parameter for the plant,

generating a reinforcement signal in accordance with the plant state signal and the plant performance signal and in further accordance with a distance parameter which represents a difference between the actual state and a desired state of the quantitative plant performance parameter, wherein the reinforcement signal represents a desired state of a performance control parameter;

receiving the reinforcement signal; and

generating a plant control signal in accordance with the plant state signal, the reinforcement signal and a plurality of fuzzy logic rules, wherein the plant control signal represents the desired state of the quantitative plant performance parameter.