Convergent actor critic-based fuzzy reinforcement learning apparatus and method

• US 6,917,925 B2
• Filed: 12/21/2001
• Issued: 07/12/2005
• Est. Priority Date: 03/30/2001
• Status: Expired due to Fees

• Current Assignee: Intelligent Inference Systems Corporation
• Original Assignee: Intelligent Inference Systems Corporation
• Inventors: Vengrov, David, Berenji, Hamid R.
• Primary Examiner(s): Knight, Anthony
• Assistant Examiner(s): Hartman, Jr., Ronald D
• Application Number:
  US10/027,817
• Publication Number:
  US 20020198854A1
• Time in Patent Office: 1,299 Days
• Field of Search: 706/12, 706 1- 10, 706/16, 706/18, 706/25, 700/50, 700/52, 318/567
• US Class Current: 706/10
• CPC Class Codes: G06N 7/023 Learning or tuning the parameters of a fuzzy system

Abstract

A system is controlled by an actor-critic based fuzzy reinforcement learning algorithm that provides instructions to a processor of the system for applying actor-critic based fuzzy reinforcement learning. The system includes a database of fuzzy-logic rules for mapping input data to output commands for modifying a system state, and a reinforcement learning algorithm for updating the fuzzy-logic rules database based on effects on the system state of the output commands mapped from the input data. The reinforcement learning algorithm is configured to converge at least one parameter of the system state to at least approximately an optimum value following multiple mapping and updating iterations. The reinforcement learning algorithm may be based on an update equation including a derivative with respect to at least one parameter of a logarithm of a probability function for taking a selected action when a selected state is encountered.

Claims

1. A method of controlling a system including a processor for applying actor-critic based fuzzy reinforcement learning to perform power control in a wireless transmitter, comprising the acts of:

mapping input data to output commands for modifying a system state according to fuzzy-logic rules;

using continuous, reinforcement learning, updating the fuzzy-logic rules based on effects on the system state of the output commands mapped from the input data; and

converging at least one parameter of the system state towards at least approximately an optimum value following multiple mapping and updating iterations.

2. The method of claim 1, wherein updating includes taking a derivative with respect to said at least one parameter of a logarithm of a probability function for taking a selected action when a selected state is encountered.

3. The method of claim 2, wherein updating includes updating the at least one parameter based on said derivative.

4. A computer-readable medium containing instructions which, when executed by a computer, control a system for applying actor-critic based fuzzy reinforcement learning, by:

maintaining a database of fuzzy-logic rules for mapping input data to output commands for modifying a system state by using continuous, reinforcement learning to update the fuzzy-logic rules database based on effects on the system state of the output commands to control a wireless transmitter, the output commands mapped from the input data; and

converging at least one parameter of the system state towards at least approximately an optimum value following multiple mapping and updating iterations.

5. The computer-readable medium of claim 4, wherein updating the fuzzy-logic database comprises taking a derivative with respect to said at least one parameter of a logarithm of a probability function for taking a selected action when a selected state is encountered.

6. The computer-readable medium of claim 5, wherein the at least one parameter is updated by taking the derivative with respect to said at least one parameter of a logarithm of a probability function for taking a selected action when a selected state is encountered.

7. A system controlled by actor-critic based fuzzy reinforcement learning, comprising:

a processor;

at least one system component whose actions are controlled by the processor; and

instructions, which, when executed by the processor;

maintain a database of fuzzy-logic rules for mapping input data to output commands for modifying a system state by using continuous, reinforcement learning to update the fuzzy-logic rules database based on effects on the system state of the output commands mapped from the input data, wherein updating the fuzzy-logic database comprises taking a derivative with respect to said at least one parameter of a logarithm of a probability function for taking a selected action when a selected state is encountered; and

converge at least one parameter of the system state towards at least approximately an optimum value following multiple mapping and updating iterations wherein updating the fuzzy-logic database comprises taking a derivative with respect to said at least one parameter of a logarithm of a probability function for taking a selected action when a selected state is encountered.

8. The system of claim 7, wherein the at least one parameter is updated by taking the derivative with respect to said at least one parameter of a logarithm of a probability function for taking a selected action when a selected is encountered.

9. The system of any of claims 7-8, wherein the system state comprises system state of a wireless transmitter.