Control policy learning and vehicle control method based on reinforcement learning without active exploration

• US 10,061,316 B2
• Filed: 05/12/2017
• Issued: 08/28/2018
• Est. Priority Date: 07/08/2016
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toyota Chuo Kenkyusho (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Nishi, Tomoki
• Primary Examiner(s): Smith, Jelani A
• Assistant Examiner(s): Martinez Borrero, Luis A
• Application Number:
  US15/594,020
• Publication Number:
  US 20180011488A1
• Time in Patent Office: 473 Days
• Field of Search: 701117, 701527, 701 1, 701123, 701119, 701 2, 701 23, 701 4, 701411, 701418
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/045 Combinations of networks

Abstract

A computer-implemented method is provided for autonomously controlling a vehicle to perform a vehicle operation. The method includes steps of applying a passive actor-critic reinforcement learning method to passively-collected data relating to the vehicle operation, to learn a control policy configured for controlling the vehicle so as to perform the vehicle operation with a minimum expected cumulative cost; and controlling the vehicle in accordance with the control policy to perform the vehicle operation.

Claims

1. A computer-implemented method for autonomously controlling a vehicle to perform a vehicle operation, the method comprising steps of:

applying a passive actor-critic reinforcement learning method to passively-collected data relating to the vehicle operation, to adapt an existing control policy so as to enable control of the vehicle by the control policy so as to perform the vehicle operation with a minimum expected cumulative cost, the step of applying a passive actor-critic reinforcement learning method to passively-collected data including steps of;

a) in a critic network, estimating a Z-value and an average cost under an optimal control policy using samples of the passively collected data;

b) in an actor network operatively coupled to the critic network, revising the control policy using samples of the passively collected data, the estimated Z-value, and the estimated average cost under an optimal control policy from the critic network; and

c) iteratively repeating steps (a)-(b) until the estimated average cost converges; and

controlling the vehicle in accordance with the adapted control policy to perform the vehicle operation.

2. The method of claim 1 wherein the vehicle operation is an operation for merging the vehicle into a traffic lane between a second vehicle and a third vehicle traveling in the traffic lane, and wherein the control policy is configured for controlling the vehicle to merge the vehicle midway between the second vehicle and the third vehicle.

3. The method of claim 1 wherein the Z-value is estimated using a linearized version of a Bellman equation.

4. The method of claim 1 wherein the step of estimating the average cost under an optimal policy comprises the step of, prior to the step of revising the control policy, updating the average cost.

5. The method of claim 1 wherein the step of estimating a Z-value comprises the steps of:

approximating a Z-value function using a linear combination of weighted radial basis functions; and

approximating a Z-value using the approximated Z-value function and samples of the passively-collected data.

6. The method of claim 5 wherein the step of approximating a Z-value function using a linear combination of weighted radial basis functions comprises the step of optimizing weights used in the weighted radial basis functions.

7. The method of claim 6 wherein the step of approximating a Z-value function using a linear combination of weighted radial basis functions comprises the step of, prior to the step of optimizing the weights, updating the weights used in the weighted radial basis functions.

8. The method of claim 1 wherein the step of revising the control policy comprises steps of:

approximating a control gain;

optimizing the control gain to provide an optimized control gain; and

revising the control policy using the optimized control gain.

9. The method of claim 8 further comprising the steps of, prior to optimizing the control gain:

determining a control input; and

determining a value of an action-value function using the control input, samples of the passively-collected data, and the approximated control gain.

10. The method of claim 8 wherein the step of approximating a control gain comprises the step of approximating the control gain using a linear combination of weighted radial basis functions.

11. The method of claim 10 further comprising the step of, prior to the step of approximating the control gain using a linear combination of weighted radial basis functions, updating weights used in the weighted radial basis functions.

12. A computer-implemented method for optimizing a control policy usable for controlling a system to perform an operation, the method comprising steps of:

providing a control policy usable for controlling the system; and

applying a passive actor-critic reinforcement learning method to passively-collected data relating to the operation to be performed, to revise the control policy such that the control policy is operable to control the system to perform the operation with a minimum expected cumulative cost, wherein the step of applying a passive actor-critic reinforcement learning method to passively-collected data includes steps of;

a) in a critic network, estimating a Z-value using samples of the passively-collected data, and estimating an average cost under an optimal policy using samples of the passively-collected data;

b) in an actor network, revising the control policy using samples of the passively-collected data, a control dynamics for the system, a cost-to-go, and a control gain;

c) updating parameters used in revising the control policy and in estimating the Z-value and the average cost under an optimal policy; and

d) iteratively repeating steps (a)-(c) until the estimated average cost converges.

13. A computing system configured for optimizing a control policy usable for autonomously controlling a vehicle to perform a vehicle operation, the computing system including one or more processors for controlling operation of the computing system, and a memory for storing data and program instructions usable by the one or more processors, wherein the memory is configured to store computer code that, when executed by the one or more processors, causes the one or more processors to:

a) receive passively-collected data relating to the vehicle operation;

b) determine a Z-value function usable for estimating a cost-to-go for the vehicle;

c) in a critic network in the computing system;

c1) determine a Z-value using the Z-value function and samples of the passively-collected data;

c2) estimate an average cost under an optimal policy using samples of the passively-collected data

d) in an actor network in the computing system, revise the control policy using samples of the passively-collected data;

a control dynamics for the vehicle;

a cost-to-go, and a control gain; and

e) iteratively repeat steps (c) and (d) until the estimated average cost converges.