Autonomous behavior control using policy triggering and execution

• US 10,474,149 B2
• Filed: 08/18/2017
• Issued: 11/12/2019
• Est. Priority Date: 08/18/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Palanisamy, Praveen, Mudalige, Upali P.
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Greene, Daniel L
• Application Number:
  US15/680,599
• Publication Number:
  US 20190056737A1
• Time in Patent Office: 816 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 30/182 Selecting between different operative modes, e.g. comfort and performance modes
• CPC Class Codes: B60W 60/0015 specially adapted for safety
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06N 7/08 using chaos models or non-linear system models

Abstract

An autonomous vehicle, a system and method of operating the autonomous vehicle. An environmental sensor obtains one or more parameters of external agents of the vehicle. A processor of the vehicle obtains a route having a destination at the autonomous vehicle, builds a Markov state model of the route that includes a plurality of states for the autonomous vehicle and one or more parameters of the external agents, generates a plurality of driving policies for navigating the route, selects a policy for navigating the route from the plurality of driving policies using a Markov Decision Process, and executes the selected policy at the autonomous vehicle to navigate the vehicle along the route towards the destination.

Claims

1. A method of operating an autonomous vehicle, comprising:

obtaining a route having a destination at the autonomous vehicle;

building a Markov state model representing a plurality of states for the autonomous vehicle;

generating a plurality of driving policies for navigating the route from the Markov state model, wherein each driving policy includes a progression through the plurality of states, wherein generating a driving policy includes combining a plurality of sub-policies from a database in an ordered progression;

selecting a policy for navigating the route from the plurality of driving policies using a Markov Decision Process; and

executing the selected policy at the autonomous vehicle to navigate the vehicle along the route towards the destination.

2. The method of claim 1, further comprising updating a state of the vehicle as the vehicle navigates the route and determining an updated policy based on the updated state of the vehicle.

3. The method of claim 1, wherein executing the selected policy at the autonomous vehicle further comprises performing an action at the vehicle to advance the vehicle towards a destination of the route according to the policy.

4. The method of claim 1, further comprising:

determining an expected reward value for each of the plurality of driving policies, wherein the reward value of a driving policy indicates an effectiveness of the driving policy in advancing the autonomous vehicle towards the destination; and

selecting the policy having a selected reward value.

5. The method of claim 4, wherein the selected policy further comprises the policy having a greatest expected reward value with no penalties.

6. The method of claim 1, wherein a state of the vehicle includes at least one of a pose vector, a kinematics vector, environmental information vector, and an actuator state of the autonomous vehicle.

7. The method of claim 6, wherein the environmental information vector includes parameters of external agents and traffic regulation devices in the environment.

8. A system for operating an autonomous vehicle, comprising:

an environmental sensor configured to obtain one or more parameters of external agents of the vehicle;

a database of pre-determined sub-policies; and

a processor configured to;

obtain a route having a destination at the autonomous vehicle;

build a Markov state model representing a plurality of states for the autonomous vehicle and the one or more parameters of the external agents;

generate a plurality of driving policies for navigating the route from the Markov state model, wherein each driving policy includes a progression through the plurality of states, wherein generating a driving policy includes combining a plurality of sub-policies from the database in an ordered progression;

select a policy for navigating the route from the plurality of driving policies using a Markov Decision Process; and

execute the selected policy at the autonomous vehicle to navigate the vehicle along the route towards the destination.

9. The system of claim 8, wherein the processor is further configured to update a state of the vehicle as the vehicle navigates the route and determine an updated policy based on the updated state of the vehicle.

10. The system of claim 8, wherein the processor is further configured to execute the selected policy at the autonomous vehicle by performing an action at the vehicle to advance the autonomous vehicle towards the destination.

11. The system of claim 8, wherein the processor is further configured to:

determine an expected reward value for each of the plurality of driving policies, wherein the reward value of a driving policy indicates an effectiveness of the driving policy in advancing the autonomous vehicle towards the destination; and

select the policy having a selected reward value.

12. The system of claim 11, wherein the selected policy further comprises the policy having a greatest expected reward value with no penalties.

13. The system of claim 8, wherein a state of the vehicle includes at least one of a pose vector, a kinematics vector, environmental information vector, and an actuator state of the autonomous vehicle.

14. The system of claim 13, wherein the environmental information vector includes parameters of external agents and traffic regulation devices in the environment.

15. An autonomous vehicle, comprising:

an environmental sensor configured to obtain one or more parameters of external agents of the vehicle;

a database of pre-determined sub-policies; and

a processor configured to;

obtain a route having a destination at the autonomous vehicle;

build a Markov state model representing a plurality of states for the autonomous vehicle and the one or more parameters of the external agents;

generate a plurality of driving policies for navigating the route from the Markov state model, wherein each driving policy includes a progression through the plurality of states, wherein generating a driving policy includes combining a plurality of sub-policies from the database in an ordered progression;

select a policy for navigating the route from the plurality of driving policies using a Markov Decision Process; and

execute the selected policy at the autonomous vehicle to navigate the vehicle along the route towards the destination.

16. The autonomous vehicle of claim 15, wherein the processor is further configured to update a state of the vehicle as the vehicle navigates the route and determine an updated policy based on the updated state of the vehicle.

17. The autonomous vehicle of claim 15, wherein a state of the vehicle includes at least one of a pose vector, a kinematics vector, environmental information vector, and an actuator state of the autonomous vehicle.

18. The autonomous vehicle of claim 15, wherein the environmental information vector includes parameters of external agents and traffic regulation devices in the environment.

19. The autonomous vehicle of claim 15, wherein the processor is further configured to:

determine an expected reward value for each of the plurality of driving policies wherein the reward of a driving policy indicates an effectiveness of the driving policy in advancing the autonomous vehicle towards the destination; and

select the policy having a selected reward value.

20. The autonomous vehicle of claim 19, wherein the selected policy further comprises the policy having a greatest expected reward with no penalties.