Interactive autonomous vehicle command controller

• US 10,248,119 B2
• Filed: 11/05/2015
• Issued: 04/02/2019
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Kentley-Klay, Timothy David, Gamara, Rachad Youssef
• Primary Examiner(s): Rink, Ryan
• Application Number:
  US14/933,706
• Publication Number:
  US 20170123422A1
• Time in Patent Office: 1,244 Days
• Field of Search: None
• CPC Class Codes: B60L 15/20 for control of the vehicle or its driving motor to achieve a desired performance, e.g. speed, torque, programmed variation of speed
• CPC Class Codes: B60L 2260/32 Auto pilot mode
• CPC Class Codes: B60Q 1/28 for indicating front of vehicle
• CPC Class Codes: B60Q 1/30 for indicating rear of vehicle, e.g. by means of reflecting surfaces
• CPC Class Codes: B60Q 3/00 Arrangement of lighting devices for vehicle interiors; Lighting devices specially adapted for vehicle interiors
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: Y02T 10/64 Electric machine technologies in electromobility
• CPC Class Codes: Y02T 10/72 Electric energy management in electromobility
• CPC Class Codes: Y02T 90/16 Information or communication technologies improving the operation of electric vehicles

Abstract

Various embodiments relate generally to autonomous vehicles and associated mechanical, electrical and electronic hardware, computer software and systems, and wired and wireless network communications to provide an autonomous vehicle fleet as a service. In particular, a method may include receiving, at an autonomous vehicle system, a command to control an ambient feature associated with the autonomous vehicle system. One or more courses of action may be determined based on the command. In addition, one or more probabilistic models associated with the one or more courses of action may also be determined. Based on the one or more probabilistic models, confidence levels may also be determined to form a subset of the one or more courses of action. A course of action from the subset of the one or more courses of action may then be executed at the autonomous vehicle system responsive to the command.

Claims

1. A method comprising:

receiving, from a sensor disposed in an autonomous vehicle, a signal corresponding to an audible request to control a route of the autonomous vehicle;

extracting one or more words from the signal;

determining one or more generative probabilistic models for interpreting the one or more words;

determining, based at least in part on the one or more generative probabilistic models, a probability associated with a first word of the one or more words;

determining that the probability is equal to or below a threshold value;

sending, based at least in part on the probability being equal to or below the threshold value, the signal to a teleoperator system;

receiving, from the teleoperator system, a command to control the route of the autonomous vehicle system in accordance with the request;

determining a plurality of courses of action based on the command;

for each of the plurality of courses of action, determining at least one second probabilistic model;

based on the second probabilistic model, determining a confidence level for each of the plurality of courses of action;

forming a subset of the plurality of courses of action, each course of action in the subset having a confidence level of at least a threshold confidence level; and

executing a course of action from the subset at the autonomous vehicle system.

2. The method of claim 1, wherein the sensor is a microphone mounted in the autonomous vehicle system.

3. The method of claim 2, further comprising:

responsive to the command, generating an audio response to the command based on the one or more probabilistic models; and

providing the response through one or more interior speakers mounted within the autonomous vehicle system.

14. The method of claim 1, wherein the interpreting the one or more words comprises an initial guess that converges toward an optimal response through a series of step-wise increments, using the one or more generative probabilistic models.

15. The method of claim 1, further comprising:

based at least in part on the probability, determining a response to the audible request using the one or more generative probabilistic models; and

outputting the response to a passenger of the autonomous vehicle.

20. The method of claim 1, further comprising:

receiving, from the sensor, a second signal corresponding to a second audible request to control the route of the autonomous vehicle;

extracting one or more second words from the second signal;

determining, based at least in part on the one or more generative probabilistic models, a second probability associated with a second word of the one or more second words;

determining that the second probability meets or exceeds the threshold value;

determining, based at least in part on the second word, a second command;

determining a second plurality of courses of action based on the second command;

determining, based on the second probabilistic model, a second confidence level for each of the second plurality of courses of action;

forming a second subset of the second plurality of courses of action, each second course of action in the second subset having a second confidence level of at least the threshold confidence level; and

executing a second course of action from the second subset at the autonomous vehicle system.

4. A method comprising:

receiving, from a first sensor of an autonomous vehicle system comprising a plurality of sensors, a signal corresponding to an audible request to change a route associated with the autonomous vehicle system;

determining one or more generative probabilistic models for interpreting at least one word in the signal;

determining, based at least in part on the one or more generative probabilistic models, a probability associated with a first word of the at least one word;

sending, based at least in part on the probability being equal to or below a threshold value, the signal to a teleoperator system;

receiving, from the teleoperator system, a command to alter the route of the autonomous vehicle system;

determining courses of action based on the command and data received from the plurality of sensors;

determining, for each of the courses of action, a second probabilistic model;

determining a confidence level for each of the courses of action, based at least in part on the second probabilistic model; and

executing, based at least in part on the confidence levels, a course of action from the courses of action at the autonomous vehicle system responsive to the command.

5. The method of claim 4, wherein the course of action comprises an instruction to contact the teleoperator system to respond to the command, the method further comprising:

sending the command to the teleoperator system;

receiving a second course of action from the teleoperator system; and

executing the second course of action.

6. The method of claim 4, further comprising selecting the course of action from the one or more courses of action based at least in part on the confidence levels.

7. The method of claim 6, wherein the course of action has an associated confidence level higher than a predetermined threshold level.

8. The method of claim 6, wherein the selecting the course of action comprises selecting the course of action from a subset of the one or more courses of action, each of the courses of action in the subset having a confidence level higher than a predetermined threshold level.

9. The method of claim 4, wherein the confidence levels are determined based at least in part on at least one of data from the sensors, log data, or information provided by an autonomous vehicle management system.

16. The method of claim 4, wherein the determining the probability comprises an initial guess that converges toward an optimal response through a series of step-wise increments, using the one or more generative probabilistic models.

17. The method of claim 4, further comprising:

sending the signal to a teleoperator system, the teleoperator system configured to generate the at least one word based, at least in part, on the signal and generate the command.

10. A system comprising:

an autonomous vehicle configured to drive autonomously on a roadway, the autonomous vehicle including one or more sensors for sensing one or more objects in an environment of the autonomous vehicle; and

a computing system communicatively coupled to the autonomous vehicle to receive data and to transmit instructions to control the autonomous vehicle, the computing system being programmed to;

control the autonomous vehicle to transport a user to a first destination along a trajectory;

receive a signal corresponding to an audible request from a user to transport the user to a second destination;

extract one or more words from the signal;

determine one or more generative probabilistic models for interpreting the one or more words;

determine, based at least in part on the one or more generative probabilistic models, a probability associated with a first word of the one or more words;

determine that the probability is equal to or below a threshold value;

sending, based at least in part on the probability being equal to or below the threshold value, the one or more words to a teleoperator system;

receive, from the teleoperator system, a command to alter the route of the autonomous vehicle system;

determine a plurality of courses of action in response to the command;

determine a confidence level for each of the courses of action; and

control the autonomous vehicle to execute, based at least in part on the confidence levels, a course of action of the courses of action.

11. The system of claim 10, the computing system being further programmed to:

determine a probabilistic model for each of the courses of action, wherein the confidence level is determined based at least in part on the probabilistic models.

12. The system of claim 10, wherein the confidence level of the course of action is higher than a predetermined threshold level.

13. The system of claim 10, wherein the confidence levels are determined based at least in part on data from the sensors.

18. The system of claim 10, wherein the computing system is programmed to determine the probability by generating an initial guess that converges toward an optimal response through a series of step-wise increments, using the one or more generative probabilistic models.

19. The system of claim 10, wherein the computing system is further programmed to:

determine a response to the audible request using the one or more generative probabilistic models; and

outputting the response to a passenger of the autonomous vehicle.