System and method for human operator intervention in autonomous vehicle operations

• US 9,488,979 B1
• Filed: 04/14/2015
• Issued: 11/08/2016
• Est. Priority Date: 04/14/2015
• Status: Active Grant

• Current Assignee: Zipline International Inc.
• Original Assignee: Zipline International Inc.
• Inventors: Chambers, Andrew, Wyrobek, Keenan, Rinaudo, Keller, Oksenhorn, Ryan, Hetzler, William
• Primary Examiner(s): Shafi, Muhammad
• Application Number:
  US14/686,698
• Time in Patent Office: 574 Days
• Field of Search: 701/3, 701/4, 701/5, 701/6, 701/7, 701/8, 701/10, 701/15, 701/16, 701/18, 701/24, 701/25, 701/26, 701/27, 701/38, 701/2, 701/23, 701/31.7, 701/468, 701/408, 244/23.A, 244/23.B, 244/36, 244/211, 244/214, 244/215, 244/56, 244/179, 244/180, 244/181, 244/80, 244/90.R, 244/90.A, 244/17.13, 244/17.23, 244/17.11, 244/177, 244/183, 340/963, 340967-970, 340/973, 340/974, 340/975, 340/977, 340/978, 342/38, 342/63, 342/176, 342/357.2, 342/357.33, 342/357.36, 342/462
• US Class Current: 1/1
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/1064 specially adapted for avoiding collisions with other aircraft

Abstract

An autonomous vehicle system is configured to receive vehicle commands from one or more parties and to execute those vehicle commands in a way that prevents the execution of stale commands. The autonomous vehicle system includes a finite state machine and a command counter or stored vehicle timestamp, which are used to help reject invalid or stale vehicle commands.

Claims

1. A method for piloting an autonomous vehicle, comprising:

receiving, at a computer server, an assistance request associated with the autonomous vehicle;

selecting, at the computer server, a remote vehicle operator from a group of remote vehicle operators, the remote vehicle operator associated with a control device;

receiving, at the computer server, a situation report associated with the autonomous vehicle;

generating, at the computer server, an operator message based at least in part on the situation report;

sending, from the computer server, the operator message to the control device;

receiving, at the computer server, at least one operator command from the control device; and

sending, from the computer server, at least one vehicle command to the autonomous vehicle based at least in part on the at least one operator command, the at least one vehicle command including a command verification value, wherein;

the at least one vehicle command causes the autonomous vehicle to change from a first path to a second path if a comparison of the command verification value to a stored command verification value of the autonomous vehicle indicates that the command verification value is valid; and

the at least one vehicle command does not cause the autonomous vehicle to change from the first path to the second path if the comparison of the command verification value to the stored command verification value of the autonomous vehicle indicates that the command verification value is not valid.

2. The method of claim 1, further comprising:

receiving, at the computer server, a status message from the autonomous vehicle, the status message including the command verification value.

3. The method of claim 2, wherein:

the stored command verification value of the autonomous vehicle is a command counter indicating a number of commands executed by the autonomous vehicle; and

the autonomous vehicle is configured to change from the first path to the second path only if the command verification value in the at least one vehicle command is equal to the command counter at a time that the at least one vehicle command is received.

4. The method of claim 1, further comprising:

receiving, at the computer server, a status message from the autonomous vehicle, the status message including a vehicle timestamp value indicating a time that a last command executed by the autonomous vehicle was issued;

wherein the command verification value included in the at least one vehicle command is a command timestamp, and the command timestamp indicates a time when the at least one vehicle command was issued; and

wherein the autonomous vehicle is configured to change from the first path to the second path only if the command timestamp in the at least one vehicle command is later than the vehicle timestamp value at a time that the at least one vehicle command is received.

5. The method of claim 1, wherein selecting the remote vehicle operator from the plurality of remote vehicle operators further comprises selecting the remote vehicle operator based on at least one of:

a measure of a quality of a communication link between the control device associated with the remote vehicle operator and the autonomous vehicle;

a score associated with the remote vehicle operator; and

a price associated with the remote vehicle operator.

6. The method of claim 1, wherein the operator message includes information useable by the control device to display, to the remote vehicle operator, a representation of an environment around the autonomous vehicle.

7. The method of claim 1, wherein:

the autonomous vehicle comprises a finite state machine;

the finite state machine comprises a plurality of vehicle states, at least one of which is active at a given time during operation of the autonomous vehicle; and

the at least one vehicle command is configured to transition the finite state machine from an active vehicle state to a second vehicle state.

8. The method of claim 7, wherein the finite state machine further comprises a plurality of valid state transitions, the method further comprising, at the autonomous vehicle:

in response to the at least one vehicle command, transitioning the finite state machine from the active vehicle state to the second vehicle state only if a transition from the active vehicle state to the second vehicle state is one of the plurality of valid state transitions.

9. The method of claim 8, wherein:

the at least one vehicle command comprises a command timestamp, the command timestamp indicating a time when the at least one vehicle command was issued;

the autonomous vehicle comprises a stored vehicle timestamp, the stored vehicle timestamp indicating a time when a vehicle command that was previously executed by the autonomous vehicle was issued; and

transitioning the finite state machine from the active vehicle state to the second vehicle state comprises transitioning the finite state machine from the active vehicle state to the second vehicle state only if the command timestamp of the vehicle command is later than the stored vehicle timestamp.

10. The method of claim 9, further comprising, at the autonomous vehicle, sending a status message toward the computer server, the status message including the value of the stored vehicle timestamp.

11. The method of claim 8, further comprising, at the autonomous vehicle:

storing a command count value, the stored command count value indicating an amount of previous vehicle commands executed by the autonomous vehicle;

receiving the at least one vehicle command including the command verification value; and

transitioning the finite state machine from the active vehicle state to the second vehicle state only if the command verification value is equal to the stored command count value.

12. The method of claim 11, further comprising, at the autonomous vehicle, sending a status message toward the computer server, the status message including the value of the stored command count value.

13. The method of claim 12, wherein:

the command verification value corresponds to a previous command count value received by the computer server in a status message from the autonomous vehicle, and

the status message is received prior to the sending of the at least one vehicle command.

14. An autonomous vehicle comprising a non-transitory computer-readable storage medium storing a representation of a plurality of vehicle states and a plurality of valid state transitions, a mission planner module, a flight controller module, and a communications module, the autonomous vehicle configured to:

receive a vehicle command using the communications module;

verify a validity of the vehicle command;

determine, based on the vehicle command, a new vehicle state that may be activated, using the mission planner module;

determine if a transition from a currently active vehicle state to the new vehicle state is one of the valid state transitions, using the mission planner;

in response to determining that the transition from the currently active vehicle state to the new vehicle state is one of the valid state transitions;

activate the new vehicle state; and

change from a first path of travel to a second path of travel based on the new vehicle state, using the flight controller; and

in response to determining that the transition from the currently active vehicle state to the new vehicle state is not one of the valid state transitions, not activating the new vehicle state.

15. The autonomous vehicle of claim 14, wherein:

the autonomous vehicle further comprises a stored command count value;

the vehicle command further comprises a command verification value;

the operation of verifying the validity of the vehicle command comprises determining if the command verification value is equal to the stored command count value; and

the autonomous vehicle activates the new vehicle state only if the command verification value is equal to the stored command count value.

16. The autonomous vehicle of claim 14, wherein:

the autonomous vehicle further comprises a stored vehicle timestamp;

the vehicle command further comprises a command timestamp;

the operation of verifying the validity of the vehicle command comprises determining if a time indicated by the command timestamp is later than a time indicated by the stored vehicle timestamp; and

the autonomous vehicle activates the new vehicle state only if the time indicated by the command timestamp is later than the time indicated by the stored vehicle timestamp.

17. A method for piloting an autonomous vehicle, comprising:

storing a first command verification value in a computer memory of an autonomous vehicle;

broadcasting, from a communications system of the autonomous vehicle, a status message comprising the first command verification value;

receiving a vehicle command via the communications system, the vehicle command comprising a second command verification value;

determining, using a mission planner of the autonomous vehicle, if a comparison of the second command verification value to the first command verification value indicates that the second command verification value is valid;

responsive to determining that the second command verification value is valid, changing the autonomous vehicle from a first path to a second path, using a flight controller; and

responsive to determining that the second command verification value is not valid, not changing the autonomous vehicle from the first path to the second path.

18. The method of claim 17, wherein:

the autonomous vehicle comprises a finite state machine;

the finite state machine comprises a plurality of vehicle states, at least one of which is active at a given time during operation of the autonomous vehicle; and

changing the autonomous vehicle from the first path to the second path further comprises transitioning the finite state machine from an active vehicle state to a second vehicle state.

19. The method of claim 18, wherein:

the first command verification value comprises a command counter indicating the number of commands executed by the autonomous vehicle; and

determining if the second command verification value is valid comprises checking if the first command verification value equals the second command verification value.

20. The method of claim 18, wherein:

the first command verification value comprises a stored vehicle timestamp indicating a time a last command executed by the autonomous vehicle was issued;

the second command verification value comprises a command timestamp indicating a time that the vehicle command was issued; and

determining if the second command verification value is valid comprises checking if the second command verification value is later than the first command verification value.