System and method for human operator intervention in autonomous vehicle operations

• US 9,910,432 B1
• Filed: 10/07/2016
• Issued: 03/06/2018
• Est. Priority Date: 04/14/2015
• Status: Active Grant

• Current Assignee: Zipline International Inc.
• Original Assignee: Zipline International Inc.
• Inventors: Chambers, Andrew, Wyrobek, Keenan, Rinaudo, Keller, Oksenhorn, Ryan, Hetzler, William
• Primary Examiner(s): Shafi, Muhammad
• Application Number:
  US15/288,733
• Time in Patent Office: 515 Days
• Field of Search: 701 3, 701 4, 701 5, 701 6, 701 7, 701 8, 701 10, 701 15, 701 16, 701 18, 701 24, 701 25, 701 26, 701 27, 701 38, 701 2, 244 23 A, 244 23 B, 244 36, 244211, 244214, 244215, 244 56, 244179, 244180, 244181, 244 80, 244 90 R, 244 90 A, 340963, 340967-970, 340973, 340974, 340975, 340977, 340978, 342 38, 342 63, 342176, 3423572, 34235733, 34235736, 342462
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/1064 specially adapted for avoiding collisions with other aircraft

Abstract

An autonomous vehicle system is configured to receive vehicle commands from one or more parties and to execute those vehicle commands in a way that prevents the execution of stale commands. The autonomous vehicle system includes a finite state machine and a command counter or stored vehicle timestamp, which are used to help reject invalid or stale vehicle commands.

Claims

1. A method for piloting an autonomous vehicle, comprising:

receiving, at a computer server, an assistance request associated with the autonomous vehicle;

selecting, at the computer server, a remote vehicle operator from a group of remote vehicle operators, the remote vehicle operator associated with a control device;

receiving, at the computer server, a situation report associated with the autonomous vehicle;

generating, at the computer server, an operator message based at least in part on the situation report;

sending, from the computer server, the operator message to the control device;

receiving, at the computer server, at least one operator command from the control device;

sending, from the computer server, at least one vehicle command to the autonomous vehicle based at least in part on the at least one operator command and including a command verification value; and

causing, with the at least one vehicle command, a finite state machine of the autonomous vehicle to transition from an active vehicle state to a second vehicle state after the autonomous vehicle determines that the command verification value is valid when compared to a stored command verification value of the autonomous vehicle,

wherein;

the finite state machine comprises a plurality of vehicle states, wherein at least one vehicle state is active at a given time during the operation of the autonomous vehicle.

2. The method of claim 1, further comprising,

receiving, at the computer server, a status message from the autonomous vehicle, the status message including the command verification value.

3. The method of claim 1, wherein:

the stored command verification value of the autonomous vehicle is a command counter indicating a number of commands executed by the autonomous vehicle; and

the finite state machine of the autonomous vehicle is configured to transition from the active vehicle state to the second vehicle state only if the command verification value is equal to the command counter.

4. The method of claim 1, wherein,

the command verification value is a command timestamp indicating a time when the at least one vehicle command was issued;

the stored command verification value is a stored vehicle time stamp value; and

the finite state machine of the autonomous vehicle is configured to change from the active vehicle state to the second vehicle state only if the command timestamp is later than the stored vehicle timestamp value.

5. The method of claim 1, wherein selecting the remote vehicle operator from the plurality of remote vehicle operators further comprises selecting the remote vehicle operator based on at least one of:

a measure of a quality of a communication link between the control device associated with the remote vehicle operator and the autonomous vehicle;

a score associated with the remote vehicle operator; and

a price associated with the remote vehicle operator.

6. The method of claim 1, wherein the operator message includes information useable by the control device to display, to the remote vehicle operator, a representation of an environment around the autonomous vehicle.

7. The method of claim 1, wherein the finite state machine further comprises a plurality of valid state transitions, the method further comprising, at the autonomous vehicle:

in response to the at least one vehicle command, transitioning the finite state machine from the active vehicle state to the second vehicle state only if a transition from the active vehicle state to the second vehicle state is one of the plurality of valid state transitions.

8. The method of claim 7, wherein:

the command verification value comprises a command timestamp, the command timestamp indicating a time when the at least one vehicle command was issued;

the stored command verification value comprises a stored vehicle timestamp, the stored vehicle timestamp indicating a time when a vehicle command that was previously executed by the autonomous vehicle was issued; and

transitioning the finite state machine from the active vehicle state to the second vehicle state comprises transitioning the finite state machine from the active vehicle state to the second vehicle state only if the command timestamp of the vehicle command is later than the stored vehicle timestamp.

9. The method of claim 8, further comprising, at the autonomous vehicle, sending a status message toward the computer server, the status message including the value of the stored vehicle timestamp.

10. The method of claim 7, further comprising, at the autonomous vehicle:

storing a command count value, the stored command count value indicating an amount of previous vehicle commands executed by the autonomous vehicle;

receiving the at least one vehicle command including the command verification value; and

transitioning the finite state machine from the active vehicle state to the second vehicle state only if the command verification value is equal to the stored command count value.

11. The method of claim 10, further comprising, at the autonomous vehicle, sending a status message toward the computer server, the status message including the value of the stored command count value.

12. The method of claim 11, wherein:

the command verification value corresponds to a previous command count value received by the computer server in a status message from the autonomous vehicle, and

the status message is received prior to the sending of the at least one vehicle command.

13. A method for piloting an autonomous vehicle, comprising:

storing a first command verification value in a computer memory of an autonomous vehicle, the autonomous vehicle comprising a finite state machine that comprises a plurality of vehicle states, wherein at least one vehicle state is active at a given time during the operation of the autonomous vehicle;

broadcasting, from a communications system of the autonomous vehicle, a status message comprising the first command verification value;

receiving a vehicle command via the communications system, the vehicle command comprising a second command verification value;

determining, using a mission planner of the autonomous vehicle, if a comparison of the second command verification value to the first command verification value indicates that the second command verification value is valid;

responsive to determining that the second command verification value is valid, transitioning the finite state machine of the autonomous vehicle from an active vehicle state to a second vehicle state; and

responsive to determining that the second command verification value is not valid, not changing the autonomous vehicle from the active vehicle state to the second vehicle state,

wherein the first command verification value comprises at least one of;

a command counter indicating a number of commands executed by the autonomous vehicle;

or

a stored vehicle timestamp indicating a time a last command executed by the autonomous vehicle was issued.

14. The method of claim 13, wherein:

the first command verification value comprises the command counter; and

determining if the second command verification value is valid comprises checking if the first command verification value equals the second command verification value.

15. The method of claim 13, wherein:

the first command verification value comprises the stored vehicle timestamp;

the second command verification value comprises a command timestamp indicating a time that the vehicle command was issued; and

determining if the second command verification value is valid comprises checking if the second command verification value is later than the first command verification value.