Multi-perspective system and method for behavioral policy selection by an autonomous agent

• US 10,564,641 B2
• Filed: 07/17/2019
• Issued: 02/18/2020
• Est. Priority Date: 07/20/2018
• Status: Active Grant

• Current Assignee: May Mobility, Inc.
• Original Assignee: May Mobility, Inc.
• Inventors: Vozar, Steve, Olson, Edwin, Voorheis, Tom
• Primary Examiner(s): Lee, Tyler J
• Application Number:
  US16/514,624
• Publication Number:
  US 20200026286A1
• Time in Patent Office: 216 Days
• CPC Class Codes: B60W 2554/402 Type
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/4045 Intention, e.g. lane change or imminent movement
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 60/0011 involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0116 from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/0145 for active traffic flow control
• CPC Class Codes: G08G 1/162 event-triggered
• CPC Class Codes: H04W 4/024 Guidance services
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/38 for collecting sensor information
• CPC Class Codes: H04W 4/40 for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/46 : for vehicle-to-vehicle communication [V2V]

Abstract

A system and a method for autonomous decisioning and operation by an autonomous agent includes: collecting decisioning data including: collecting a first stream of data includes observation data obtained by onboard sensors of the autonomous agent, wherein each of the onboard sensors is physically arranged on the autonomous agent; collecting a second stream of data includes observation data obtained by offboard infrastructure devices, the offboard infrastructure devices being arranged geographically remote from and in an operating environment of the autonomous agent; implementing a decisioning data buffer that includes the first stream of data from the onboard sensors and the second stream of data from the offboard sensors; generating current state data; generating/estimating intent data for each of one or more agents within the operating environment of the autonomous agent; identifying a plurality of candidate behavioral policies; and selecting and executing at least one of the plurality of candidate behavioral policies.

Claims

1. A system for intelligently implementing an autonomous agent, the system comprising:

a plurality of offboard infrastructure devices arranged geographically remote an autonomous agent and that;

collect observation data relating to circumstances surrounding a travel route of the autonomous agent;

the autonomous agent comprising;

a communication interface that enables the autonomous agent to communicate with each of the plurality of offboard infrastructure devices;

an onboard sensor suite comprising a plurality of distinct sensors arranged on the autonomous agent and that collect observation data relating to circumstances surrounding the autonomous agent from a perspective that is distinct a perspective of the plurality of offboard infrastructure devices;

a decisioning data buffer storing at least a first stream of observation data from the onboard sensor suite and a second stream of observation data from the plurality of offboard infrastructure devices;

one or more onboard computing devices that;

implements a synchronization module that synchronizes the first stream of observation data of the autonomous agent and the second stream of observation data from offboard infrastructure devices according to a common clock of the autonomous agent;

implements a tracking of one or more agents within the circumstances surrounding the autonomous agent based on the first stream of data and the second stream of data;

computes an intent estimation for each of the one or more agents based on the first stream of data and the second stream of data;

computes state data for each of the one or more agents and for the autonomous agent;

implements a multi-policy decision-making module that identifies a plurality of candidate behavioral policies for the autonomous agent based on the tracking and the intent estimation;

selects one of the plurality of candidate behavioral policies and executes the selected one of the plurality of candidate behavioral policies for controlling an autonomous operation of the autonomous agent.

2. The system according to claim 1, wherein the one or more onboard computing devices further implement:

a time synchronization module that synchronizes the first stream of observation data and the second stream of observation data to the autonomous agent by synchronizing;

the first stream of observation data to the autonomous agent based on a first computed communication latency between the onboard sensor suite and the autonomous agent;

the second stream of observation data to the autonomous agent based on a second computed communication latency between the plurality of offboard infrastructure devices and the autonomous agent.

3. The system according to claim 2, wherein the one or more onboard computing devices further:

rearrange a position of the first stream of observation data from a first position to a second position within the decisioning data buffer based on the synchronization; and

rearrange a position of the second stream of observation data from a first position to a second position within the decisioning data buffer based on the synchronization.

4. The system according to claim 1, wherein the one or more onboard computing devices:

in response to the synchronization of the first stream of observation data and the second stream of observation data, data from the first stream of observation data and data from the second stream of observation data are repositioned to a historical position within the decisioning data buffer that is associated with an earlier point in time relative to data that is positioned beyond the data from the first stream and the second stream of observation data.

5. The system according to claim 1, wherein the one or more onboard computing devices:

stores data obtained from each of the plurality of offboard infrastructure devices and data obtained from each of the plurality of distinct sensors of the onboard sensor suite within a distinct track of memory that is independent from other tracks of memory,

wherein an intent estimation for any agents identified within the data obtained from each of the plurality of offboard infrastructure devices and the data obtained from each of the plurality of distinct sensors is computed based on the distinct track of memory for the respective offboard infrastructure device or the respective distinct sensor.

6. The system according to claim 5, wherein:

the distinct track of memory for the data obtained from each of the plurality of offboard infrastructure devices and the data obtained from each of the plurality of distinct sensors is combined into a master track of memory;

the intent estimation for each of the one or more agents is based on the master track of memory.

7. The system according to claim 1, further comprising:

a remote autonomous agent service being implemented by a distributed network of computing devices and that is in operable communication with each of the autonomous agent and each of the plurality of offboard infrastructure devices, wherein the remote autonomous agent service computes one or more;

the tracking of one or more agents within the circumstances surrounding the autonomous agent based on the first stream of data and the second stream of data;

the intent estimation for each of the one or more agents based on the first stream of data and the second stream of data.

8. A method for autonomous decisioning and operation by an autonomous agent, the method comprising:

collecting decisioning data including;

collecting a first stream of data comprising observation data obtained by one or more onboard sensors of the autonomous agent, wherein each of the one or more onboard sensors is physically arranged on the autonomous agent;

collecting a second stream of data comprising observation data obtained by one or more offboard infrastructure devices, the one or more offboard infrastructure devices being arranged geographically remote from and in an operating environment of the autonomous agent, wherein;

the first stream of data includes data relating to sensed observations of circumstances surrounding the autonomous agent obtained from a perspective of the autonomous agent by each of the one or more onboard sensors;

the second stream of data includes data relating to sensed observations of circumstances within the operating environment of the autonomous agent obtained from an external perspective toward a route of the autonomous agent that is made by each of the offboard infrastructure devices;

implementing a decisioning data buffer that includes the first stream of data from the one or more onboard sensors and the second stream of data from the offboard sensors;

generating current state data;

generating or estimating intent data for each of one or more agents within the operating environment of the autonomous agent;

identifying a plurality of candidate behavioral policies; and

selecting and executing at least one of the plurality of candidate behavioral policies.

9. The method according to claim 8, wherein

the operating environment of the autonomous agent including a predetermined radius from a geographical position of the autonomous agent while operating along a structured or an unstructured route of the autonomous agent.

10. The method according to claim 8, wherein

the second stream of data includes data relating to an operating state of at least one of the one or more offboard infrastructure devices.

11. The method according to claim 8, wherein

in response to traveling into a communication proximity of at least one of the one or more offboard infrastructure devices, automatically collecting the second stream of data from the at least one or more offboard infrastructure devices within the communication proximity.

12. The method according to claim 8, wherein:

a field-of-sensing of the one or more offboard infrastructure devices comprises a geometrically defined region,

the one or more offboard infrastructure devices may be configured to sense or collect semantic abstractions of objects within the geometrically defined region.

13. The method according to claim 8, wherein:

the second stream of data from the offboard infrastructure devices includes semantically dense state data of a scene.

14. The method according to claim 8, wherein implementing the decisioning data buffer includes:

sequentially storing data received from the first stream of data and the second stream of data based on a time at which each of the first stream of data and the second stream of data was received.

15. The method according to claim 8, wherein implementing the decisioning data buffer further includes:

computing a global time synchronization between the first stream of data and the second stream of data based on timestamp data appended with the second stream of data provided by the one or more offboard infrastructure device,

wherein computing the global time synchronization includes computing a latency value for the second stream of data based on calculating a difference between a first timestamp data indicating a first time at which the second stream of data was obtained by the one or more offboard infrastructure devices and a second timestamp data indicating a second time at which the second stream of data was collected by the autonomous agent.

16. The method according to claim 15, wherein implementing the decisioning data buffer includes:

repositioning within the decisioning data buffer data from the second data stream based on the computed latency value,

wherein the repositioning includes moving the data from the second data stream from a first position to a second position within the decisioning data buffer that is earlier in historical time.

17. A method for autonomous decisioning and operation by an autonomous agent, the method comprising:

collecting decisioning data including;

collecting a first stream of data comprising observation data obtained by one or more onboard sensors of the autonomous agent, wherein each of the one or more onboard sensors is physically arranged on the autonomous agent;

collecting a second stream of data comprising observation data obtained by one or more offboard infrastructure devices, the one or more offboard infrastructure devices being arranged geographically remote from and in an operating environment of the autonomous agent, wherein;

the second stream of data from the offboard infrastructure devices includes semantically dense state data of a scene;

implementing a decisioning data buffer that includes the first stream of data from the one or more onboard sensors and the second stream of data from the offboard sensors;

generating current state data;

generating or estimating intent data for each of one or more agents within the operating environment of the autonomous agent;

identifying a plurality of candidate behavioral policies; and

selecting and executing at least one of the plurality of candidate behavioral policies.

18. The method according to claim 17, wherein implementing the decisioning data buffer further includes:

computing a global time synchronization between the first stream of data and the second stream of data based on timestamp data appended with the second stream of data provided by the one or more offboard infrastructure device,

wherein computing the global time synchronization includes computing a latency value for the second stream of data based on calculating a difference between a first timestamp data indicating a first time at which the second stream of data was obtained by the one or more offboard infrastructure devices and a second timestamp data indicating a second time at which the second stream of data was collected by the autonomous agent.