Cyclist hand signal detection by an autonomous vehicle

• US 9,014,905 B1
• Filed: 01/28/2014
• Issued: 04/21/2015
• Est. Priority Date: 01/28/2014
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Kretzschmar, Henrik
• Primary Examiner(s): Cheung, Mary
• Assistant Examiner(s): MAZZARA, ANNE MARIE
• Application Number:
  US14/166,502
• Time in Patent Office: 448 Days
• Field of Search: 701/28
• US Class Current: 701/28
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 30/143 Speed control B60W30/16 takes precedence
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Methods and systems for detecting hand signals of a cyclist by an autonomous vehicle are described. An example method may involve a computing device receiving a plurality of data points corresponding to an environment of an autonomous vehicle. The computing device may then determine one or more subsets of data points from the plurality of data points indicative of at least a body region of a cyclist. Further, based on an output of a comparison of the one or more subsets with one or more predetermined sets of cycling signals, the computing device may determine an expected adjustment of one or more of a speed of the cyclist and a direction of movement of the cyclist. Still further, based on the expected adjustment, the computing device may provide instructions to adjust one or more of a speed of the autonomous vehicle and a direction of movement of the autonomous vehicle.

Claims

1. A method, comprising:

a computing device receiving a plurality of data points corresponding to an environment of an autonomous vehicle;

the computing device determining one or more subsets of data points from the plurality of data points, wherein the one or more subsets of data points are indicative of at least a body region of a cyclist, wherein the body region of the cyclist comprises an upper-body region of the cyclist including at least one arm of the cyclist;

based on an output of a comparison of the one or more subsets with one or more predetermined sets of cycling signals, the computing device determining an expected adjustment of one or more of a speed of the cyclist and a direction of movement of the cyclist; and

based on the expected adjustment, the computing device providing instructions to adjust one or more of a speed of the autonomous vehicle and a direction of movement of the autonomous vehicle.

2. The method of claim 1, wherein the one or more predetermined sets of cycling signals are indicative of predefined situational contexts of other autonomous vehicles, wherein the predefined situational contexts include:

an environment of an autonomous vehicle which includes a cyclist, an environment of an autonomous vehicle which includes a cyclist providing a left turn hand signal, an environment of an autonomous vehicle which includes a cyclist providing a right turn hand signal, and an environment of an autonomous vehicle which includes a cyclist providing a stop hand signal.

3. The method of claim 1, wherein determining the expected adjustment based on the output of the comparison of the one or more subsets with the one or more predetermined sets of cycling signals comprises:

based on the output of the comparison of the one or more subsets with the one or more predetermined sets of cycling signals, determining a probability distribution of possible hand signals associated with the one or more subsets,

identifying one or more candidate hand signals which exceed a probability threshold from the probability distribution of possible hand signals,

selecting a hand signal from the one or more candidate hand signals, and

determining the expected adjustment of one or more of the speed of the cyclist and the direction of movement of the cyclist based on the selected hand signal.

4. The method of claim 3, further comprising:

determining one or more parameters of the cyclist, wherein the one or more parameters include one or more of;

a current speed of the cyclist, a current acceleration of the cyclist, and a current direction of movement of the cyclist,

wherein determining the expected adjustment of one or more of the speed of the cyclist and the direction of movement of the cyclist is further based on the one or more parameters of the cyclist.

5. The method of claim 1, wherein determining the one or more subsets indicative of at least the body region of the cyclist is based on a comparison of a point density of the plurality of data points with point densities associated with data points for the one or more predetermined sets of cycling signals.

6. The method of claim 1, further comprising:

determining one or more subsets indicative of a type of vehicle of the cyclist,

wherein determining the expected adjustment of one or more of the speed of the cyclist and the direction of movement of the cyclist is further based on the type of vehicle of the cyclist.

7. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving a plurality of data points corresponding to an environment of an autonomous vehicle;

determining one or more subsets of data points from the plurality of data points, wherein the one or more subsets of data points are indicative of at least a body region of a cyclist wherein the body region of the cyclist comprises an upper-body region of the cyclist including at least one arm of the cyclist;

based on an output of a comparison of the one or more subsets with one or more predetermined sets of cycling signals, determining an expected adjustment of one or more of a speed of the cyclist and a direction of movement of the cyclist; and

based on the expected adjustment, providing instructions to adjust one or more of a speed of the autonomous vehicle and a direction of movement of the autonomous vehicle.

8. The non-transitory computer readable medium of claim 7, wherein the computing device is a first computing device, the functions further comprising:

receiving the one or more predetermined sets of cycling signals, wherein the one or more predetermined sets of cycling signals are associated with at least one other cyclist previously identified by a second computing device;

determining one or more features of the one or more predetermined sets of cycling signals, wherein the one or more features are associated with a physical structure of the at least one other cyclist; and

determining clusters of the one or more predetermined sets of cycling signals based on the one or more features.

9. The non-transitory computer readable medium of claim 8, wherein the comparison of the one or more subsets with the one or more predetermined sets of cycling signals comprises:

determining a physical structure of the cyclist that includes the body region of the cyclist,

making a comparison of the one or more subsets with the clusters of the one or more predetermined sets of cycling signals based on the determined one or more features of the one or more subsets, and

based on an output of the comparison, determining a given label for the plurality of data points indicative of a situational context of the autonomous vehicle.

10. The non-transitory computer readable medium of claim 9, wherein the situational context of the autonomous vehicle includes one of:

the environment of the autonomous vehicle includes the cyclist, the environment of the autonomous vehicle includes the cyclist providing a left turn hand signal, the environment of the autonomous vehicle includes the cyclist providing a right turn hand signal, or the environment of the autonomous vehicle includes the cyclist providing a stop hand signal.

11. The non-transitory computer readable medium of claim 7, wherein the computing device is a first computing device, wherein the one or more predetermined sets of cycling signals include representations of respective distances from a left hand of one or more other cyclists previously identified by a second computing device to a head of the one or more other cyclists and further include representations of respective distances from a right hand of the one or more other cyclists to the head of the one or more other cyclists, the functions further comprising:

based on the one or more subsets, identifying a head of the cyclist, a left hand of the cyclist, and a right hand of the cyclist; and

determining a first distance from the left hand of the cyclist to the head of the cyclist and a second distance from the right hand of the cyclist to the head of the cyclist,

wherein the output of the comparison of the one or more subsets with the one or more predetermined sets of cycling signals is based on the first distance and the second distance.

12. The non-transitory computer readable medium of claim 7, wherein the computing device is a first computing device, wherein the one or more predetermined sets of cycling signals include respective heights of one or more other cyclists previously identified by a second computing device, and wherein determining the one or more subsets of data points from the plurality of data points comprises:

based on the plurality of data points, detecting an object in the environment of the autonomous vehicle that represents a candidate cyclist,

determining a height of the object,

making a comparison of the height of the object with the respective heights of the one or more predetermined sets of cycling signals,

based on an output of the comparison, determining that the object is a given cyclist, and

from the plurality of data points, determining one or more subsets of data points representative of the given cyclist to be the one or more subsets of data points that are indicative of at least the body region of the cyclist.

13. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising;

receiving a plurality of data points corresponding to an environment of an autonomous vehicle,

determining one or more subsets of data points from the plurality of data points, wherein the one or more subsets of data points are indicative of at least a body region of a cyclist, wherein the body region of the cyclist comprises an upper-body region of the cyclist including at least one arm of the cyclist,

based on an output of a comparison of the one or more subsets with one or more predetermined sets of cycling signals, determining an expected adjustment of one or more of a speed of the cyclist and a direction of movement of the cyclist, and

based on the expected adjustment, providing instructions to adjust one or more of a speed of the autonomous vehicle and a direction of movement of the autonomous vehicle.

14. The system of claim 13, wherein determining the expected adjustment based on the output of the comparison of the one or more subsets with the one or more predetermined sets of cycling signals comprises:

based on the output of the comparison of the one or more subsets with the one or more predetermined sets of cycling signals, determining a probability distribution of possible hand signals associated with the one or more subsets,

identifying one or more candidate hand signals which exceed a probability threshold from the probability distribution of possible hand signals,

selecting a hand signal from the one or more candidate hand signals, and

determining the expected adjustment of one or more of the speed of the cyclist and the direction of movement of the cyclist based on the selected hand signal.

15. The system of claim 14, further comprising:

an object tracing module, wherein the object tracing module is configured to determine one or more parameters of the cyclist, wherein the one or more parameters include one or more of;

a current speed of the cyclist, a current acceleration of the cyclist, and a current direction of travel of the cyclist, and wherein determining the expected adjustment of one or more of the speed of the cyclist and the direction of movement of the cyclist is further based on the one or more parameters of the cyclist.

16. The system of claim 13, wherein the upper-body region of the cyclist further includes at least a portion of a torso of the cyclist and at least a portion of a head of the cyclist.

17. The system of claim 14, wherein the one or more predetermined sets of cycling signals are indicative of predefined situational contexts of other autonomous vehicles, wherein the predefined situational contexts include:

an environment of an autonomous vehicle which includes a cyclist, an environment of an autonomous vehicle which includes a cyclist providing a left turn hand signal, an environment of an autonomous vehicle which includes a cyclist providing a right turn hand signal, and an environment of an autonomous vehicle which includes a cyclist providing a stop hand signal.

18. The system of claim 13, wherein the system is a first system, wherein the one or more predetermined sets of cycling signals include representations of respective distances from a left hand of one or more other cyclists previously identified by a second system to a head of the one or more other cyclists and further include representations of respective distances from a right hand of the one or more other cyclists to the head of the one or more other cyclists, the functions further comprising:

based on the one or more subsets, identifying a head of the cyclist, a left hand of the cyclist, and a right hand of the cyclist; and

determining a first distance from the left hand of the cyclist to the head of the cyclist and a second distance from the right hand of the cyclist to the head of the cyclist,

wherein the output of the comparison of the one or more subsets with the one or more predetermined sets of cycling signals is based on the first distance and the second distance.

19. The system of claim 13, further comprising:

a light detection and ranging (LIDAR) device coupled to the autonomous vehicle, wherein the LIDAR device is configured to provide LIDAR-based information comprising a three-dimensional (3D) point cloud that includes the plurality of data points, and wherein the plurality of data points are based on light emitted from the LIDAR device and reflected from the environment of the autonomous vehicle, the environment including the cyclist;

a camera coupled to the autonomous vehicle, wherein the camera is configured to provide one or more images of the cyclist; and

a radio detection and ranging (RADAR) device coupled to the autonomous vehicle, wherein the RADAR device is configured to provide RADAR-based information relating to at least one characteristic of the cyclist.