Systems and methods for rear signal identification using machine learning

• US 10,691,962 B2
• Filed: 09/22/2017
• Issued: 06/23/2020
• Est. Priority Date: 09/22/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Mei, Xue, Nagasaka, Naoki, Lee, Kuan-Hui, Prokhorov, Danil V.
• Primary Examiner(s): Melton, Todd M
• Application Number:
  US15/713,491
• Publication Number:
  US 20190092318A1
• Time in Patent Office: 1,005 Days
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

System, methods, and other embodiments described herein relate to identifying rear indicators of a nearby vehicle. In one embodiment, a method includes, in response to detecting a nearby vehicle, capturing signal images of a rear portion of the nearby vehicle. The method includes computing a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing the signal images according to a brake classifier. The method includes computing a turn state for rear turn signals of the nearby vehicle that indicates which of the turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier. The brake classifier and the turn classifier are comprised of a convolutional neural network and a long short-term memory recurrent neural network (LSTM-RNN). The method includes providing electronic outputs identifying the braking state and the turn state.

Claims

1. A signal identification system for identifying rear indicators of a nearby vehicle, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing;

a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to, in response to detecting the nearby vehicle, capturing signal images of a rear portion of the nearby vehicle; and

an indicator module including instructions that when executed by the one or more processors cause the one or more processors to;

i) compute a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing the signal images according to a brake classifier, and

ii) compute a turn state for rear turn signals of the nearby vehicle that indicates which of the rear turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier,

wherein the brake classifier and the turn classifier are each comprised of a combined network architecture including both a convolutional neural network (CNN) and a long short-term memory recurrent neural network (LSTM-RNN) configured in series with the LSTM-RNN accepting an input that is a final output of the CNN, and

wherein the indicator module includes instructions to provide electronic outputs identifying the braking state and the turn state and to control one or more vehicle systems of a host vehicle in response to the electronic outputs.

2. The signal identification system of claim 1, wherein the indicator module further includes instructions to:

generate the regions of interest from the signal images by

i) compensating for movement of the nearby vehicle between the signal images to produce flow images from the signal images,

ii) comparing the flow images to generate difference images that indicate areas of changed pixels in the flow images, and

iii) extracting the regions of interest from the difference images.

3. The signal identification system of claim 2, wherein the indicator module further includes instructions to compensate for the movement by computing the flow images using a scale invariant feature transformation (SIFT) flow algorithm to transform the signal images into the flow images, wherein the regions of interest include a region for a left turn indicator and a region for a right turn indicator of the nearby vehicle, and wherein each of the regions of interest are a separate series of difference images that isolate the left turn indicator and the right turn indicator.

4. The signal identification system of claim 1, wherein the indicator module further includes instructions to compute the braking state by:

i) extracting image spatial features of the nearby vehicle from the signal images by convolving the signal images into layered spatial features and pooling the layered spatial features over multiple layers of a braking convolutional neural network (CNN) of the brake classifier to generate the image spatial features as an electronic output at a fully connected layer of the braking CNN, and

ii) determining, using the image spatial features, temporal dependencies between the signal images that are indicative of the braking state by recurrently analyzing the image spatial features from the signal images according to a braking long short-term memory recurrent neural network (LSTM-RNN) of the brake classifier that indicates the braking state as a probability that the brake lights are presently active,

wherein the braking CNN and the braking LSTM-RNN are trained to identify the braking state.

5. The signal identification system of claim 1, wherein the indicator module further includes instructions to compute the turn state by:

i) extracting spatial features from the regions of interest by convolving the regions into layered spatial features and pooling the layered spatial features over multiple layers of a turn convolutional neural network (CNN) of the turn classifier to generate the spatial features as an electronic output at a fully connected layer of the turn CNN, and

ii) determining, using the spatial features and the regions of interest, temporal dependencies that are indicative of the turn state by recurrently analyzing the spatial features from the regions of interest according to a turn long short-term memory recurrent neural network (LSTM-RNN) of the turn classifier that iteratively analyzes the regions of interest in relation to a series of the signal images and indicates the turn state as a probability of a particular dynamic flashing state of the rear turn signals,

wherein the turn CNN and the turn LSTM-RNN are trained to identify the turn state.

6. The signal identification system of claim 1, wherein the monitoring module includes the instructions to capture the signal images including instructions to capture the signal images as a series over a defined period of time in order to capture temporal changes in the rear turn signals, and wherein the monitoring module further include instructions to detect the nearby vehicle including instructions to control at least a camera sensor to analyze acquire scan images of a surrounding environment and analyze the scan images for a presence of the nearby vehicle.

7. The signal identification system of claim 1, wherein controlling the one or more vehicle systems of the host vehicle by modifying operating parameters of the one or more vehicle systems in response to the braking state and the turn state.

8. The signal identification system of claim 1, wherein the signal identification system is embedded within an autonomous driving module of a host vehicle.

9. A non-transitory computer-readable medium storing for identifying rear indicators of a nearby vehicle and including instructions that when executed by one or more processors cause the one or more processors to:

compute a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing signal images according to a brake classifier, the signal images being captured of a rear portion of the nearby vehicle; and

compute a turn state for rear turn signals of the nearby vehicle that indicates which of the rear turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier,

wherein the brake classifier and the turn classifier are each comprised of a combined network architecture including both a convolutional neural network (CNN) and a long short-term memory recurrent neural network (LSTM-RNN) configured in series with the LSTM-RNN accepting an input that is a final output of the CNN;

provide electronic outputs identifying the braking state and the turn state; and

control one or more vehicle systems of a host vehicle in response to the electronic outputs.

10. The non-transitory computer-readable medium of claim 9, further comprising:

instructions to, in response to detecting the nearby vehicle, capture the signal images of a rear portion of the nearby vehicle.

11. The non-transitory computer-readable medium of claim 10, further comprising instructions to:

generate the regions of interest from the signal images by

i) compensating for movement of the nearby vehicle between the signal images to produce flow images from the signal images,

ii) comparing the flow images to generate difference images that indicate areas of changed pixels in the flow images, and

iii) extracting the regions of interest from the difference images.

12. The non-transitory computer-readable medium of claim 9, wherein the instructions to compute the braking state include instructions to:

i) extract image spatial features of the nearby vehicle from the signal images by convolving the signal images into layered spatial features and pooling the layered spatial features over multiple layers of a braking convolutional neural network (CNN) of the brake classifier to generate the image spatial features as an electronic output at a fully connected layer of the braking CNN, and

ii) determine, using the image spatial features, temporal dependencies between the signal images that are indicative of the braking state by recurrently analyzing the image spatial features from the signal images according to a braking long short-term memory recurrent neural network (LSTM-RNN) of the brake classifier that indicates the braking state as a probability that the brake lights are presently active,

wherein the braking CNN and the braking LSTM-RNN are trained to identify the braking state.

13. The non-transitory computer-readable medium of claim 9, wherein the instructions to compute the turn state include instructions to:

i) extract spatial features from the regions of interest by convolving the regions into layered spatial features and pooling the layered spatial features over multiple layers of a turn convolutional neural network (CNN) of the turn classifier to generate the spatial features as an electronic output at a fully connected layer of the turn CNN, and

ii) determine, using the spatial features and the regions of interest, temporal dependencies that are indicative of the turn state by recurrently analyzing the spatial features from the regions of interest according to a turn long short-term memory recurrent neural network (LSTM-RNN) of the turn classifier that iteratively analyzes the regions of interest in relation to a series of the signal images and indicates the turn state as a probability of a particular dynamic flashing state of the rear turn signals,

wherein the turn CNN and the turn LSTM-RNN are trained to identify the turn state.

14. A method of identifying rear indicators of a nearby vehicle, comprising:

in response to detecting the nearby vehicle, capturing signal images of a rear portion of the nearby vehicle;

computing a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing the signal images according to a brake classifier;

computing a turn state for rear turn signals of the nearby vehicle that indicates which of the rear turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier,

wherein the brake classifier and the turn classifier are each comprised of a combined network architecture including both a convolutional neural network (CNN) and a long short-term memory recurrent neural network (LSTM-RNN) configured in series with the LSTM-RNN accepting an input that is a final output of the CNN;

providing electronic outputs identifying the braking state and the turn state; and

controlling one or more vehicle systems of a host vehicle in response to the electronic outputs.

15. The method of claim 14, further comprising:

generating the regions of interest from the signal images by

i) compensating for movement of the nearby vehicle between the signal images to produce flow images from the signal images,

ii) comparing the flow images to generate difference images that indicate areas of changed pixels in the flow images, and

iii) extracting the regions of interest from the difference images.

16. The method of claim 15, wherein compensating for the movement includes computing the flow images using a scale invariant feature transformation (SIFT) flow algorithm to transform the signal images into the flow images, wherein the regions of interest include a region for a left turn indicator and a region for a right turn indicator of the nearby vehicle, and wherein each of the regions of interest are a separate series of difference images that isolate the left turn indicator and the right turn indicator.

17. The method of claim 14, wherein analyzing the signal images according to the brake classifier includes:

i) extracting image spatial features of the nearby vehicle from the signal images by convolving the signal images into layered spatial features and pooling the layered spatial features over multiple layers of a braking convolutional neural network (CNN) of the brake classifier to generate the image spatial features as an electronic output at a fully connected layer of the braking CNN, and

ii) determining, using the image spatial features, temporal dependencies between the signal images that are indicative of the braking state by recurrently analyzing the image spatial features from the signal images according to a braking long short-term memory recurrent neural network (LSTM-RNN) of the brake classifier that indicates the braking state as a probability that the brake lights are presently active,

wherein the braking CNN and the braking LSTM-RNN are trained to identify the braking state.

18. The method of claim 14, wherein analyzing the regions of interest according to the turn classifier includes:

i) extracting spatial features from the regions of interest by convolving the regions into layered spatial features and pooling the layered spatial features over multiple layers of a turn convolutional neural network (CNN) of the turn classifier to generate the spatial features as an electronic output at a fully connected layer of the turn CNN, and

ii) determining, using the spatial features and the regions of interest, temporal dependencies that are indicative of the turn state by recurrently analyzing the spatial features from the regions of interest according to a turn long short-term memory recurrent neural network (LSTM-RNN) of the turn classifier that iteratively analyzes the regions of interest in relation to a series of the signal images and indicates the turn state as a probability of a particular dynamic flashing state of the rear turn signals,

wherein the turn CNN and the turn LSTM-RNN are trained to identify the turn state.

19. The method of claim 14, wherein capturing the signal images includes capturing the signal images as a series over a defined period of time in order to capture temporal changes in the rear turn signals, and wherein detecting the nearby vehicle includes using at least a camera sensor to analyze a surrounding environment for a presence of the nearby vehicle.

20. The method of claim 14, wherein controlling the one or more vehicle systems of the host vehicle by modifying operating parameters of the one or more vehicle systems in response to the braking state and the turn state.