DEVICE AND A METHOD FOR PROCESSING DATA SEQUENCES USING A CONVOLUTIONAL NEURAL NETWORK

• US 20190325306A1
• Filed: 04/03/2019
• Published: 10/24/2019
• Est. Priority Date: 04/23/2018
• Status: Active Grant

• Current Assignee: Aptiv Technologies AG (Aptiv PLC)
• Original Assignee: Aptiv Technologies Limited (Aptiv PLC)
• Inventors: Zhu, Weimeng, Su, Yu, Nunn, Christian
• Granted Patent: US 11,521,059 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

A device for processing data sequences by means of a convolutional neural network is configured to carry out the following steps: receiving an input sequence comprising a plurality of data items captured over time using a sensor, each of said data items comprising a multi-dimensional representation of a scene, generating an output sequence representing the input sequence processed item-wise by the convolutional neural network, wherein generating the output sequence comprises: generating a grid-generation sequence based on a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the output sequence or the grid-generation sequence, generating a sampling grid on the basis of the grid-generation sequence, generating an intermediate output sequence by sampling from the past portion of the output sequence according to the sampling grid, and generating the output sequence based on a weighted combination of the intermediate output sequence and the input sequence.

Claims

1. A device for processing data sequences comprising a convolutional neural network, wherein

the device is configured to receive an input sequence (I_t) comprising a plurality of data items captured over time, each of said data items comprising a multi-dimensional representation of a scene,

the convolutional neural network is configured to generate an output sequence (h_t) representing the input sequence processed item-wise by the convolutional neural network,

the convolutional neural network comprises a sampling unit configured to generate an intermediate output sequence ({tilde over (h)}_t) by sampling from a past portion of the output sequence (h_t−

1) according to a sampling grid (G_t),

the convolutional neural network is configured to generate the sampling grid (G_t) item-wise on the basis of a grid-generation sequence, wherein the grid-generation sequence is based on a combination of the input sequence (I_t) and an intermediate grid-generation sequence representing a past portion of the output sequence (h<sub>t−

1</sub>) or the grid-generation sequence (C<sub>t−

1</sub>),

the convolutional neural network is configured to generate the output sequence (h_t) based on a weighted combination of the intermediate output sequence ({tilde over (h)}_t) and the input sequence (I_t).

2. The device according to claim 1,

wherein the grid-generation sequence is based on an item-wise combination of the input sequence (I_t) and the intermediate grid-generation sequence.

3. The device according to claim 1,

wherein the intermediate grid-generation sequence is formed by the past portion of the output sequence (h_t−

1)

or wherein the intermediate grid-generation sequence is formed by the past portion of the output sequence (h<sub>t−

1</sub>) processed with an inner convolutional neural network,

or wherein the intermediate grid-generation sequence (C<sub>t−

1</sub>) is formed by the past portion of the grid-generation sequence processed with an inner convolutional neural network.

4. The device according to claim 1,

wherein the convolutional neural network is configured to generate the sampling grid (G_t) by processing the grid-generation sequence with at least one inner convolutional neural network.

5. The device according to claim 1, wherein

the convolutional neural network is configured to generate the output sequence (h_t) by generating a first weighting sequence (f_t) and a second weighting sequence (i_t) based on one of

the input sequence (I_t),

the intermediate output sequence ({tilde over (h)}_t),

the intermediate grid-generation sequence (h_t−

1, C<sub>t−

1</sub>),

the grid-generation sequence processed by an inner convolutional network ({tilde over (C)}_t, C_t),

generating an intermediate input sequence by processing the input sequence (I_t) with an inner convolutional neural network (18),

weighting the intermediate output sequence ({tilde over (h)}_t) with the first weighting sequence (f_t),

weighting the intermediate input sequence with the second weighting sequence (i_t),

superimposing the weighted intermediate output sequence and the weighted intermediate input sequence.

6. The device according to claim 5, wherein

generating the first weighting sequence (f_t) and/or the second weighting sequence (i_t) comprises:

forming a combination of at least two of

the input sequence (I_t),

the intermediate output sequence ({tilde over (h)}_t),

the intermediate grid-generation sequence (h<sub>t−

1</sub>, C<sub>t−

1</sub>),

the grid-generation sequence processed by an inner convolutional network ({tilde over (C)}_t, C_t), and

forming a processed combination (i_t, f_t, z_t) by processing the combination with an inner convolutional neural network (22).

7. The device according to claim 6,

wherein one of the first weighting sequence or the second weighting sequence is formed by the processed combination and wherein the other of the first weighting sequence or the second weighting sequence is formed by the processed combination (z_t) subtracted from a constant.

8. The device according to one of claim 5, wherein the convolutional neural network is configured to generate the first weighting sequence (f_t) and the second weighting sequence (i_t) correspondingly.

9. The device according to claim 1,

wherein the sampling grid (G_t) comprises a plurality of sampling locations, each of the sampling locations being defined by a respective pair of an offset and one of a plurality of data points of a data item of the intermediate output sequence ({tilde over (h)}_t).

10. The device according to claim 1,

wherein each data item of the input sequence (I_t) comprises a plurality of data points, each data point representing a location in the scene and comprising a plurality of parameters, in particular coordinates, of the location.

11. The device according to claim 1, wherein each data item of the input sequence (I_t) is formed by an image comprising a plurality of pixels.

12. A system for processing data sequences, the system comprising a sensor for capturing a data sequence and a device according to one of the preceding claims.

13. The system according to claim 12, wherein the sensor comprises at least one of a radar sensor, a light-detection-and-ranging sensor, an ultrasonic sensor, or a camera.

14. A method for processing data sequences by means of a convolutional neural network, the method comprising:

receiving an input sequence (I_t) comprising a plurality of data items captured over time using a sensor, each of said data items comprising a multi-dimensional representation of a scene,

generating an output sequence (h_t) representing the input sequence processed item-wise by the convolutional neural network,

wherein generating the output sequence comprises;

generating a grid-generation sequence based on a combination of the input sequence (I_t) and an intermediate grid-generation sequence representing a past portion of the output sequence (h<sub>t−

1</sub>) or the grid-generation sequence (C<sub>t−

1</sub>),

generating a sampling grid (G_t) on the basis of the grid-generation sequence,

generating an intermediate output sequence ({tilde over (h)}_t) by sampling from the past portion of the output sequence (h<sub>t−

1</sub>) according to the sampling grid (G_t), and

generating the output sequence (h_t) based on a weighted combination of the intermediate output sequence ({tilde over (h)}_t) and the input sequence (I_t).