Object recognition from videos using recurrent neural networks

• US 10,013,640 B1
• Filed: 12/21/2015
• Issued: 07/03/2018
• Est. Priority Date: 12/21/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Angelova, Anelia, Bogun, Ivan
• Primary Examiner(s): Bhatnagar, Anand
• Application Number:
  US14/976,147
• Time in Patent Office: 925 Days
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for identifying an object from a video. One of the methods includes obtaining multiple frames from a video, where each frame of the multiple frames depicts an object to be recognized, and processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized.

Claims

1. A computer-implemented method, comprising:

obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and

processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,

wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,

wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames;

processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and

processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

2. The method of claim 1,

wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and

wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

3. The method of claim 2,

wherein the recurrent neural network further comprises one or more classification layers, and

wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

4. The method of claim 3,

wherein the recurrent neural network further comprises a backward LSTM layer, and

wherein processing the plurality of frames further comprises:

processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and

processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

5. The method of claim 1, wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

6. The method of claim 1, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

7. The method of claim 1, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

8. The method of claim 1, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

9. The method of claim 1, wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

10. The method of claim 1, wherein a count of the multiple frames is five or fewer.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and

processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,

wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,

wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames;

processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and

processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

12. The system of claim 11,

wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and

wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

13. The system of claim 12,

wherein the recurrent neural network further comprises one or more classification layers, and

wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

14. The system of claim 13,

wherein the recurrent neural network further comprises a backward LSTM layer, and

wherein processing the plurality of frames further comprises:

processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and

processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

15. The system of claim 11, wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

16. The system of claim 11, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

17. The system of claim 11, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

18. The system of claim 11, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

19. The system of claim 11, wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

20. The system of claim 11, wherein a count of the multiple frames is five or fewer.

21. A computer program product encoded on one or more non-transitory computer storage media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and

processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,

wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,

wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames;

processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and

processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

22. The computer program product of claim 21,

wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and

wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

23. The computer program product of claim 22,

wherein the recurrent neural network further comprises one or more classification layers, and

wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

24. The computer program product of claim 23,

wherein the recurrent neural network further comprises a backward LSTM layer, and

wherein processing the plurality of frames further comprises:

processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and

processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

25. The computer program product of claim 21, wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

26. The computer program product of claim 21, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

27. The computer program product of claim 21, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

28. The computer program product of claim 21, wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

29. The computer program product of claim 21, wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

30. The computer program product of claim 21, wherein a count of the multiple frames is five or fewer.