Object detection using cascaded convolutional neural networks

• US 9,697,416 B2
• Filed: 06/29/2016
• Issued: 07/04/2017
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shen, Xiaohui, Li, Haoxiang, Lin, Zhe, Brandt, Jonathan W.
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US15/196,478
• Publication Number:
  US 20160307074A1
• Time in Patent Office: 370 Days
• Field of Search: 382156, 382157, 382170, 382171, 382190, 382209, 382278, 358537, 358538, 358452
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7747 Organisation of the process, e.g. bagging or boosting
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Different candidate windows in an image are identified, such as by sliding a rectangular or other geometric shape of different sizes over an image to identify portions of the image (groups of pixels in the image). The candidate windows are analyzed by a set of convolutional neural networks, which are cascaded so that the input of one convolutional neural network layer is based on the input of another convolutional neural network layer. Each convolutional neural network layer drops or rejects one or more candidate windows that the convolutional neural network layer determines does not include an object (e.g., a face). The candidate windows that are identified as including an object (e.g., a face) are analyzed by another one of the convolutional neural network layers. The candidate windows identified by the last of the convolutional neural network layers are the indications of the objects (e.g., faces) in the image.

Claims

1. A method comprising:

identifying multiple candidate windows in an image, each candidate window including a group of pixels of the image, the multiple candidate windows including overlapping candidate windows;

identifying one or more of the multiple candidate windows that include an object, the identifying including analyzing the multiple candidate windows using cascaded convolutional neural networks, the cascaded convolutional neural networks including multiple cascade layers, each cascade layer comprising a convolutional neural network, the multiple cascade layers including a first cascade layer that analyzes the identified multiple candidate windows, a second cascade layer that analyzes ones of the multiple candidate windows identified by the first cascade layer as including an object, and a third cascade layer that analyzes ones of the multiple candidate windows identified by the second cascade layer as including an object; and

outputting, as an indication of one or more objects in the image, an indication of one or more of the multiple candidate windows identified by the third cascade layer as including an object.

2. A method as recited in claim 1, the object comprising a face.

3. A method as recited in claim 1, the cascaded convolutional neural networks further including one or more additional cascade layers situated between the first cascade layer and the third cascade layer.

4. A method as recited in claim 1, further comprising:

re-using, for one of the multiple candidate windows identified by the first cascade layer as including an object, a feature of the first cascade layer in the second cascade layer; and

re-using, for one of the multiple candidate windows identified by both the first cascade layer and the second cascade layer as including an object, a feature of the first cascade layer in the third cascade layer.

5. A method as recited in claim 4, further comprising re-using, for one of the multiple candidate windows identified by the second cascade layer as including an object, a feature of the second cascade layer in the third cascade layer.

6. A method as recited in claim 1, the object comprising a face, and the method further comprising using a window alignment neural network layer, situated between two cascade layers of the multiple cascade layers, to re-position one of the multiple candidate windows to more accurately identify a full face rather than just a portion of a face.

7. A method as recited in claim 6, the window alignment neural network layer generating both a translation value and a scale value for the one of the multiple candidate windows, and the using the window alignment neural network layer to re-position the one of the multiple candidate windows including moving the one of the multiple candidate windows in accordance with the translation value and scaling the one of the multiple candidate windows in accordance with the scale value.

8. A method as recited in claim 7, further comprising:

re-using, for one of the multiple candidate windows identified by the first cascade layer as including an object, a feature of the first cascade layer in the second cascade layer; and

re-using, for one of the multiple candidate windows identified by both the first cascade layer and the second cascade layer as including an object, a feature of the first cascade layer in the third cascade layer.

9. A method as recited in claim 6, the window alignment neural network layer being situated between the first cascade layer and the second cascade layer.

10. A method as recited in claim 6, the window alignment neural network layer being situated between the second cascade layer and the third cascade layer.

11. A cascaded convolutional neural networks object detection system comprising:

an image access module configured to obtain an image;

a first cascade layer comprising a first convolutional neural network, the first cascade layer configured to analyze multiple candidate windows identified in the image to identify a first set of the multiple candidate windows that include an object, each candidate window including a group of pixels of the image, the multiple candidate windows including overlapping candidate windows;

a second cascade layer comprising a second convolutional neural network, the second cascade layer configured to analyze the first set of the multiple candidate windows to identify a second set of the multiple candidate windows that include an object;

a third cascade layer comprising a third convolutional neural network, the third cascade layer configured to analyze the second set of the multiple candidate windows to identify a third set of the multiple candidate windows that include an object; and

an output module configured to output, for each candidate window of the third set of the multiple candidate windows, an indication of the object included in the candidate window.

12. A system as recited in claim 11, the output module being further configured to output the indication of the identified object by displaying on the image a rectangular shape around the object.

13. A system as recited in claim 11, further comprising:

the second cascade layer being further configured to re-use, for one of the multiple candidate windows identified by the first cascade layer as including an object, a feature of the first cascade layer; and

the third cascade layer being further configured to re-use, for one of the multiple candidate windows identified by both the first cascade layer and the second cascade layer as including an object, a feature of the first cascade layer.

14. A system as recited in claim 13, the third cascade layer being further configured to re-use, for one of the multiple candidate windows identified by the second cascade layer as including an object, a feature of the second cascade layer.

15. A system as recited in claim 11, the object comprising a face, and the system further comprising a window alignment neural network layer, situated between two cascade layers of the multiple cascade layers, configured to re-position one of the multiple candidate windows to more accurately identify a full face rather than just a portion of a face.

16. A system as recited in claim 15, the window alignment neural network layer being configured to generate both a translation value and a scale value for the one of the multiple candidate windows, and to re-position the one of the multiple candidate windows by moving the one of the multiple candidate windows in accordance with the translation value and scaling the one of the multiple candidate windows in accordance with the scale value.

17. A computing device comprising:

one or more processors; and

one or more computer-readable storage media having stored thereon multiple instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

identifying multiple candidate windows in an image, each candidate window including a group of pixels of the image, the multiple candidate windows including overlapping candidate windows;

identifying one or more of the multiple candidate windows that include an object, the identifying including analyzing the multiple candidate windows using cascaded convolutional neural networks, the cascaded convolutional neural networks including multiple cascade layers, each cascade layer comprising a convolutional neural network, the multiple cascade layers including a first cascade layer that analyzes the identified multiple candidate windows, a second cascade layer that analyzes ones of the multiple candidate windows identified by the first cascade layer as including an object, and a third cascade layer that analyzes ones of the multiple candidate windows identified by the second cascade layer as including an object; and

outputting, as an indication of one or more objects in the image, an indication of one or more of the multiple candidate windows identified by the third cascade layer as including an object.

18. A computing device as recited in claim 17, the object comprising a face, and the acts further comprising using a window alignment neural network layer, situated between two cascade layers of the multiple cascade layers, to re-position one of the multiple candidate windows to more accurately identify a full face rather than just a portion of a face.

19. A computing device as recited in claim 18, the window alignment neural network layer generating both a translation value and a scale value for the one of the multiple candidate windows, and the using the window alignment neural network layer to re-position the one of the multiple candidate windows including moving the one of the multiple candidate windows in accordance with the translation value and scaling the one of the multiple candidate windows in accordance with the scale value.

20. A computing device as recited in claim 17, the object comprising a face, the acts further comprising:

re-using, for one of the multiple candidate windows identified by the first cascade layer as including a face, a feature of the first cascade layer in the second cascade layer; and

re-using, for one of the multiple candidate windows identified by both the first cascade layer and the second cascade layer as including a face, a feature of the first cascade layer in the third cascade layer.