Object recognition method and apparatus

• US 10,127,439 B2
• Filed: 01/14/2016
• Issued: 11/13/2018
• Est. Priority Date: 01/15/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Wang, Xiaotao, Wang, Qiang, Guo, Ping, Wang, Shandong, Mao, Wentao, Shao, Guangqi, Ryu, Eric Hyunsurk, Lee, Kyoobin, Park, Keun Joo
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US14/995,275
• Publication Number:
  US 20160210513A1
• Time in Patent Office: 1,034 Days
• Field of Search: None
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

An object recognition apparatus and an object recognition method are provided. The object recognition method includes generating an input image based on an event flow of an object, generating a composite feature based on features extracted by a plurality of recognizers, and recognizing the object based on the composite feature.

Claims

1. An object recognition method comprising:

generating an image based on an event flow of an object;

generating, based on the generated image, a plurality of region of interest (ROI) images corresponding to a plurality of ROIs;

inputting the plurality of ROI images to a plurality of recognizers, the recognizers being configured to extract features of the plurality of ROIs;

generating a composite feature based on the features extracted by the plurality of recognizers; and

recognizing the object based on the composite feature.

2. The object recognition method of claim 1, further comprising:

extracting, by each of the plurality of recognizers, the features of the plurality of ROIs from the plurality of ROI images, respectively,

wherein the plurality of ROI images is input to the plurality of recognizers in parallel.

3. The object recognition method of claim 1, wherein the generating of the composite feature comprises:

combining the features extracted by the plurality of recognizers; and

generating the composite feature based on a result of the combining.

4. The object recognition method of claim 1, wherein the plurality of recognizers comprises a multiple-input convolutional neural network (CNN) including a plurality of CNNs, and

the multiple-input CNN has a multiple-input and single-output (MISO) architecture configured to extract the composite feature for identifying the object from the plurality of ROI images input to the plurality of CNNs in parallel.

5. The object recognition method of claim 1, wherein at least one of the plurality of recognizers comprises:

a convolution layer to perform a convolutional operation on an input ROI image;

a pooling layer to sample an output of the convolution layer; and

a full connection layer to extract a feature of an ROI based on a full connection network and an output of the pooling layer.

6. The object recognition method of claim 1, wherein when the object is a person, the plurality of ROIs comprises at least one from among regions corresponding to an entire body, a head, an upper half body, a lower half body, a leg, and an arm.

7. The object recognition method of claim 1, wherein the generating of the image comprises:

detecting a bounding box of the object through a vertical projection and a horizontal projection; and

generating the image based on the bounding box.

8. The object recognition method of claim 1, wherein the generating of the plurality of ROI images comprises:

extracting a pre-defined ROI including a target of interest of the object from the image;

generating a ROI image based on the pre-defined ROI;

acquiring a width and a height of the target of interest based on the ROI image;

resizing the ROI image based on a preset distance using the width and the height; and

generating the plurality of ROIs based on the resized ROI image.

9. The object recognition method of claim 8, wherein the acquiring comprises:

acquiring the width of the target of interest by applying a vertical projection to the ROI image; and

acquiring the height of the target of interest based on a fixed ratio of the target of interest to the width.

10. The object recognition method of claim 1, further comprising:

outputting validities of the plurality of ROI images using a filter configured to classify the validities; and

inputting the plurality of ROI images to the plurality of recognizers based on a result of the outputting.

11. The object recognition method of claim 1, wherein the generating of the image comprises:

generating an integral image of a pre-defined time based on the event flow;

filtering the integral image using a spatio-temporal filter; and

generating the image based on the filtered integral image.

21. The object recognition method of claim 1, wherein the object is sensed by an event-based vision sensor.

22. The object recognition method of claim 1, wherein the plurality of ROIs is defined in advance.

12. An object registration method comprising:

generating an image based on an event flow of an object;

generating, based on the image, a plurality of region of interest (ROI) images corresponding to a plurality of ROIs;

inputting the plurality of ROI images to a plurality of recognizers, the recognizers being configured to extract features of the plurality of ROIs;

generating a composite feature based on the features extracted by the plurality of recognizers; and

registering the object in connection with the composite feature.

23. The object recognition method of claim 12, wherein the object is sensed by an event-based vision sensor.

24. The object recognition method of claim 12, wherein the plurality of ROIs is defined in advance.

13. An object recognition apparatus comprising:

a processor; and

a memory storing instructions executable by the processor,

wherein the processor is configured to;

generate an image based on an event flow of an object;

generate a plurality of ROI images corresponding to a plurality of ROIs, based on the image;

input the plurality of ROI images to a plurality of recognizers, the recognizers being configured to extract features of the plurality of ROIs;

generate a composite feature based on the features extracted by the plurality of recognizers; and

recognize the object based on the composite feature.

14. The object recognition apparatus of claim 13,

wherein the processor is further configured to extract the features of the plurality of ROIs from the plurality of ROI images, respectively,

wherein the plurality of ROI images is input to the plurality of recognizers in parallel.

15. The object recognition apparatus of claim 13, wherein the processor is further configured to combine the features extracted through the plurality of recognizers and generate the composite feature based on a result of the combining.

16. The object recognition apparatus of claim 13, wherein the plurality of recognizers comprises a multiple-input convolutional neural network (CNN) including a plurality of CNNs, and

the multiple-input CNN has a multiple-input and single-output (MISO) architecture configured to extract the composite feature for identifying the object from the plurality of ROI images input to the plurality of CNNs in parallel.

17. The object recognition apparatus of claim 13, wherein the processor is further configured to generate an integral image of a pre-defined time based on the event flow, filter the integral image using a spatio-temporal filter, and generate the image based on the filtered integral image.

18. The object recognition apparatus of claim 13, wherein the processor is further configured to detect a bounding box of the object through a vertical projection and a horizontal projection and generate the image based on the bounding box.

19. The object recognition apparatus of claim 13, wherein the processor is further configured to:

extract a pre-defined ROI including a target of interest of the object from the image;

generate an ROI image based on the pre-defined ROI;

acquire a width and a height of the target of interest based on the ROI image;

resize the ROI image based on a preset distance using the width and the height; and

generate the plurality of ROIs based on the resized ROI image.

20. The object recognition apparatus of claim 13,

wherein the processor is further configured to output validities of the plurality of ROI images using a filter configured to classify the validities and input the plurality of ROI images to the plurality of recognizers based on a result of the outputting.

25. The object recognition apparatus of claim 13, wherein the object is sensed by an event-based vision sensor.

26. The object recognition apparatus of claim 13, wherein the plurality of ROIs is defined in advance.