Apparatus and method for recognizing object using depth image

• US 9,690,985 B2
• Filed: 11/07/2013
• Issued: 06/27/2017
• Est. Priority Date: 02/20/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Han, Jae Joon, Lee, Chang Kyo, Choi, Chang Kyu
• Primary Examiner(s): Azarian, Seyed
• Application Number:
  US14/074,247
• Publication Number:
  US 20140233848A1
• Time in Patent Office: 1,328 Days
• Field of Search: 382100, 382103, 382106-107, 382154, 382155, 382162-168, 382173, 382181, 382190, 382199, 382209, 382232, 382254, 382274, 382276, 382284-291, 382305, 382312, 382180, 3482221, 348708
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/72 Data preparation, e.g. statistical preprocessing of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An apparatus recognizes an object using a hole in a depth image. An apparatus may include a foreground extractor to extract a foreground from the depth image, a hole determiner to determine whether a hole is present in the depth image, based on the foreground and a color image, a feature vector generator to generate a feature vector, by generating a plurality of features corresponding to the object based on the foreground and the hole, and an object recognizer to recognize the object, based on the generated feature vector and at least one reference feature vector.

Claims

1. An apparatus for recognizing an object, the apparatus comprising:

a processor configured to

extract a foreground from a depth image comprising a depth associated with the object;

determine whether a hole is present in the depth image, based on the foreground and a color image comprising a color associated with the object;

generate, using the foreground and the hole, a feature vector comprising a plurality of features corresponding to the object, based on a result of the determining; and

recognize the object by calculating a distance between the generated feature vector and a reference feature vector corresponding to a predetermined shape of the object.

2. The apparatus of claim 1, wherein the processor is further configured to extract the hole based on the foreground and the color image when it is determined that the hole is present in the depth image.

3. The apparatus of claim 2, wherein the processor is further configured to calculate a probability that pixels included in the color image are included in the object, based on color information of the pixels, and to extract the hole based on the probability.

4. The apparatus of claim 3, wherein the processor is further configured to determine that an area excluded from the foreground of the depth image corresponds to the hole when a probability that the area is included in the object is greater than or equal to a predetermined threshold probability.

5. The apparatus of claim 3, wherein the object comprises a hand of a user, and the probability comprises a skin probability.

6. The apparatus of claim 2, wherein the plurality of features comprises information regarding whether a feature corresponds to a hole.

7. The apparatus of claim 6, wherein the processor is further configured to compare, based on the information regarding whether a feature corresponds to a hole, remaining features, excluding a feature corresponding to a hole from a plurality of features included in the feature vector, to reference features included in the at least one reference feature vector.

8. The apparatus of claim 6, wherein the at least one reference feature vector comprises a plurality of codewords generated by a hierarchical clustering scheme, and

wherein the processor is further configured to

generate a foreground feature vector comprising the remaining features based on the information regarding whether a feature corresponds to a hole,

detect a codeword most similar to the foreground feature vector, among the plurality of codewords, by comparing the plurality of codewords to the foreground feature vector, and

recognize the object using the detected codeword.

9. The apparatus of claim 6, wherein the processor is further configured to:

obtain a decision tree for comparing the feature vector to the at least one reference feature vector;

recognize a branch node corresponding to a feature corresponding to a hole, among the plurality of features, based on the information regarding whether a feature corresponds to a hole;

traverse a plurality of branches based on the branch node; and

estimate the object based on information included in a plurality of leaf nodes to be reached based on a result of the traversing,

wherein the decision tree comprises a plurality of branch nodes corresponding to a plurality of features included in the feature vector, and at least one leaf node comprising information related to a shape of the object.

10. The apparatus of claim 9, wherein the processor is further configured to

calculate a probability that pixels included in the color image are included in the object, based on color information of the pixels,

obtain weights of the plurality of leaf nodes based on the calculated probability, and

estimate the object based on the information included in the plurality of leaf nodes and the weights of the plurality of leaf nodes.

11. The apparatus of claim 1, wherein the processor is further configured to determine a value of a feature corresponding to the hole, based on statistical values associated with features corresponding to the foreground, when a hole is present in the depth image.

12. The apparatus of claim 11, wherein the statistical values comprise at least one of a minimum value, a maximum value, an intermediate value, an average, and a standard deviation of depth values of the plurality of features corresponding to the foreground.

13. The apparatus of claim 1, wherein the at least one reference feature vector comprises a plurality of codewords generated by a hierarchical clustering scheme, and

wherein the processor is further configured to detect a codeword most similar to the feature vector, among the plurality of codewords, by comparing the plurality of codewords to the feature vector, and to recognize the object using the detected codeword.

14. The apparatus of claim 1, wherein the depth image and the color image of the object are captured simultaneously.

15. The apparatus of claim 1, wherein

the at least one reference feature vector comprises a plurality of codewords,

the plurality of codewords is stored in a database provided in advance, and

the processor is further configured to detect a codeword most similar to the feature vector, among the plurality of codewords, by comparing the plurality of codewords to the feature vector using the database, and to recognize the object using the detected codeword.

16. The apparatus of claim 1, wherein the processor is further configured to determine that a first area, from among a second area excluded from the foreground of the depth image, corresponds to the hole when a probability that the first area is included in the object is greater than or equal to a predetermined threshold probability.

17. An apparatus for recognizing an object, the apparatus comprising:

a processor configured to

extract a foreground and a background from a depth image comprising a depth associated with the object;

determine a depth value of the background, based on statistical values associated with depth values of pixels corresponding to the foreground;

generate a feature vector, by generating a plurality of features corresponding to the object based on the depth value of the background and the depth values of the pixels corresponding to the foreground; and

recognize the object, based on the feature vector and at least one reference feature vector corresponding to a predetermined shape of the object.

18. The apparatus of claim 17, wherein the statistical values comprise at least one of a minimum value, a maximum value, an intermediate value, an average, and a standard deviation of the depth values of the pixels corresponding to the foreground.

19. The apparatus of claim 17, wherein the processor is further configured to determine the depth value of the background, by calculating a sum of a minimum value and a standard deviation of the depth values of the pixels corresponding to the foreground.

20. The apparatus of claim 17, wherein the at least one reference feature vector comprises a plurality of codewords generated by a hierarchical clustering scheme, and

wherein the processor is further configured to detect a codeword most similar to the feature vector, among the plurality of codewords, by comparing the plurality of codewords to the feature vector, and to recognize the object using the detected codeword.

21. A method of recognizing an object, the method comprising:

acquiring a depth image comprising a depth associated with the object, and a color image comprising a color associated with the object;

extracting a foreground from the depth image;

determining whether a hole is present in the depth image, using the foreground and the color image;

generating, using at least one processor, a feature vector comprising a plurality of features corresponding to the object using the foreground and the hole, based on a result of the determination; and

recognizing the object by calculating a distance between the generated feature vector and a reference feature vector corresponding to a predetermined shape of the object.