APPARATUS AND METHOD FOR RECOGNIZING OBJECT USING DEPTH IMAGE

• US 20140233848A1
• Filed: 11/07/2013
• Published: 08/21/2014
• Est. Priority Date: 02/20/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: HAN, Jae Joon, CHOI, Chang Kyu, LEE, Chang Kyo
• Granted Patent: US 9,690,985 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/72 Data preparation, e.g. statistical preprocessing of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An apparatus recognizes an object using a hole in a depth image. An apparatus may include a foreground extractor to extract a foreground from the depth image, a hole determiner to determine whether a hole is present in the depth image, based on the foreground and a color image, a feature vector generator to generate a feature vector, by generating a plurality of features corresponding to the object based on the foreground and the hole, and an object recognizer to recognize the object, based on the generated feature vector and at least one reference feature vector.

Claims

1. An apparatus for recognizing an object, the apparatus comprising:

a foreground extractor to extract a foreground from a depth image comprising a depth associated with the object;

a hole determiner to determine whether a hole is present in the depth image, based on the foreground and a color image comprising a color associated with the object;

a feature vector generator, using at least one processor, to generate a feature vector, by generating a plurality of features corresponding to the object using at least one of the foreground and the hole, based on a result of the determining; and

an object recognizer to recognize the object, based on the generated feature vector and at least one reference feature vector corresponding to a predetermined shape of the object.

2. The apparatus of claim 1, wherein the hole determiner comprises:

a hole extractor to extract a hole based on the foreground and the color image when it is determined that the hole is present in the depth image.

3. The apparatus of claim 2, wherein the hole extractor calculates a probability that pixels included in the color image are included in the object, based on color information of the pixels, and extracts the hole based on the probability.

4. The apparatus of claim 3, wherein the hole extractor determines that an area excluded from the foreground of the depth image corresponds to the hole when a probability that the area is included in the object is greater than or equal to a predetermined threshold probability.

5. The apparatus of claim 3, wherein the object comprises a hand of a user, and the probability comprises a skin probability.

6. The apparatus of claim 2, wherein the plurality of features comprises information regarding whether a feature corresponds to a hole.

7. The apparatus of claim 6, wherein the object recognizer compares, based on the information regarding whether a feature corresponds to a hole, remaining features, excluding a feature corresponding to a hole from a plurality of features included in the feature vector, to reference features included in the at least one reference feature vector.

8. The apparatus of claim 6, wherein the at least one reference feature vector comprises a plurality of codewords generated by a hierarchical clustering scheme,

wherein the object recognizer generates a foreground feature vector comprising the remaining features based on the information regarding whether a feature corresponds to a hole, detects a codeword most similar to the foreground feature vector, among the plurality of codewords, by comparing the plurality of codewords to the foreground feature vector, and recognizes the object using the detected codeword.

9. The apparatus of claim 6, wherein the object recognizer comprises:

a decision tree obtainer to obtain a decision tree for comparing the feature vector to the at least one reference feature vector; and

an object estimator to recognize a branch node corresponding to a feature corresponding to a hole, among the plurality of features, based on the information regarding whether a feature corresponds to a hole, traverse a plurality of branches based on the branch node, and estimate the object based on information included in a plurality of leaf nodes to be reached based on a result of the traversing,

wherein the decision tree comprises a plurality of branch nodes corresponding to a plurality of features included in the feature vector, and at least one leaf node comprising information related to a shape of the object.

10. The apparatus of claim 9, wherein

the hole extractor calculates a probability that pixels included in the color image are included in the object, based on color information of the pixels, and

the object estimator obtains weights of the plurality of leaf nodes based on the calculated probability, and estimates the object based on the information included in the plurality of leaf nodes and the weights of the plurality of leaf nodes.

11. The apparatus of claim 1, wherein the feature vector generator determines a value of a feature corresponding to the hole, based on statistical values associated with features corresponding to the foreground, when a hole is present in the depth image.

12. The apparatus of claim 11, wherein the statistical values comprise at least one of a minimum value, a maximum value, an intermediate value, an average, and a standard deviation of depth values of the plurality of features corresponding to the foreground.

13. The apparatus of claim 1, wherein the at least one reference feature vector comprises a plurality of codewords generated by a hierarchical clustering scheme,

wherein the object recognizer detects a codeword most similar to the feature vector, among the plurality of codewords, by comparing the plurality of codewords to the feature vector, and recognizes the object using the detected codeword.

14. The apparatus of claim 1, wherein the depth image and the color image of the object are captured simultaneously.

15. The apparatus of claim 1, wherein

the at least one reference feature vector comprises a plurality of codewords,

the plurality of codewords is stored in a database provided in advance, and

the object recognizer detects a codeword most similar to the feature vector, among the plurality of codewords, by comparing the plurality of codewords to the feature vector using the database, and recognizes the object using the detected codeword.

16. An apparatus for recognizing an object, the apparatus comprising:

an extractor to extract a foreground and a background from a depth image comprising a depth associated with the object;

a determiner to determine a depth value of the background, based on statistical values associated with depth values of pixels corresponding to the foreground;

a feature vector generator, using at least one processor, to generate a feature vector, by generating a plurality of features corresponding to the object based on the depth value of the background and the depth values of the pixels corresponding to the foreground; and

an object recognizer to recognize the object, based on the feature vector and at least one reference feature vector corresponding to a predetermined shape of the object.

17. The apparatus of claim 16, wherein the statistical values comprise at least one of a minimum value, a maximum value, an intermediate value, an average, and a standard deviation of the depth values of the pixels corresponding to the foreground.

18. The apparatus of claim 16, wherein the determiner determines the depth value of the background, by calculating a sum of a minimum value and a standard deviation of the depth values of the pixels corresponding to the foreground.

19. The apparatus of claim 16, wherein the at least one reference feature vector comprises a plurality of codewords generated by a hierarchical clustering scheme,

wherein the object recognizer detects a codeword most similar to the feature vector, among the plurality of codewords, by comparing the plurality of codewords to the feature vector, and recognizes the object using the detected codeword.

20. A method of recognizing an object, the method comprising:

acquiring a depth image comprising a depth associated with the object, and a color image comprising a color associated with the object;

extracting a foreground from the depth image;

determining whether a hole is present in the depth image, using the foreground and the color image; and

generating, using at least one processor, a feature vector for recognizing the object, by generating a plurality of features corresponding to the object using at least one of the foreground and the hole, based on a result of the determination.