Method and system for recognizing an object

• US 9,922,245 B2
• Filed: 08/15/2014
• Issued: 03/20/2018
• Est. Priority Date: 08/15/2014
• Status: Active Grant

• Current Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Original Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Inventors: Zhang, Yongmian, Tai, Hung-Shou, Gu, Haisong
• Primary Examiner(s): Hallenbeck-Huber, Jeremiah C
• Assistant Examiner(s): Hodges, Susan E
• Application Number:
  US14/461,057
• Publication Number:
  US 20160048727A1
• Time in Patent Office: 1,313 Days
• Field of Search: 348164
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths

Abstract

A method, a system, and a non-transitory computer readable medium for recognizing an object. The method includes emitting an array of infrared rays from an infrared emitter towards a projection region, the projection region including a first object; generating a reference infrared image by recording an intensity of ray reflection from the projection region without the first object; generating a target infrared image by recording the intensity of ray reflection from the projection region with the first object; comparing the target infrared image to the reference infrared image to generate a predetermined intensity threshold; and extracting the first object from the target infrared image, if the intensity of ray reflection of the target infrared image of the first object exceeds the predetermined intensity threshold.

Claims

1. A method for recognizing movement of a finger of a user, the method comprising:

projecting an image onto a surface;

emitting an array of infrared rays from an infrared emitter towards a projection region, the projection region including a hand of the user, wherein the hand of the user is located between the infrared emitter and the surface in which the image is projected, and the projection region includes at least a portion of the image projected onto the surface and the hand of the user;

generating a reference infrared image by recording an intensity of ray reflection from the projection region without the hand of the user;

generating a target infrared image by recording the intensity of ray reflection from the projection region with the hand of the user;

comparing the target infrared image to the reference infrared image to generate a predetermined intensity threshold;

extracting the hand of the user from the target infrared image, if the intensity of ray reflection of the target infrared image of the hand of the user exceeds the predetermined intensity threshold;

wherein if the intensity of ray reflection of the target infrared image of the hand of the user exceeds the predetermined intensity threshold;

generating a two-dimensional binarized image of the hand of the user from the target infrared image; and

comparing the two-dimensional binarized image of the hand of the user to a model hand to detect and track movement of one or more fingers of the hand of the user and to detect contact of a finger tip with the surface upon which the image is projected, wherein the contact of the finger tip with the surface upon which the image is projected is differentiated from a hover of the finger tip by;

detecting a plurality of pixels around an edge of the finger tip;

superimposing a box around a region of the finger tip and a contact surface; and

extracting Haar-like features from the box superimposed around the region of the finger tip and the contact surface, wherein the Haar-like features define a difference in a sum of pixels of rectangle areas inside the box, and then a classifier uses the Haar-like features to determine if the region looks like a predefined image with characteristics of touch or hover by determining a number of pixels in the box; and

wherein if the number of pixels are equal to or exceeds a predetermined pixel threshold, classifying the finger tip as touching the contract surface and if the number of pixels are less than the predetermined pixel threshold, classifying the finger tip as not touching the contact surface; and

tracking the finger and the hand upon the detection of the touch of the finger tip on the surface to recognize memorized touch-based gestures.

2. The method of claim 1, wherein the hand model comprises a palm, a plurality of finger tips, and corresponding roots between each of a plurality of fingers, and wherein generating the two-dimensional binarized image of the hand comprises:

generating a segmented hand image from the two-dimensional binarized image of the hand;

extracting a hand profile from the segmented hand image;

extracting hand convexities and convexity defects from the hand profile;

removing convexity points that have a depth of the corresponding roots, which are less than a predetermined depth threshold;

determining a finger with a deepest root and eliminating other fingers that point in an opposite direction of the finger;

determining a finger length from a deepest finger root along the opposite direction of the finger as an upper boundary of the palm and a lower boundary of the palm being a point of the deepest root; and

determining a gravity center between the upper boundary and the lower boundary to define a center of the palm, and wherein a radius of the palm is a distance between the gravity center and the point of the deepest root.

3. The method of claim 2, comprising:

eliminating any tips of fingers within a given threshold of the center of the palm;

classifying the non-eliminated tips as finger tips.

4. The method of claim 2, comprising:

tracking the finger tip using a global nearest neighbor, wherein the global nearest neighbor includes a gate around a predicted position of the palm center and a predicted position of finger tips associated with the palm by Kalman filtering.

5. A system for recognizing movement of a finger of a user, the system comprising:

a projector configured to project an image onto a surface;

an infrared emitter configured to emit an array of infrared rays from an infrared emitter towards a projection region, the projection region including the hand of a user, wherein the hand of the user is located between the infrared emitter and the surface in which the image is projected, and the projection region includes at least a portion of the image projected onto the surface and the hand of the user;

an infrared camera configured to record an intensity of ray reflection from the projection region without the hand of the user as a reference infrared image and the intensity of ray reflection of the projection region with the hand of the user infrared image of the hand of the user as a target infrared image; and

a processor for;

comparing the target infrared image to the reference infrared image to generate a predetermined intensity threshold;

extracting the hand of the user from the target infrared image, if the intensity of ray reflection of the target infrared image of the hand of the user exceeds the predetermined intensity threshold;

generating a two-dimensional binarized image of the hand of the user from the target infrared image;

comparing the two-dimensional binarized image of the hand of the user to a model hand to detect and track movement of one or more fingers of the hand of the user and to detect contact of a finger tip with the surface upon which the image is projected, wherein the contact of the finger tip with the surface upon which the image is projected is differentiated from a hover of the finger tip by;

detecting a plurality of pixels around an edge of the finger tip;

superimposing a box around a region of the finger tip and a contact surface;

extracting Haar-like features from the box superimposed around the region of the finger tip and the contact surface, wherein the Haar-like features define a difference in a sum of pixels of rectangle areas inside the box, and then a classifier uses the Haar-like features to determine if the region looks like a predefined image with characteristics of touch or hover by determining a number of pixels in the box; and

wherein if the number of pixels are equal to or exceeds a predetermined pixel threshold, classifying the finger tip as touching the contract surface and if the number of pixels are less than the predetermined pixel threshold, classifying the finger tip as not touching the contact surface; and

tracking the finger and the hand upon the detection of the touch of the finger tip on the surface to recognize memorized touch-based gestures.

6. The system of claim 5, wherein the hand model comprises a palm, a plurality of finger tips, and corresponding roots between each of a plurality of fingers, and wherein generating the two-dimensional binarized image of the hand comprises:

generating a segmented hand image from the two-dimensional binarized image of the hand;

extracting a hand profile from the segmented hand image;

extracting hand convexities and convexity defects from the hand profile;

removing convexity points that have a depth of the corresponding roots, which are less than a predetermined depth threshold;

determining a finger with a deepest root and eliminating other fingers that point in an opposite direction of the finger;

determining a finger length from a deepest finger root along the opposite direction of the finger as an upper boundary of the palm and a lower boundary of the palm being a point of the deepest root; and

determining a gravity center between the upper boundary and the lower boundary to define a center of the palm, and wherein a radius of the palm is a distance between the gravity center and the point of the deepest root.

7. The system of claim 5, comprising:

an infrared pass filter configured to increase a contrast of the intensity of ray reflection of the target and reference infrared images.

12. The system of claim 6, wherein the processor is configured to:

eliminate any tips of fingers within a given threshold of the center of the palm;

classify the non-eliminated tips as finger tips.

13. The system of claim 6, wherein the processor is configured to:

track the finger tip using a global nearest neighbor, wherein the global nearest neighbor includes a gate around a predicted position of the palm center and a predicted position of finger tips associated with the palm by Kalman filtering.

8. A non-transitory computer readable medium containing a computer program having computer readable code embodied to carry out a process for recognizing an object, the process comprising:

projecting an image onto a surface;

emitting an array of infrared rays from an infrared emitter towards a projection region, the projection region including a hand of a user, wherein the hand of the user is located between the infrared emitter and the surface in which the image is projected, and the projection region includes at least a portion of the image projected onto the surface and the hand of the user;

generating a reference infrared image by recording an intensity of ray reflection from the projection region without the hand of the user;

generating a target infrared image by recording the intensity of ray reflection from the projection region with the hand of the user;

comparing the target infrared image to the reference infrared image to generate a predetermined intensity threshold;

extracting the hand of the user from the target infrared image, if the intensity of ray reflection of the target infrared image of the hand of the user exceeds the predetermined intensity threshold; and

wherein if the intensity of ray reflection of the target infrared image of the hand of the user exceeds the predetermined intensity threshold;

generating a two-dimensional binarized image of the hand of the user from the target infrared image; and

comparing the two-dimensional binarized image of the hand of the user to a model hand to detect and track movement of one or more fingers of the hand of the user and to detect contact of a finger tip with the surface upon which the image is projected, wherein the contact of the finger tip with the surface upon which the image is projected is differentiated from a hover of the finger tip by;

detecting a plurality of pixels around an edge of the finger tip;

superimposing a box around a region of the finger tip and a contact surface;

extracting Haar-like features from the box superimposed around the region of the finger tip and the contact surface, wherein the Haar-like features define a difference in a sum of pixels of rectangle areas inside the box, and then a classifier uses the Haar-like features to determine if the region looks like a predefined image with characteristics of touch or hover by determining a number of pixels in the box; and

wherein if the number of pixels are equal to or exceeds a predetermined pixel threshold, classifying the finger tip as touching the contract surface and if the number of pixels are less than the predetermined pixel threshold, classifying the finger tip as not touching the contact surface; and

tracking the finger and the hand upon the detection of the touch of the finger tip on the surface to recognize memorized touch-based gestures.

9. The computer readable medium of claim 8, wherein the hand model comprises a palm, a plurality of finger tips, and corresponding roots between each of a plurality of fingers, and wherein generating the two-dimensional binarized image of the hand comprises:

generating a segmented hand image from the two-dimensional binarized image of the hand;

extracting a hand profile from the segmented hand image;

extracting hand convexities and convexity defects from the hand profile;

removing convexity points that have a depth of the corresponding roots, which are less than a predetermined depth threshold;

determining a finger with a deepest root and eliminating other fingers that point in an opposite direction of the finger;

determining a finger length from a deepest finger root along the opposite direction of the finger as an upper boundary of the palm and a lower boundary of the palm being a point of the deepest root; and

determining a gravity center between the upper boundary and the lower boundary to define a center of the palm, and wherein a radius of the palm is a distance between the gravity center and the point of the deepest root.

10. The computer readable medium of claim 9, comprising:

eliminating any tips of fingers within a given threshold of the center of the palm;

classifying the non-eliminated tips as finger tips.

11. The computer readable medium of claim 9, comprising:

tracking the finger tip using a global nearest neighbor, wherein the global nearest neighbor includes a gate around a predicted position of the palm center and a predicted position of finger tips associated with the palm by Kalman filtering.