Real-time recognition of pointing information from video

• US 5,168,531 A
• Filed: 06/27/1991
• Issued: 12/01/1992
• Est. Priority Date: 06/27/1991
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Digital Equipment Corporation (HP Inc.)
• Inventors: Sigel, Claude
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): KLOCINSKI, STEVEN
• Application Number:
  US07/722,082
• Time in Patent Office: 523 Days
• Field of Search: 382/42, 382/59, 382/48, 382/19, 382/16, 340/706, 340/709
• US Class Current: 382/291
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/443 by matching or filtering

Abstract

An occurrence of a predefined object in an image is recognized by receiving image data, convolving the image data with a set of predefined functions to analyze the image data into occurrences of predefined elementary features, and examining the occurrences for an occurrence of a predefined combination of the elementary features that is characteristic of the predefined object. Preferably the image data are convolved directly with a first predefined function to determine blob responses, and a second predefined function to determine ganglia responses indicating edges of objects. Then the ganglia responses are convolved with a third predefined function to determine simple responses indicating lines in the image, and the simple responses are combined with the ganglia responses to determine complex responses indicating terminated line segments in the image. A pointing finger, for example, is recognized from the combination of a blob response and a complex response. The method, for example, permits a data input terminal to recognize in real time the presence, position, and orientation of a pointing finger, to eliminate the need for data input devices such as "mice" or "joysticks." Therefore a user can direct an application program in the most natural way, without the distraction of manipulating a data input device.

Claims

1. A method of operating a digital computer to recognize an occurrence of a predefined object in an image, said method comprising the steps of:

a) receiving image data representing samples of said image at spaced locations in said image;

b) convolving said image data with a set of predefined functions to analyze the image data into occurrences of predefined elementary features; and

c) examining the occurrences of the predefined elementary features for an occurrence of a predefined combination of occurrences of said elementary features that is characteristic of the predefined object;

wherein said predefined elementary features include blobs, simples and complexes, wherein the blobs indicate regions of similar luminosity, simples indicate lines in said image, and complexes indicate terminated line segments in said image; and

wherein the image data are convolved with a first one of said functions to provide blob responses, the image data are convolved with a second one of said functions to provide ganglia responses indicating edges of objects, the ganglia responses are convolved to provide simple responses, and the simple responses and ganglia responses are combined to provide complex responses.

2. The method as claimed in claim 1, wherein the simples include respective indications of lines at six different angular orientations at 30 degree increments.

3. The method as claimed in claim 1, wherein said predefined combination of said elementary features includes a complex extending from a blob.

4. The method as claimed in claim 3, wherein said predefined object is a human hand having a pointing finger.

5. The method as claimed in claim 1, further comprising the step of determining an angular orientation for an occurrence of said predefined combination of said elementary features that is characteristic of the predefined object.

6. The method as claimed in claim 5, wherein said predefined elementary features include lines at a plurality of angular orientations.

7. The method as claimed in claim 6, wherein said step of determining an angular orientation for an occurrence includes computing an angle from convolutions for lines at two neighboring angular orientations.

8. The method as claimed in claim 1, further comprising the step of computing a position for said occurrence of said predefined object by interpolating between positions of occurrences of said predefined combination which occur at neighboring spaced locations in said image.

9. The method as claimed in claim 1, further comprising the step d) of computing a response for a plurality of occurrences of said predefined combination of said elementary features that is characteristic of the predefined object, and determining a location for the predefined object from one of said occurrences of said predefined combination having a maximum response.

10. The method as claimed in claim 1, further comprising the step of suppressing features that remain constant with time.

11. The method as claimed in claim 1, wherein for at least one of said predefined functions, said convolving is performed by computing, for each spaced location in said image, a vector scalar product of said image data and said one of said predefined functions after shifting said one of said predefined functions so that said one of said predefined functions is centered upon said each spaced location.

12. A method of operating a digital computer to recognize an occurrence of a predefined object in an image, said method comprising the steps of:

a) receiving image data representing samples of said image at spaced locations in said image;

b) convolving said image data with a first predefined function to determine blob responses indicating regions of similar luminance in said image;

c) convolving said image data with a second predefined function to determine ganglia responses indicating edges of objects;

d) convolving said ganglia responses with a third predefined function to determine simple responses indicating lines in said image;

e) combining the simple responses and the ganglia responses to determine complex responses indicating terminated line segments in said image; and

f) examining combinations of blob responses and complex responses for an occurrence of a predefined combination that is characteristic of the predefined object.

13. The method as claimed in claim 12 wherein said simple responses include responses indicating lines in said image at a plurality of angular orientations, and said method further comprises the step g) of determining angular orientation for the occurrence of said predefined combination that is characteristic of the predefined object by computing an angle from two of said simple responses indicating lines having different and neighboring angular orientations.

14. The method as claimed in claim 12, wherein said predefined object is a human hand, and said predefined combination is a complex response and a blob response indicating a terminated line segment extending from a region of similar luminance in said image.

15. The method as claimed in claim 12, wherein one of said simple responses is combined with ganglia responses to provide a complex response only when said one of said simple responses has a value exceeding a predetermined threshold.

16. The method as claimed in claim 12, further comprising the step of computing a position for said occurrence of said predefined object by interpolating between positions of occurrences of said predefined combination which occur at neighboring spaced locations in said image, based upon responses computed for said occurrences of said predefined combination.

17. The method as claimed in claim 12, wherein step f) further comprises the step of computing responses for a plurality of combinations of blob responses and complex responses that are characteristic of the predefined object, and determining a location for the predefined object from the combination of blob and complex responses having a maximum response.

18. The method as claimed in claim 12, further comprising the step of suppressing responses that do not change with time.

19. The method as claimed in claim 12, wherein for each of said first and second predefined functions, said convolving is performed by computing, for each spaced location in said image, a vector scalar product of said image data and said each of said predefined functions after shifting said each of said predefined functions so that said each of said predefined functions is centered upon said each spaced location.

20. A method of operating a digital computer to receive information from a human operator at a computer terminal in response to a pointing finger of the human operator, said method comprising the step of:

a) scanning a region of space near said computer terminal to obtain image data representing samples of an image at spaced locations near said computer terminal; and

b) analyzing said image data to recognize when the pointing finger is present in the image and to determine the position of the pointing finger in the image when the pointing finger is present, wherein said analyzing includes convolving said image data to resolve the image data into blobs characteristic of a fist of the human operator and complexes characteristic of the pointing finger, and examining the blobs and the complexes for a combination of a complex extending from a blob to recognize the presence of the pointing finger, and wherein said convolving said image data includes;

convolving said image data with a first predefined function to determine blob responses;

convolving said image data with a second predefined function to determine ganglia responses indicating edges of said finger;

convolving said ganglia responses with a third predefined function to determine simple responses; and

combining the simple responses and the ganglia responses to determine complex responses.

21. The method as claimed in claim 20, further comprising the step of determining an angular orientation of said finger from said image data.

22. The method as claimed in claim 21, wherein said step of determining an angular orientation of said finger includes computing an angle from convolutions of the image data with predetermined functions having different angular orientations.

23. The method as claimed in claim 20, wherein said step b) is not responsive to finger-like features of said image that remain constant with time.

24. A data input terminal for a digital computer, said data input terminal comprising, in combination:

a video display;

a video camera mounted for scanning a region of space near said video display to provide a video signal representing an image of said region of space;

an analog-to-digital converter receiving said video signal and providing digital samples of image data; and

digital computing means for receiving said digital samples of image data and analyzing said image data to recognize when a pointing finger of a human operator is present in said image and to determine the location of the pointing finger in the image when the pointing finger is present;

wherein said digital computing means includes means for convolving said image data to resolve the image data into blobs characteristic of a fist of the human operator and complexes characteristic of the pointing finger, and examining the blobs and the complexes for a combination f a complex extending from a blob to recognize the presence of the pointing finger, and

wherein said means for convolving said image data includes means for convolving said image data with a first predefined function to determine blob responses, means for convolving said image data with a second predefined function to determine ganglia responses indicating edges of the pointing finger, means for convolving said ganglia responses with a third predefined function to determine simple responses, and means for combining the simple responses and the ganglia responses to determine complex responses.

25. The data input terminal as claimed in claim 24, wherein said digital computing means includes means for computing responses for a plurality of combinations of complexes extending from blobs, and means for determining a location for the pointing finger from one of said plurality of combinations having a maximum response.

26. The data input terminal as claimed in claim 24, wherein said digital computing means further includes means for determining an angular orientation of the pointing finger from the image data.

27. The data input terminal as claimed in claim 26, wherein said means for determining an angular orientation of said finger includes means for computing an angle from convolutions of the image data with predetermined functions having different angular orientations.

28. The data input terminal as claimed in claim 24, wherein said digital computing means further comprises means for computing the location of said pointing finger by interpolating between positions of a complex of an occurrence of said pointing finger and a complex at a neighboring location in said image.

29. The data input terminal as claimed in claim 24, wherein said digital computing means further comprises means for suppressing complexes that remain constant with time.