3D human interface apparatus using motion recognition based on dynamic image processing

• US 5,581,276 A
• Filed: 09/08/1993
• Issued: 12/03/1996
• Est. Priority Date: 09/08/1992
• Status: Expired due to Term

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Kuno, Yoshinori, Cipolla, Roberto, Okamoto, Yasukazu
• Primary Examiner(s): Weldon, Ulysses
• Application Number:
  US08/117,930
• Time in Patent Office: 1,182 Days
• Field of Search: 382/190, 382/199, 382/201, 382/203, 348/13, 348/20, 395/135, 345/156
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A 3D human interface apparatus using a motion recognition based on a dynamic image processing in which the motion of an operator operated object as an imaging target can be recognized accurately and stably. The apparatus includes: an image input unit for entering a plurality of time series images of an object operated by the operator into a motion representing a command; a feature point extraction unit for extracting at least four feature points including at least three reference feature points and one fiducial feature point on the object, from each of the images; a motion recognition unit for recognizing the motion of the object by calculating motion parameters, according to an affine transformation determined from changes of positions of the reference feature points on the images, and a virtual parallax for the fiducial feature point expressing a difference between an actual position change on the images and a virtual position change according to the affine transformation; and a command input unit for inputting the command indicated by the motion of the object recognized by the motion recognition unit.

Claims

1. A 3D human interface apparatus for facilitating 3D pointing and controlling command inputs by an operator, comprising:

image input means for entering a plurality of time series images of an object operated by the operator into a motion representing a command;

feature point extraction means for extracting at least four feature points including at least three reference feature points and one fiducial feature point on the object, from each of said images;

motion recognition means for recognizing the motion of the object by calculating motion parameters, according to an arline transformation determined from changes of positions of the reference feature points on said images, and a virtual parallax for the fiducial feature point expressing a difference between an actual position change on said images and a virtual position change according to the affine transformation; and

command input means for inputting the command indicated by the motion of the object recognized by the motion recognition means.

2. The apparatus of claim 1, wherein the motion recognition means calculates the motion parameters for a relative motion of an observer taking said images with respect to the object, according to the affine transformation which represents the relative motion.

3. The apparatus of claim 2, wherein the motion recognition means calculates the motion parameters by eliminating factors not due to the motion of the object from invariant features derived from the affine transformation.

4. The apparatus of claim 3, wherein the motion recognition means eliminates the factors not due to the motion of the object which are obtained by using a translational motion direction on said images determined from the virtual parallax.

5. The apparatus of claim 1, wherein the feature point extraction means also extracts a multiplicity of feature points distributed over an entire imaging field of said images, and which further comprises:

object detection means for detecting the object in said images by calculating translational motion directions on said images for said multiplicity of feature points, plotting a distribution plot of the translational motion directions over the entire imaging field, and separating an area having identical and distinctive translational motion directions distributed therein in the distribution plot as the object.

6. The apparatus of claim 5, wherein the object detection means calculates the translational motion directions according to virtual parallaxes for said multiplicity of feature points, each virtual parallax expressing a difference between an actual position change on said images and a virtual position change according to an affine transformation determined from changes of positions of said multiplicity of feature points on said images.

7. The apparatus of claim 1, wherein the feature point extraction means also extracts a multiplicity of feature points distributed over the object in said images, and which further comprises:

structure extraction means for extracting a structure of the object by determining relative positions of said multiplicity of feature points in a depth direction on said images according to a ratio of the virtual parallaxes for each two closely located ones of said multiplicity of feature points.

8. The apparatus of claim 7, wherein the structure extraction means also determines a texture data indicating a gradient of the object surface according to the determined relative positions of said multiplicity of feature points.

9. The apparatus of claim 1, wherein the command input means determines the command indicated by the motion by generating a gesture pattern expressing the motion and making a matching of the generated gesture pattern with one of predetermined gesture patterns corresponding to predetermined commands.

10. The apparatus of claim 1, further comprising:

display means for displaying a 3D model which moves in accordance with the motion indicated by the command inputted by the command input means.

11. The apparatus of claim 10, wherein the display means also displays said images in superposition to the 3D model.

12. A 3D human interface apparatus for facilitating 3D pointing and controlling command inputs by an operator, comprising:

image input means for entering a plurality of time series images of an object operated by the operator into a motion representing a command;

feature point extraction means for extracting a multiplicity of feature points distributed over an entire imaging field of said images;

object detection means for detecting the object in said images by calculating translational motion directions on said images for said multiplicity of feature points, plotting a distribution plot of the translational motion directions over the entire imaging field, and separating an area having identical and distinctive translational motion directions distributed therein in the distribution plot as the object;

motion recognition means for recognizing the motion of said area separated as the object by image processing said images; and

command input means for inputting the command indicated by the motion of the object recognized by the motion recognition means.

13. The apparatus of claim 12, wherein the object detection means calculates the translational motion directions according to virtual parallaxes of the feature points, each virtual parallax expressing a difference between an actual position change on said images and a virtual position change according to an arline transformation determined from changes of positions of the feature points on said images.

14. A 3D human interface apparatus for facilitating 3D pointing and controlling command inputs by an operator, comprising:

image input means for entering a plurality of images of an object operated by the operator into a motion representing a command;

feature point extraction means for extracting a multiplicity of feature points distributed over the object in said images;

structure extraction means for extracting a structure of the object by determining relative positions of said multiplicity of feature points in a depth direction on said images according to a ratio of virtual parallaxes for each two closely located ones of said multiplicity of feature points, each virtual parallax expressing a difference between an actual position change on said images and a virtual position change according to an arline transformation determined from changes of positions of the feature points on said images;

motion recognition means for recognizing the motion of the object by image processing said images;

command input means for inputting the command indicated by the motion of the object recognized by the motion recognition means; and

display means for displaying a 3D model having the structure of the object as extracted by the structure extraction means which moves in accordance with the motion indicated by the command inputted by the command input means.

15. The apparatus of claim 14, wherein the structure extraction means also determines a texture data indicating a gradient of the object surface according to the determined relative positions of said multiplicity of feature points.