Hand pointing device

• US 6,385,331 B2
• Filed: 03/18/1998
• Issued: 05/07/2002
• Est. Priority Date: 03/21/1997
• Status: Expired due to Fees

• Current Assignee: Takenaka Corporation
• Original Assignee: Takenaka Corporation
• Inventors: Harakawa, Kenichi, Unno, Kenichi, Igawa, Norio
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US09/040,436
• Publication Number:
  US 20010043719A1
• Time in Patent Office: 1,511 Days
• Field of Search: 345/137, 345/158, 345/156, 345/157, 345/355, 345/329, 345/358, 345/173, 345/330, 345/331, 434/45, 382/291, 382/106, 382/107, 382/105, 382/203, 382/294, 382/295, 382/279, 710/74, 178/18.01, 178/18, 178/19, 600/595, 395/2.8, 364/410, 351/210, 356/34
• US Class Current: 382/106
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Over an information input space to which an information inputting person comes, a pair of near-infrared light illuminators are arranged in such a manner that the illumination ranges thereof are adjusted so as to illuminate the information inputting person from different directions. A pair of near-infrared-light-sensitive video cameras are also arranged in different positions so as to correspond to the illuminators. The image pickup range of the video cameras is adjusted so that it is out of the range on the floor surface illuminated by the corresponding illuminator, while the information inputting person is within the image pickup range. A controller allows one illuminator at a time to be switched on/off. An image of the information inputting person is picked up by the video camera corresponding to the switched-on illuminator. The information inputting person is extracted based on the images picked up by the video cameras, whereby the position or direction pointed to by the information inputting person is determined.

Claims

1. A hand pointing apparatus comprising:

an illuminating device for illuminating a person to recognized;

a plurality of image pickup devices, each of which picks up said person to be recognized, located in different positions, wherein said illuminating device and said plurality of image pickup devices are adjusted such that said person to be recognized, who is illuminated by said illuminating device, is within each of image pickup ranges corresponding to a respective one of said plurality of image pickup devices, and an illuminated range on a floor surface, which is illuminated by said illuminating device, is out of the pickup range of each of the image pickup ranges; and

a determiner for determining one of a position or a direction pointed to by said person to be recognized on the basis of a plurality of images of situation picked up by said plurality of image pickup devices, the situation being indicative of said person to be recognized pointing to one of the position or the direction, by extracting an image part corresponding to said person to be recognized from the plurality of images.

2. A hand pointing apparatus according to claim 1, further comprising:

a preventer for preventing an object which is not a subject to be recognized from remaining on the floor surface around said person to be recognized.

3. A hand pointing apparatus comprising:

a plurality of illuminating devices for illuminating a person to be recognized form different directions;

a plurality of image pickup devices located in different positions,

a controller for switching on/off said plurality of illuminating devices one by one in sequence, and for effecting control such that an image of said person to be recognized pointing to one of a position or a direction, which person is illuminated by a switched-on illuminating device among said plurality of illuminating devices, is picked up; and

a determiner for determining one of the position or the direction pointed to by said person to be recognized, on the basis of a plurality of images picked up by said plurality of image pickup devices, by extracting an image part corresponding to said person to be recognized from the plurality of images,

wherein, each of said plurality of illuminating devices has an illuminated range on a floor surface in which said person to be recognized exists,

each of said plurality of image pickup devices has an image pickup range,

each of said plurality of illuminating devices corresponds to a respective one of said plurality of image pickup devices,

an illuminating device illuminates said person to be recognized, and the illuminating device and an image pickup device corresponding to the illuminating device are adjusted such that

said person to be recognized, who is illuminated by the illuminating device, is within an image pickup range of the image pickup device, corresponding to the illuminating device, and

an illuminated range on the floor surface of the illuminating device is out of the pickup range of the image pickup device corresponding to the illuminating device.

4. A hand pointing apparatus according to claim 3, further comprising:

a preventer for preventing an object which is not a subject to be recognized from remaining on the floor surface around said person to be recognized.

5. A hand pointing apparatus according to claim 3, wherein said plurality of illuminating devices include at least a first illuminating device and a second illuminating device and plurality of image pickup devices include at least a first image pickup device and a second image pickup device, said first illuminating device corresponds to said first image pickup device and said second illuminating device corresponds to said second image pickup device, said first illuminating device and said first image pickup device are adjusted such that said person to be recognized, who is illuminated by said first illuminating device is within an image pickup range of said first image pickup device, and an illuminated range on a floor surface of said first illuminating device is out of the image pickup range of said first image pickup device, said second illuminating device and said second image pickup device are adjusted such that said person to be recognized, who is illuminated by said second illuminating device is within an image pickup range of said second image pickup device, and an illuminated range on the floor surface of said second illuminating device is out of the image pickup range of said second image pickup device.

6. A hand pointing apparatus comprising:

a plurality of illuminating devices for illuminating a person to be recognized from different directions;

at least one image pickup device for picking up an image of said person to be recognized, who is illuminated by said illuminating device;

a discriminator for switching on/off said plurality of illuminating devices one by one in sequence, for comparing a plurality of images of said person to be recognized pointing to one of a position or a direction picked up by said at least one image pickup devices, each of said plurality of images being picked up during respective said plurality of illuminating devices are switched on, and for discriminating between an image part corresponding to said person to be recognized and an image part other than the image part corresponding to said person to be recognized, from said plurality of images, for said at least one image pickup device; and

a determiner for extracting the image part corresponding to said person to be recognized from said plurality of images picked up by said at least one image pickup device based on a result of a discrimination by said discriminator, and for determining one of the position or the direction pointed to by said person to be recognized,

wherein said discriminator generates image data, on the basis of a predetermined condition, from said plurality of images, and on the basis of the image data, said discriminator discriminates between the image part corresponding to said person to be recognized and the image part other than the image part corresponding to said person to be recognized,

said discriminator switches on/off said plurality of illuminating devices such that, during a predetermined period, the image part corresponding to said person to be recognized has higher luminance than that of the image part other than the image part corresponding to said person to be recognized due to said plurality of illuminating devices that are switched on during said predetermined period, and

wherein an illuminated range on the floor surface of each of the plurality of illuminating devices is different from one another.

7. A hand pointing apparatus according to claim 6, further comprising:

a preventer for preventing an object which is not a subject to be recognized from remaining on the floor surface around said person to be recognized.

8. A hand pointing apparatus comprising:

an illuminating device for illuminating a person to be recognized who arrives at a predetermined region;

a plurality of image pickup devices for picking up an image of said person to be recognized, who is illuminated by said illuminating device, from different directions;

a positioner for positioning, beginning at a start position and ending at a termination position, a plurality of points in a plane in said predetermined region, wherein said positioner successively positions said plane at different positions in said predetermined region, wherein successive positions are located at a fixed distance from each other;

storage for storing information for corresponding three-dimensional coordinates of the plurality of points, positioned within said predetermined region, to two-dimensional coordinates of said plurality of points which the two-dimensional coordinates are in said plurality of images picked up by said plurality of image pickup devices; and

a determiner for extracting an image part corresponding to said person to be recognized from a plurality of images based on a plurality of images of situations picked up by said plurality of image pickup devices, the situations being indicative of said person to be recognized pointing to one of a position or a direction;

for determining the two-dimensional coordinates of a feature point of said person to be recognized in each of said images;

for determining the three-dimensional coordinates of the feature point based on the determined two-dimensional coordinates of the feature point and the information stored in said storage; and

for determining one of the position or the direction pointed to by said person to be recognized based on the determined three-dimensional coordinates of the feature point.

9. A hand pointing apparatus according to claim 8, further comprising:

a preventer for preventing an object which is not a subject to be recognized from remaining on the floor surface around said person to be recognized.

10. A hand pointing apparatus according to claim 8, wherein said determiner determines the two-dimensional coordinates of the feature point of said person to be recognized in said images, extracts from said images the points positioned in a region within a predetermined range including said feature point on said images from said images, and determines the three-dimensional coordinates of said feature point on the three-dimensional coordinated of the common points extracted from said images.

11. A hand pointing apparatus according to claim 8, further comprising:

a generator for allowing said plurality of image pickup devices to pick up images of the situations where markers are positioned in the two-dimensional coordinates of said points, for generating the information for corresponding the three-dimensional coordinates of said points to the two-dimensional coordinates of said points on said images, based on the three-dimensional coordinates of said points and the marker positions on said images picked up by said plurality of image pickup devices, and for allowing said storage to store the generated information.

12. A hand pointing apparatus comprising:

an illuminating device for illuminating a person to be recognized;

a plurality of image pickup devices for picking up images of said person to be recognized, who is illuminated by said illuminated device, from different directions;

a determiner for extracting an image part corresponding to said person to be recognized from a plurality of images based on a plurality of images of situations picked up by said plurality of image pickup devices, the situations being indicative of said person to be recognized pointing to one of a position or a direction, and for determining one of the position or the direction pointed to by said person to be recognized;

a first detector (1) for extracting, from said plurality of images, the image part corresponding to a predetermined part of a body of said person to be recognized, and calculating at least one of an area of the extracted image part and outline of the extracted image part and a length of an outline of the extracted image part so as to obtain a first calculation value, (2) after a predetermined time elapses for extracting from said plurality of images, the image part corresponding to the predetermined part of the body of said person to be recognized and calculating the at least one of an area of the extracted image part, and outline of the extracted image part and a length of an outline of the extracted image part so as to obtain a second calculation value and (3) detecting a changed value between the first calculation value and the second calculation value; and

a processor for executing a predetermined processing when the changed value detected by said first detector is more than or equal to a predetermined changed value.

13. A hand pointing apparatus according to claim 12, further comprising:

a second detector for extracting the image part corresponding to the arm of said person to be recognized from said plurality of images, and for detecting whether or not the arm of said person to be recognized is lowered,

wherein said processor continues the current state when said second detecting means detects that the arm of said person to be recognized is lowered.

14. A hand pointing apparatus according to claim 12, further comprising:

a preventer for preventing an object which is not a subject to be recognized from remaining on the floor surface around said person to be recognized.

15. A hand pointing apparatus comprising:

an illuminating device for illuminating a person to be recognized;

a plurality of image pickup devices for picking up images of said person to be recognized, who is illuminated by said illuminating device from different directions;

a determiner for extracting an image part corresponding to said person to be recognized from a plurality of images based on a plurality of images of situations picked up by said plurality of image pickup devices, the situations being indicative of said person to be recognized pointing to either a specific position or a specific direction, for determining the three-dimensional coordinates of the feature point whose position is changed when said person to be recognized bends or extends an arm, and the three-dimensional coordinates of a reference point whose position is not changed even if said person to be recognized bends and extends an arm, and for determining either the position or the direction pointed to by said person to be recognized based on the three-dimensional coordinates of the feature point and the three-dimensional coordinates of the reference point; and

a processor for calculating the distance between said reference point and said feature point and for executing a predetermined processing based on the change in the distance between said reference point and said feature point.

16. A hand pointing apparatus according to claim 15, wherein said processor performs a first predetermined processing when the distance between said reference point and said feature point is increased, and performs a second predetermined processing differing from said first predetermined processing when the distance between said reference point and said feature point is reduced.

17. A hand pointing apparatus according to claim 15, wherein said processor detects a rate of change in the distance between said reference point and said feature point, and executes a predetermined processing when the detected rate of change is a threshold value or more.

18. A hand pointing apparatus according to claim 16, wherein said processor detects a rate of change in the distance between said reference point and said feature point, and executes a predetermined processing when the detected rate of change is a threshold value or more.

19. A hand pointing apparatus according to claim 17, further comprising:

a threshold value setter for requesting said person to be recognized to bend or extend an arm in order to allow said processor to perform a predetermined processing, and for setting said threshold value based on the rate of the change in the distance between said reference point and said feature point when said person to be recognized bends or extends an arm.

20. A hand pointing apparatus according to claim 18, further comprising:

a threshold value setter for requesting said person to be recognized to bend or extend an arm in order to allow said processing means to perform a predetermined processing, and for setting said threshold value based on the rate of the change in the distance between said reference point and said feature point when said person to be recognized bends or extends an arm.

21. A hand pointing apparatus according to claim 15, further comprising:

a second detector for extracting the image part corresponding to the arm of said person to be recognized from said plurality of images, and for detecting whether or not the arm of said person to be recognized is lowered,

wherein said processor continues the current state when said second detector detects that the arm of said person to be recognized is lowered.

22. A hand pointing apparatus according to claim 15, further comprising:

a preventer for preventing an object which is not a subject to be recognized from remaining on the floor surface around said person to be recognized.

23. A hand pointing apparatus comprising:

illuminating means for illuminating a person to be recognized;

pickup means for picking up images of said person to be recognized, who is illuminated by said illuminating means;

determining means for extracting an image part corresponding to said person to be recognized from a plurality of images based on a plurality of images of situations picked up by said image pickup means, the situations being that a member relating to said person to be recognized is moved; and

processing means for executing a predetermined processing when said member is moved with a predetermined condition.

24. A hand pointing apparatus according to claim 23, wherein said predetermined condition is such that said member is moved more than a predetermined speed.

25. A hand pointing apparatus according to claim 23, wherein said member is a hand of said person to be recognized.