Hand pointing apparatus

• US 6,434,255 B1
• Filed: 10/23/1998
• Issued: 08/13/2002
• Est. Priority Date: 10/29/1997
• Status: Expired due to Fees

• Current Assignee: Takenaka Corporation
• Original Assignee: Takenaka Corporation
• Inventors: Harakawa, Kenichi
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US09/177,428
• Time in Patent Office: 1,390 Days
• Field of Search: 382/103, 382/153, 382/141, 382/291, 382/107, 396/330, 364/513
• US Class Current: 382/103
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

An image which represents a 3-D space virtually using a perspective method is displayed on a display, and an image of an information inputting person is picked up by a plurality of video cameras from different directions. While the information inputting person is pointing at an arbitrary position within a virtual 3-D space represented by the displayed image, a reference point which corresponds to the back of the information inputting person and a characteristic point which corresponds to a finger tip are respectively extracted from a plurality of images which are picked up by each of the video cameras, and the 3-D coordinates of these points are determined. The direction in which the position pointed to by the information inputting person is disposed within the virtual 3-D space is determined on the basis of the direction from the reference point to the characteristic point, and the distance between the information inputting person and the position pointed to by the information inputting person within the virtual 3-D space is determined on the basis of the distance between the reference point and the characteristic point. As a result, the 3-D coordinates of the position pointed to by the information inputting person within the virtual 3-D space can be determined.

Claims

1. A hand pointing apparatus, comprising:

a plurality of image pickup devices for picking up, from a plurality of different directions, a plurality of images including a person to be recognized;

a computing section which (1) extracts a plurality of image portions which correspond to the person to be recognized on the basis of the plurality of images obtained by said plurality of image pickup devices respectively, the plurality of images including the person to be recognized pointing to a specific point which is within one of a 3-D space to be pointed by the person to be recognized or a dummy 3-D space to be pointed by the person to be recognized, and (2) determines a coordinate of a position of a characteristic point related to the person to be recognized, the position of the characteristic point changing by the person to be recognized bending or stretching his or her arm, and determines a coordinate of a position of a reference point, the position of the reference point not changing even when the person to be recognized bends or stretches the arm, on the basis of the extracted image portions;

a determining section for (1) determining a specific point direction in which direction the specific point exists, on the basis of a direction from the reference point to the characteristic point, and (2) determining a position of the specific point along the specific point direction, on the basis of a distance between the reference point and the characteristic point, and thereby the determining section determining a 3-D coordinate of the specific point within one of the 3-D space or the dummy 3-D space; and

a processing section which detects a speed at which the distance between the reference point and the characteristic point changes, and executes a predetermined process when the detected speed at which the distance between the reference point and the characteristic point changes is greater than or equal to a threshold value.

2. A hand pointing apparatus according to claim 1, wherein said determining means determines the location of said specific position within said 3-D space in the depth direction thereof by converting the distance between said reference point and said characteristic point into the distance between said person to be recognized and said specific position in accordance with predetermined conversion conditions.

3. A hand pointing apparatus according to claim 2, further comprising:

conversion conditions setting means which requests said person to be recognized to carry out said arm bending or stretching action, and sets in advance said conversion conditions which convert the distance between said reference point and said characteristic point into the distance between the person to be recognized and said specific position on the basis of the extent of the change in the distance between said reference point and said characteristic point when said person to be recognized carries out said arm bending or stretching action.

4. A hand pointing apparatus according to claim 1, further comprising:

display means which displays a three dimensional image;

display control means which displays said three dimensional image on said display means; and

marker display means which displays a marker having an arbitrary shape which can be easily recognized by said person to be recognized,

wherein said three dimensional image is an image which represents a virtual 3-D space, and includes an image which is formed conforming to a one-point perspective method or two-point perspective method, and image which uses a liquid crystal shutter or a lenticular lens, and a stereographic image which is displayed by applying holographic technology, and

said conversion conditions setting means causes the person to be recognized to point to a single position which is marked by said marker which is positioned at the intermediate portion of said 3-D space along the depth direction thereof, and sets said conversion conditions on the basis of the distance between said reference point and said characteristic point when the person to be recognized points to the single position which is marked by the marker which is positioned at the intermediate portion of the 3-D space along the depth direction thereof.

5. A hand pointing apparatus according to claim 3, further comprising:

display means which displays a three dimensional image;

display control means which displays said three dimensional image on said display means; and

marker display means which displays a marker having an arbitrary shape which can be easily recognized by said person to be recognized,

wherein said three dimensional image is an image which represents a virtual 3-D space, and includes an image which is formed conforming to a one-point perspective method or two-point perspective method, an image which uses a liquid crystal shutter or a lenticular lens, and a stereographic image which is displayed by applying holographic technology, and

said conversion conditions setting means causes said person to be recognized to point to a plurality of different positions which are marked by said marker along the depth direction of said 3-D space, and sets said conversion conditions on the basis of the distance between said reference point and said characteristic point.

6. A hand pointing apparatus according to claim 3, wherein said conversion conditions are conversion characteristics in which the distance between said person to be recognized and the specific position within said 3-D space which is pointed to by said person to be recognized changes linearly in response to the change in the distance between said reference point and said characteristic point.

7. A hand pointing apparatus according to claim 3, wherein said conversion conditions are conversion characteristics in which the distance between said person to be recognized and the specific position within said 3-D space which is pointed to by said person to be recognized changes non-linearly in response to the change in the distance between said reference point and said characteristic point.

8. A hand pointing apparatus according to claim 3, comprising:

display means for displaying a three dimensional image; and

display control means which displays said three dimensional image on said display means,

wherein said three dimensional image is an image which represents a virtual 3-D space, and includes an image which is formed conforming to a one-point perspective method or two-point perspective method, an image which uses a liquid crystal shutter or a lenticular lens, and a stereographic image which is displayed by applying holographic technology, and

said conversion conditions setting means corrects said conversion conditions in accordance with the variation of the display magnification of said three dimensional image which is displayed on said display means in an enlarged or reduced state.

9. A hand pointing apparatus according to claim 3, comprising:

display means for displaying a three dimensional image; and

display control means which displays said three dimensional image on said display means,

wherein said three dimensional image is an image which represents a virtual 3-D space, and includes an image which is formed conforming to a one-point perspective method or two-point perspective method, an image which uses a liquid crystal shutter or a lenticular lens, and a stereographic image which is displayed by applying holographic technology, and

the display magnification of said three dimensional image which is displayed on said display means may vary linearly in response to the change in the distance between said reference point and said characteristic point.

10. A hand pointing apparatus according to claim 3, comprising:

display means for displaying a three dimensional image; and

display control means which displays said three dimensional image on said display means,

wherein said three dimensional image is an image which represents a virtual 3-D space, and includes an image which is formed conforming to a one-point perspective method or two-point perspective method, an image which uses a liquid crystal shutter or a lenticular lens, and a stereographic image which is displayed by applying holographic technology, and

the display magnification of said three dimensional image which is displayed on said display means may vary non-linearly in response to the change in the distance between said reference point and said characteristic point.

11. A hand pointing apparatus according to claim 1, further comprising:

display means which displays a three dimensional image;

display control means which displays said three dimensional image on said display means; and

determining means which determines whether the hand of said person to be recognized is in a specific shape,

wherein said three dimensional image is an image which represents a virtual 3-D space, and includes an image which is formed confirming to a one-point perspective method or two-point perspective method, an image which uses a liquid crystal shutter or a lenticular lens, and a stereographic image which is displayed by applying holographic technology, and said person to be recognized points to a specific position within said virtual 3-D space which is represented by said three dimensional image which is displayed on said display means, and

in a state in which the hand of said person to be recognized is determined to be in a specific shape, when the distance between said reference point and said characteristic point changes, said display control means controls said display means such that said three dimensional image displayed on said display means is displayed so as to be enlarged or reduced according to the change in the distance between said reference point and said characteristic point.

12. A hand pointing apparatus according to claim 11, wherein said display control means displays said three dimensional image on said display means so as to be enlarged or reduced by changing the display magnification of said three dimensional image which is displayed on said display means in response to the change in the distance between said reference point and said characteristic point.

13. A hand point apparatus according to claim 1, wherein said processing means executes a fist predetermined process when the distance between said reference point and said characteristic point increases at a speed which is greater than or equal to said threshold value, and executes a second predetermined process which is different from the first predetermined process when the distance between said reference point and said characteristic point decreases at a speed which is greater than or equal to said threshold value.

14. A hand pointing apparatus according to claim 1, further comprising:

threshold value setting means which requests said person to be recognized to carry out the arm bending or stretching action to cause said processing mean to execute said predetermined process, and hereby sets said threshold value in advance on the basis of the speed at which the distance between said reference point and said characteristic point changes when said person to be recognized carries out the arm bending or stretching action.

15. A hand pointing apparatus according to claim 14, wherein said threshold value setting means sets, as said threshold value, a value which is less than the mean value of the speed at which said distance changes when said person to be recognized has carried out the arm bending or stretching action.

16. A hand pointing apparatus according to claim 14, wherein said threshold value setting means sets, as said threshold value, the minimum value of the speed at which said distance changes when said person to be recognized carries out the arm bending or stretching action.

17. A hand pointing apparatus according to claim 14, wherein said threshold value setting means sets said threshold value anew each time a person to be recognized is replaced by another person to be recognized.

18. A hand pointing apparatus, comprising:

an image pickup device for picking up a plurality of images including a person to be recognized;

a computing section which (1) extracts a plurality of image portions which correspond to the person to be recognized on the basis of the plurality of images obtained by said image pickup device the plurality of images including the person to be recognized pointing to a specific point which is within one of a 3-D space to be pointed by the person to be recognized or a dummy 3-D space to be pointed by the person to be recognized, and (2) determines a coordinate of a position of a characteristic point related to the person to be recognized, the position of the characteristic point changing by the person to be recognized bending or stretching his or her arm, and determines a coordinate of a position of a reference point, the position of the reference point not changing even when the person to be recognized bends or stretches the arm, on the basis of the extracted image portions;

a determining section for (1) determining a specific point direction in which direction the specific point exists, on the basis of a direction from the reference point to the characterisitc point, and (2) determining a position of the specific point along the specific point direction, on the basis of a distance between the reference point and the characteristic point, and thereby the determining section determining a 3-D coordinate of the specific point within one of the 3-D space or the dummy 3-D space; and

a processing section which detects a speed at which the distance between the reference point and the characteristic point changes, and executes a predetermined process when the detected speed at which the distance between the reference point and the characteristic point changes is greater than or equal to s threshold value.

19. A hand pointing apparatus according to claim 18, further comprising a light reflecting device, wherein one of the plurality of images including the person to be recognized is a reflected image on said light device.

20. A hand pointing apparatus, comprising:

a plurality of image pickup devices for picking up, from a plurality of different directions, a plurality of images including a person to be recognized, the person to be recognized pointing to a specific point which is within one of a 3-D space to be pointed by the person to be recognized or a dummy 3-D space to be pointed by the person to be recognized or a dummy 3-D space to be pointed by the person to be recognized;

a computing section which (1) extracts, from the plurality of images picked up from the plurality of image pickup devices, a plurality of image portions which correspond to the person to be recognized, and (2) determines, on the basis of the extracted image portions, a coordinate of a position of a characteristic point related to the person to be recognized, the position of the characteristic point changing by the person to be recognized moving a portion of a body of the person to be recognized, and determines a coordinate of a position of a reference point, the position of the reference point not changing even when the person to be recognized moves the portion of the body of the person to be recognized;

a determining section for (1) determining a specific point direction in which direction the specific point exists, on the basis of a direction in which direction the specific point exists, on the basis of a direction from the reference point to the characteristic point, and (2) determining a position of the specific point along the specific point direction, on the basis of a distance between the reference point and the characteristic point, thereby the determining section determining a 3-D coordinate of the specific point within one of the 3-D space or the dummy 3-D space; and

a processing section which detects a speed at which the distance between the reference point and the characteristic point changes, and executes a predetermined process when the detected speed at which the distance between the reference point and the characteristic point changes is greater than or equal to a threshold value.

21. A hand pointing apparatus, comprising:

a plurality of image pickup devices for picking up, from a plurality of different directions, a plurality of images including a person to be recognized;

a computing section which (1) extracts a plurality of image portions which correspond to the person to be recognized on the basis of the plurality of images obtained by said plurality of image pickup devices restively, the plurality of images including the person to be recognized pointing to a specific point which is with one of a 3-D space to be pointed by the person to be recognized or a dummy 3-D space to be pointed by the person to be recognized (2) determines a coordinate of a position of a characteristic point related to the person to be recognized, the position of the characteristic point changing by the person to be recognized bending or stretching his or her arm, and determines a coordinate of a position of a reference point, the position of the reference point not changing even when the person to be recognized bends or stretches the arm, on the basis of the extracted image portions, and calculating a distance between the characteristic point and the reference point;

a determining section for (1) determining a specific point direction in which direction the specific point exists, on the basis of a direction from the reference pit to the characteristic point, and (2) determining a position of the specific point along the specific point direction, by corresponding the calculated distance between the reference point and the characteristic point to a distance between the person and the specific point, and thereby the determining section determining a 3-D coordinate of the specific point within one of the 3-D space or the dummy 3-D space.

22. A hand pointing apparatus, comprising:

an image pickup device for picking up a plurality of images including a person to be recognized;

a computing section which (1) extracts a plurality of image portions which correspond to the person to be recognized on the basis of the plurality of images obtained by said image pickup device, the plurality of images including the person to be recognized pointing to a specific point which is within one of a 3-D space to be pointed by the person to be recognized or a dummy 3-D space to be pointed by the person to be recognized, (2) determines a coordinate of a position of a characteristic point related to the person to be recognized, the position of the characteristic point changing by the person to be recognized bending or stretching his or her arm, and determines a coordinate of a position of a reference point, the position of the reference point not changing even when the person to be recognized bends or stretches the arm, on the basis of the extracted image portions, and calculating a distance between the characteristic point and the reference point;

a determining section for (1) determining a specific point direction in which direction the specific point exists, on the basis of a direction from the reference point to the characteristic point, and (2) determining a position of the specific point along the specific point direction, by corresponding the calculated distance between the reference point and the characteristic point to a distance between the person and the specific point, and thereby the determining section determining a 3-D coordinate of the specific point within one of the 3-D space or the dummy 3-D space.