Interface system

• US 20060168523A1
• Filed: 12/17/2003
• Published: 07/27/2006
• Est. Priority Date: 12/18/2002
• Status: Active Grant

• Current Assignee: National Institute of Advanced Industrial Science and Technology (Government of Japan)
• Original Assignee: National Institute of Advanced Industrial Science and Technology (Government of Japan)
• Inventors: Sakaue, Katsuhiko, Yoda, Ikushi
• Granted Patent: US 7,821,531 B2
• Time in Patent Office: Days
• US Class Current: 715/728
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

To provide an interface apparatus capable of achieving noncontact and unrestricted arm pointing actions of multiple users in an indoor space, and facilitating the recognition of all typical arm pointing actions in standing, sitting, and lying postures and the operation of indoor units in the indoor space by the arm pointing actions. The interface apparatus includes: image processing means for picking up images of the interior of an indoor space 5 with a plurality of stereo cameras 1-1 to 1-n, and producing a distance image based on the picked-up images within the visual field on a camera-by-camera basis and the coordinate system of the indoor space 5; means for extracting the posture and arm pointing of a user 4 from the distance information from the stereo cameras 1-1 to 1-n; and means for determining, when the arm pointing has been identified, whether the arm pointing is an intended signal, from the direction pointed by the arm and the motion of the arm.

Claims

1. An interface apparatus comprising:

(a) image processing means for picking up images of the interior of an indoor space with a plurality of stereo cameras, and producing a distance image based on the picked up images within the visual field and an indoor coordinate system on a camera-by-camera basis;

(b) positional-posture and arm-pointing recognition means for extracting the positional posture and arm pointing of a user from the distance information from the stereo cameras; and

(c) pointing-action recognition means for determining, when arm pointing by the user has been identified, whether or not the arm pointing is a pointing action, from the pointing direction and the motion of the arm.

2. The interface apparatus according to claim 1, further comprising an operation-command transmitter-receiver that transmits a command for operating a registered indoor device according to the operation of the pointing-action recognition means and receives the result.

3. The interface apparatus according to claim 1, wherein the positional-posture and arm-pointing recognition means includes a positional-posture recognition section that performs a posture recognition process for multiple users by extracting the distance data from the collected three-dimensional information along the indoor-space coordinate system by a different-level extracting method and projecting the data onto a two-dimensional plane, and an arm-pointing recognition section that determines the direction of the arm pointing from the obtained two-dimensional projection drawing of the multiple users.

4. The interface apparatus according to claim 1, wherein the pointing-action recognition means includes an pointing-action recognition section that recognizes the pointing action from time-series data on the obtained arm-pointing direction, an operating-unit database that stores information on the operation object unit that is an object of arm pointing and an operation method for the same, and an operating-unit registration section that stores basic information, position, and an operating method of the operation object unit.

5. The interface apparatus according to claim 1, wherein the positional-posture and arm-pointing recognition means for extracting arm pointing recognizes a pointing action in such a way that:

the recognition means divides three-dimensional distance information obtained from the stereo cameras into levels by 20 cm according to the indoor coordinate system by a different-level extracting method;

projects a dot sequence in each level onto a two-dimensional plane and then binarizes it into a two-dimensional binary image;

labels the images on a level-to-level basis;

determines the overall center of gravity of the clusters;

stacks the center of gravity determined in each two-dimensional plane in levels on object-to-object basis again to use as a three-dimensional dot sequence;

plots the center of gravity of each level along the Z-axis, in which eight levels (the upper half of a body) from the uppermost level (the head) are plotted on the X-Y plane;

wherein when there is a large outlier compared to the overall center of gravity, determines that an arm-pointing action has been made; and

determines the direction of the body by calculating the image moments of the binary images of the eight levels from the uppermost, the image moment being a rectangle equivalent to the binary image, and determining the vertical direction of the long side of a level having an area within a specified range and in which the difference between the long side and the short side of the image moment is the largest of the acquired eight sets of data as the direction of the body;

when the arm-pointing action has been recognized, the recognition means determines the direction of the arm pointing in such a way that;

it determines the direction of the arm pointing on the X-Y plane by drawing a perpendicular bisector between the overall center of gravity and the center of gravity of the binary image of a level of which the center of gravity is farthest from the overall center of gravity and the overall center of gravity, erasing the binary image in the region corresponding to the body of the person to leave only the image of the arm;

calculates the image moment of the binary image of only the arm to determine the long side, the center of gravity, the position of the distal end of the arm, and the overall center of gravity; and

determines the Z-direction of the arm pointing from the stature, the height of eyes, and arm-length coefficient.

6. The interface apparatus according to claim 5, wherein the lower limit of determination on arm pointing is set from the head height and the height corresponding to the sitting height, wherein false arm pointing which is sensed lower than the lower limit is determined not to be arm pointing.

7. The interface apparatus according to claim 6, wherein false arm pointing which is sensed lower than the lower limit is the case in which the user stretches out his leg.

8. The interface apparatus according to claim 5, wherein when the ratio of the long side of the image moment to the stature is less than a given value, it is determined not to be arm pointing.

9. The interface apparatus according to claim 8, wherein slight arm pointing comes under.

10. The interface apparatus according to claim 5, wherein when r1/r2 is smaller than or equal to a value set from the stature, it is determined not to be arm pointing, where r1 is the distance from an average center of gravity to the distal end of the arm pointing and r2 is the distance from the average center of gravity to the base end of the arm pointing.

11. The interface apparatus according to claim 10, wherein slight arm pointing comes under.

12. The interface apparatus according to claim 5, wherein the area S of the image moment is determined from the long side L1 and the short side L2, and the upper limit is set for the area S and the lower limit is set for the long side L1, wherein, when the area S or the long side L1 is outside the limit, it is determined not to be arm pointing.

13. The interface apparatus according to claim 12, wherein when the area S or the long side L1 is outside the limit, it is determined that both arms are spread out.

14. The interface apparatus according to claim 5, wherein when the ratio of the distance between the distal end of the arm pointing and an average center of gravity to the distance between the base end of the arm pointing and the average center of gravity is greater than a set value, it is determined not to be arm pointing.

15. The interface apparatus according to claim 14, wherein when the ratio of the distance between the distal end of the arm pointing and an average center of gravity to the distance between the base end of the arm pointing and the average center of gravity is greater than a set value, it is determined that both arms are spread out.

16. The interface apparatus according to claim 5, wherein when the average center of gravity of a partner is found within a specified radius around the distal end of the user'"'"'s arm pointing, it is determined not to be arm pointing.

17. The interface apparatus according to claim 1, wherein a specified area in the indoor space is registered in advance, wherein when a user is present in the specified area, the arm pointing of the user is identified.

18. The interface apparatus according to claim 17, wherein, with the periphery of the head part on a nursing bed being set as the specified area, when a user is present in the specified area, arm pointing for the specified area is identified.

19. The interface apparatus according to claim 18, wherein, with the periphery of the head part on a nursing bed being set as the specified area, when no user is present in the specified area, it is determined whether the user is in a standing, sitting, or lying posture and then arm pointing is identified for the respective postures.