Instruction recognition system having gesture recognition function

• US 6,677,969 B1
• Filed: 09/24/1999
• Issued: 01/13/2004
• Est. Priority Date: 09/25/1998
• Status: Expired due to Fees

• Current Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Inventors: Hongo, Hitoshi
• Primary Examiner(s): Kincaid, Kristine
• Assistant Examiner(s): TRAN, MYLINH T
• Application Number:
  US09/404,877
• Time in Patent Office: 1,572 Days
• Field of Search: 345/863, 345/700, 345/179, 345/156, 345/860, 345/157, 345/701, 345/702, 382/186, 382/187, 382/202, 382/118, 704/251
• US Class Current: 715/863
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

There is provided a recognition system capable of correctly recognizing nodding action of a user and controlling interaction reflecting the user'"'"'s intention. The instruction recognition system includes a gesture recognition section for recognizing the user'"'"'s nodding action on the basis of a photographed image of the user'"'"'s head; a line-of-sight detection section for detecting the line of the user'"'"'s sight; a display section for indicating image data; and a controller which determines the candidate for selection, toward which the line of sight is directed, to be selected and which performs predetermined processing when the nodding action recognition section detects nodding action and when the line of sight detected by the line-of-sight detection section is oriented toward any one of the candidates for selection during the nodding action, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section.

Claims

1. An instruction recognition system having a gesture recognition function for recognizing a user'"'"'s gesture, the system comprising:

a nodding action recognition section which detects the user'"'"'s nodding action on the basis of a photographed image of a user'"'"'s head;

a line-of-sight detection section for detecting the line of a user'"'"'s sight;

a display section for indicating image data;

a determination section which determines whether or not the nodding action recognition section has detected nodding action defined as one of a down then up and an up then down head movement performed in continuum within a specified period of time and the line of sight detected by the line-of-sight detection section during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section;

an execution section which performs predetermined processing, on the assumption that the candidate toward which the line of the user'"'"'s sight is directed will be selected, when the determination section determines that the nodding action recognition section has detected nodding action and that the line of sight detected by the line-of-sight detection section during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section.

2. The instruction recognition system as defined in claim 1, wherein the detected line of sight is determined to be directed toward the candidate for selection on condition that the line of sight detected by the line-of-sight detection section corresponds to a gazing state where the eyes remain fixed in one orientation for a predetermined period of time.

3. The instruction recognition system as defined in claim 2, wherein the gazing state corresponds to a state in which the line of sight is directed toward a predetermined candidate for selection from the beginning of the nodding action to the end thereof.

4. The instruction recognition system as defined in claim 1, wherein the nodding action recognition section detects nodding action according to the amount of vertical oscillation of the user'"'"'s head.

5. The instruction recognition system as defined in claim 1, wherein the nodding action recognition section recognizes nodding action when a certain portion of the user'"'"'s head moves vertically within a predetermined time period and when the amount of vertical movement exceeds predetermined amounts.

6. The instruction recognition system as defined in claim 1, further comprising a photography section for photographing the user'"'"'s head;

wherein

the nodding action recognition section detects nodding action from the image data photographed by the photography section; and

the line-of-sight detection section detects a line of sight from the image data produced through photography by the photography section.

7. The instruction recognition system as defined in claim 6, wherein the nodding action recognition section detects nodding action through use of at least data pertaining to the positions of the user'"'"'s eyes and/or mouth of the image data produced by the photography section.

8. The instruction recognition system as defined in claim 6, wherein the nodding action recognition section detects nodding action through use of at least data pertaining to the center of gravity of a flesh-colored region on the user'"'"'s head of the image data obtained by the photography section.

9. An instruction recognition system having a gesture recognition function for recognition the user'"'"'s gesture, the system comprising:

a nodding action recognition means for detecting the user'"'"'s nodding action defined as one of a down then up and an up then down head movement performed in continuum within a specified period of time on the basis of a photographed image of the user'"'"'s head;

a line-of-sight detection means for detecting the line of the user'"'"'s sight;

a display means for indicating image data;

a determination means which determines whether or not the nodding action recognition means has detected nodding action and the line of sight detected by the line-of-sight detection means during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display means;

an execution means which performs predetermined processing, on the assumption that the candidate toward which the line of the user'"'"'s sight is directed will be selected, when the determination means determines that the nodding action recognition means has detected nodding action and that the line of sight detected by the line-of-sight detection means during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display means.

10. An instruction recognition system having a gesture recognition function for recognizing a user'"'"'s gesture, the system comprising:

a photographing device for photographing a user'"'"'s head;

a display device for displaying image information; and

a controller performing a nodding action recognition operation for detecting the user'"'"'s nodding action defined as one of a down then up and an up then down head movement performed in continuum within a specified period of time on the basis of the image photographed by the photographing device;

a line-of-sight detection operation for detecting the line of a user'"'"'s sight on the basis of the image photographed by the photographing device;

a determination operation for determining whether or not the nodding action has been detected by the nodding action recognition operation and the line of sight detected by the line-of-sight detection operation during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section;

an execution operation for performing predetermined processing, on the assumption that the candidate toward which the line of the user'"'"'s sight is directed will be selected, when it is determined that nodding action has been detected by the nodding action recognition operation and that the line of sight detected by the line-of-sight detection operation during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section.

11. The instruction recognition system according to claim 10, wherein the controller determines that the detected line of sight is directed toward the candidate for selection on condition that the line of sight detected by the line-of-sight detection operation corresponds to a gazing state where the eyes remain fixed in one orientation for a predetermined period of time.

12. The instruction recognition system according to claim 11,

wherein the gazing state corresponds to a state in which the line of sight is directed toward a predetermined candidate for selection from the beginning of the nodding action to the end thereof.

13. The instruction recognition system according to claim 10, wherein during the nodding action detection operation, the controller detects nodding action according to the amount of vertical oscillation of the user'"'"'s head.

14. The instruction recognition system according to claim 10, wherein during the nodding action detection operation, the controller recognizes nodding action when a certain portion of the user'"'"'s head moves vertically within a predetermined time period and when the amount of vertical movement exceeds predetermined amounts.

15. The instruction recognition system according to claim 10, wherein the controller detects nodding action through use of at least data pertaining to the positions of the user'"'"'s eyes and/or mouth of the image data produced by the photographing device.

16. The instruction recognition system according to claim 10, wherein the controller detects nodding action through use of at least data pertaining to the center of gravity of a flesh-colored region on the user'"'"'s head of the image data obtained by the photographing device.

17. An instruction recognition system having a gesture recognition function for recognizing a user'"'"'s gesture, the system comprising:

a photographing device for photographing a user'"'"'s head;

a display device for displaying image information; and

a controller performing a nodding action recognition operation for detecting the user'"'"'s nodding action defined as one of a down then up and an up then down head movement performed in continuum within a specified period of time on the basis of the image photographed by the photographing device;

a line-of-sight detection operation for detecting the line of a user'"'"'s sight on the basis of the image photographed by the photographing device;

a determination operation for determining whether or not the nodding action has been detected by the nodding action recognition operation and the line of sight detected by the line-of-sight detection operation during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section;

an execution operation for performing predetermined processing, on the assumption that the candidate toward which the line of the user'"'"'s sight is directed will be selected, when it is determined that nodding action has been detected by the nodding action recognition operation and that the line of sight detected by the line-of-sight detection operation during the nodding action is directed toward any of the candidates for selection, while the system awaits the user'"'"'s instruction and one or a plurality of candidates for selection are displayed on the display section;

wherein the controller detects nodding action through use of at least data pertaining to the positions of the user'"'"'s eyes and/or mouth of the image data produced by the photographing device.