Apparatus and method for controlling an electronic device with user action

• US 6,111,580 A
• Filed: 09/06/1996
• Issued: 08/29/2000
• Est. Priority Date: 09/13/1995
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Kazama, Hisashi, Onoguchi, Kazunori, Fukui, Kazuhiro, Yuasa, Mayumi
• Primary Examiner(s): Kim, Matthew M.
• Assistant Examiner(s): dela Torre, Crescelle N.
• Application Number:
  US08/709,493
• Time in Patent Office: 1,453 Days
• Field of Search: 345/326, 345/358, 345/327, 345/978, 345/156, 345/157, 345/158, 704/246, 704/251, 704/260, 704/275, 364/400.01, 364/709.01, 364/709.02, 364/188, 364/146, 351/209, 351/210, 382/117, 382/118
• US Class Current: 715/863
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

An input unit for controlling a device. The input unit first determines whether a user is actively operating the device. If such a detection is made, the input unit then determines whether the user performs an act that corresponds to a command. User attention may be indicated by a user observing the device, approaching the device, facing the device, or speaking. An input command may be indicated by touch, gesture, speech, or line of sight.

Claims

1. An input apparatus for detecting a user'"'"'s action that indicates an input command and for outputting an operation signal corresponding to the input command to an electronic device, comprising:

attention detection means for continuously detecting a first action of the user that indicates an attention status of the user, and for deciding that the electronic device is under the attention status of the user while the first action of the user is continuously detected;

action detection means for detecting a second action of the user that indicates the input command in accordance with the second action of the user, wherein the second action is different from the first action; and

operation signal output means for outputting the operation signal corresponding to the input command to the electronic device, if the second action is detected while the first action is continuously detected.

2. The input apparatus according to claim 1,

wherein the first action comprises continuously gazing at a predetermined part of a device receiving the operation signal, and the attention detection means comprises means for extracting feature points from a face image of the user and means for calculating a watching point of the user based on said feature points,

wherein the second action comprises a gesture corresponding to the input command, and the action detection means comprises means for recognizing the gesture from a predetermined movement in a partial image of the user.

3. The input apparatus according to claim 1,

wherein the first action comprises producing voice as a predetermined word, and the attention detection means comprises means for recognizing the predetermined word,

wherein the second action comprises a gesture corresponding to the input command, and the action detection means comprises means for recognizing the gesture from a predetermined movement in a partial image of the user.

4. The input apparatus according to claim 1,

wherein the first action comprises continuously touching by the user a part of a device receiving the operation signal, and the attention detection means comprises means for detecting said touching with a sensor attached to the part of the device, wherein the second action comprises a gesture corresponding to the input command, and the action detection means comprises means for recognizing the gesture from a predetermined movement in a partial image of the user.

5. The input apparatus according to claim 1,

wherein the first action comprises touching by the user a part of the device receiving the operation signal and the attention detection means comprises means for detecting said touching when a switch attached to the part is turned on, and

wherein the second action comprises gazing at a predetermined part of the device, and the action detection means comprises means for extracting feature points from a face image of the user and means for calculating a watching point of the user based on said feature points.

6. The input apparatus according to claim 1,

wherein the first action comprises continuously facing by the user the front of a device receiving the operation signal, and the attention detection means comprises means for determining whether the user'"'"'s face direction is frontal of the device by feature points of the user'"'"'s face image, and

wherein the second action comprises gazing at a predetermined part of the device, and the action detection means comprises means for extracting feature points from the user'"'"'s face image and means for calculating a watching point of the user based on said feature points.

7. The input apparatus according to claim 1,

wherein the first action comprises producing voice as a predetermined word, and the attention detection means comprises means for recognizing the predetermined word, and

wherein the second action comprises gazing at a predetermined part of a device receiving the operation signal, and the action detection means comprises means for extracting feature points from the user'"'"'s face image and means for calculating a watching point of the user based on said feature points.

8. The input apparatus according to claim 1,

wherein the first action comprises continuously facing by the user the front of a device receiving the operation signal and the attention detection means comprises means for determining whether the user'"'"'s face direction is frontal of the device based on feature points of the users face image, and

wherein the second action comprises a gesture corresponding to the input command, and the action detection means comprises means for recognizing the gesture from a predetermined movement in a partial image of the user.

9. The input apparatus according to claim 1,

wherein the first action comprises gazing at a predetermined part of a device receiving the operation signal, and the attention detection means comprises means for extracting feature points from the user'"'"'s face image and means for calculating a watching point of the user based on said feature points, and

wherein the second action comprises touching the device by the user, and the action detection means comprises means for detecting said touching by the user with a sensor attached to the device.

10. The input apparatus according to claim 1,

wherein the first action comprises continuously facing by the user the front of a device receiving the operation signal, and the attention detection means comprises means for determining whether the user'"'"'s face direction is frontal of the device by feature points of a facial image of the user, and

wherein the second action comprises approaching the device by the user, and the action detection means comprises means for detecting said approaching with a sensor attached to the device for detecting an approach by the user.

11. The input apparatus according to claim 1,

wherein the first action comprises a gesture corresponding to a predetermined movement, and the attention detection means comprises means for recognizing the gesture from the predetermined movement in a partial image of the user, and

wherein the second action comprises gazing at a predetermined part of a device receiving the operation signal, and the action detection means comprises means for extracting feature points from a facial image of the user and means for calculating a watching point of the user based on said feature points.

12. The input apparatus according to claim 1, further comprising means for outputting a report that the attention status is detected by the attention detection means.

13. The input apparatus according to claim 1, wherein said signal output means includes means for outputting the operation signal to a watch.

14. The input apparatus according to claim 1, wherein said signal output means includes means for outputting the operation signal to a computer having a keyboard, said operation signal corresponding to a key pushed by touching of the keyboard by the user.

15. The input apparatus according to claim 1, wherein said signal output means includes means for outputting the operation signal to an automatic door.

16. The input apparatus according to claim 1, wherein said signal output means includes means for outputting the operation signal to a home electronic device.

17. A method for detecting a user'"'"'s action that indicates an input command and for outputting an operation signal corresponding to the input command to an electronic device, comprising the steps of:

continuously detecting a first action of the user that indicates an attention status of the user;

deciding that the electronic device is under the attention status of the user while the first action is continuously detected;

detecting a second action of the user that indicates an input command in accordance with the second action of the user, wherein the second action is different from the first action; and

outputting the operation signal corresponding to the input command, if the second action is detected while the first action is continuously detected.

18. The method according to claim 17, wherein the first action comprises continuously gazing at a predetermined part of a device receiving the operation signal, and wherein a watching point of the user is calculated from a face image of the gazing user.

19. The method according to claim 17,

wherein the first action comprises continuously touching a part of a device receiving the operation signal, the user'"'"'s touching being detected by a sensor attached to the part of the device.

20. The method according to claim 17,

wherein the first action comprises continuously facing by the user the front of a device receiving the operation signal, and wherein the user'"'"'s face direction is determined to be frontal of the device by feature points of the user'"'"'s face image.

21. A computer readable memory containing computer-readable instructions, comprising:

instruction means for causing a computer to continuously detect a first action of the user that indicates the attention status of the user;

instruction means for causing the computer to decide that an electronic device is under the attention status of the user while the first action of the user is continuously detected;

instruction means for causing the computer to detect a second action of the user that indicates an input command in accordance with the second action of the user, wherein the second action is different from the first action; and

instruction means for causing the computer to generate an operation signal corresponding to the input command to the electronic device, if the second action is detected while the first action of the user is continuously detected.