SYSTEM AND METHOD FOR HUMAN COMPUTER INTERACTION

• US 20140125584A1
• Filed: 11/04/2013
• Published: 05/08/2014
• Est. Priority Date: 11/07/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Maolin, Chen, XUN, Sun
• Granted Patent: US 9,684,372 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A system and method for human computer interaction (HCI). may acquire image data, determine an interaction intended to be conducted by a user based on various gestures and poses of the user detected from the image data, and perform an operation and/or function which is displayed on a display of a display unit, in response to a result of the interaction.

Claims

1. A system for human computer interaction (HCI), the system comprising:

an image acquisition unit to acquire image data;

a line of sight (LOS) catch unit to detect a LOS of a user by detecting a pitch direction and a deflection direction of a head of a user based on the image data; and

an interaction determining unit to determine an interaction based on the LOS of the user detected by the LOS catch unit.

2. The system of claim 1, further comprising:

a pose tracking unit to track and recognize a pose and a gesture of a body of the user based on the image data.

3. The system of claim 2, wherein the pose tracking unit determines a gesture of a hand of the user by tracking and detecting a hand node of the user in the image data, and determines the pose and the gesture of the body of the user by detecting a skeleton node of the body of the user.

4. The system of claim 2, wherein the pose tracking unit recognizes the gesture of the user by tracking and recognizing a gesture of a finger of the user when the user is within a predetermined distance from the image acquisition unit, and by tracking and recognizing a gesture of an arm of the user when the user is located beyond the predetermined distance or farther from the image acquisition unit.

5. The system of claim 3, wherein the interaction determining unit further determines an interaction intended to be conducted by the user based on the pose and the gesture of the body of the user.

6. The system of claim 5, wherein the interaction determining unit determines whether an interaction is initiated based on the LOS of the user detected by the LOS catch unit and the gesture of the hand of the user detected by the pose tracking unit.

7. The system of claim 3, further comprising:

a display unit to display on a screen a result of the interaction,

wherein the interaction determining unit initiates a predetermined interaction with respect to a display item displayed on the screen when both the LOS of the user and a direction of the hand of the user faces a direction of the display item for a period of time longer than a preset period of time.

8. The system of claim 7, wherein the interaction determining unit stops the interaction with respect to the display item when neither the LOS of the user nor the direction of the hand of the user faces the direction of the display item.

9. The system of claim 1, further comprising:

a user-defined pose register unit to register an interaction instruction corresponding to a user-defined pose.

10. The system of claim 1, wherein the interaction determining unit initiates an interaction and terminates an interaction, based on the LOS of the user detected by the LOS catch unit.

11. A method for human computer interaction (HCI), the method comprising:

acquiring image data;

detecting a gesture of a user by tracking and recognizing a gesture of a first body part of the user when the user is within a predetermined distance from an image acquisition unit to acquire the image data; and

detecting the gesture of the user by tracking and recognizing a gesture of a second body part of the user when the user is located at or beyond the predetermined distance from the image acquisition unit.

12. The method of claim 11, further comprising:

determining at least one of a start and an end of an interaction based on the detected gesture of the user.

13. The method of claim 11, further comprising:

initiating a predetermined interaction with respect to a display item displayed on a screen when both a line of sight (LOS) of the user and a direction of a hand of the user face a direction of the display item for a period of time longer than a preset period of time.

14. The method of claim 13, further comprising:

stopping the interaction with respect to the display item when neither the LOS of the user nor the direction of the hand of the user faces the direction of the display item.

15. The method of claim 11, further comprising:

detecting a LOS of a user from the image data by detecting a pitch direction and a deflection direction of a head of the user based on the image data; and

determining at least one of a start and an end of an interaction based on the detected LOS of the user and the detected gesture of the user.

16. The method of claim 11, further comprising:

registering an interaction instruction corresponding to a user-defined pose; and

providing information associated with reproducibility and ambiguity of the user-defined pose to the user.

17. The method of claim 11, further comprising:

displaying a plurality of display items;

initiating an interaction with respect to the plurality of display items when both a direction indicated by a hand and a line of sight (LOS) of the user are determined to face a direction of the plurality of display items;

selecting a first item among the plurality of display items when a gesture of a first finger is detected as indicating the first item; and

displaying a sub-menu corresponding to the item when a gesture of a second finger is detected as indicating the first item.

18. The method of claim 11, further comprising:

displaying a three-dimensional (3D) item;

initiating an interaction with respect to the 3D item when both a direction indicated by one of the first or second body part and a line of sight (LOS) of the user are determined to face a direction of the 3D item; and

performing at least one of moving the 3D item, expanding a size of the 3D item, or reducing a size of the 3D item, based on the detected gesture of one of the first or second body part.

19. The method of claim 11, further comprising:

designating a preset area of a screen of a display as a hand writing area;

initiating an interaction with respect to the preset area when both a direction indicated by one of the first or second body part and a line of sight (LOS) of the user are determined to face a direction of the preset area;

determining a text input by the user, based on the detected gesture of one of the first or second body part; and

conducting an interaction corresponding to the determined text.

20. A method for human computer interaction, the method comprising:

acquiring, using a camera, image data of a user positioned away from a display;

determining a line of sight and a pose of a user based on the image data;

calculating an intersection point at which a direction indicated by at least one body part of the user intersects with a screen of the display;

detecting a gesture of the user by tracking and recognizing a gesture of the at least one body part of the user; and

determining an interaction intended to be performed by the user based on the calculated intersection point, detected gesture, and the direction indicated by the at least one body part.