Gesture based human computer interaction

• US 9,304,597 B2
• Filed: 10/29/2013
• Issued: 04/05/2016
• Est. Priority Date: 10/29/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Grunnet-Jepsen, Anders, Sweetser, John N., Winer, Paul, Bateman, Steven S., Takagi, Akihiro, Keselman, Leonid M., Jayant, Chandrika, Thompson, Jonathan C.
• Primary Examiner(s): Siddiqui, Md Saiful A
• Application Number:
  US14/127,106
• Publication Number:
  US 20150116214A1
• Time in Patent Office: 889 Days
• Field of Search: 345156-158, 345173-179, 345/419, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

Apparatus, computer-readable storage medium, and method associated with human computer interaction. In embodiments, a computing device may include a plurality of sensors, including a plurality of light sources and a camera, to create a three dimensional (3-D) interaction region within which to track individual finger positions of a user of the computing device. The light sources and the camera may be complementarily disposed for the camera to capture the finger or hand positions. The computing device may further include a 3-D interaction module configured to analyze the individual finger positions within the 3-D interaction region, the individual finger movements captured by the camera, to detect a gesture based on a result of the analysis, and to execute a user control action corresponding to the gesture detected. Other embodiments may be described and/or claimed.

Claims

1. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by a computing device, provide the computing device with a three-dimensional (3-D) interaction module configured to:

analyze sensor data associated with individual finger positions of a user of the computing device, only within a 3-D interaction region, sensed by a plurality of sensors of the computing device that include a plurality of light sources and a camera, wherein the plurality of light sources and the camera are complementarily disposed for the camera to capture the individual finger positions in three dimensions, and wherein the 3-D interaction region is a region in which light emitted from the plurality of light sources intersects a viewing angle of the camera and within a viewing area of a display such that the individual finger positions within the viewing angle of the camera and outside of the 3-D interaction region are not to be included as a gesture of the user;

detect a gesture of the user, based at least in part on a result of the analysis,

wherein the gesture comprises at least a first individual finger of a hand hovering in a same position for a predetermined period of time and at least one movement using the first individual finger, using a second individual finger of the hand, or using the first individual finger and the second individual finger;

determine a location of the first individual finger or the second individual finger within the 3-D interaction region;

correlate the location of the first individual finger or the second individual finger, within the 3-D interaction region, with a location of a cursor rendered on a display of the computing device,

wherein the at least one movement comprises holding the first individual finger of the hand, extending beyond remaining fingers of the hand, in a fixed position within the 3-D interaction region for a first predetermined period of time to cause the correlated location of the cursor to freeze for a second predetermined period of time, and the at least one movement further comprises a forward or downward movement of a tip of the first individual finger within the second predetermined period of time; and

execute a user control action corresponding to the detected gesture, to control the computing device, and wherein to execute further comprises execution of a user control action proximate to the correlated location of the cursor rendered on the display.

2. The one or more non-transitory computer-readable media of claim 1, wherein the 3-D interaction module is further configured to analyze the at least one movements to detect a mouse click gesture based on the at least one movements and execute a mouse click in response to detection of the mouse-click gesture.

3. The one or more non-transitory computer-readable media of claim 1, wherein the at least one movement comprises holding the first individual finger of the hand, extending beyond the remaining fingers of the hand, in a fixed location for the predetermined period of time within the 3-D interaction region.

4. The one or more non-transitory computer-readable media of claim 1, wherein the at least one movement comprises holding the first individual finger of the individual fingers in a fixed location within the 3-D interaction region and bringing the second finger of the individual fingers into contact with the first individual finger, and wherein to execute further comprises execution of a user control action proximate to the correlated location of the cursor rendered on the display.

5. The one or more non-transitory computer-readable media of claim 1, wherein the at least one movement comprises holding the first individual finger of the individual fingers in a fixed location, and wherein the at least one movement further comprises bringing the second individual finger, to a right of the first individual finger, into contact with the first individual finger, and wherein to execute further comprises execution of a user control action proximate to the correlated location of the cursor rendered on the display.

6. The one or more non-transitory computer-readable media of claim 1, wherein the at least one movement comprises holding the first individual finger of the individual fingers in a fixed location, and wherein the at least one movement further comprises bringing the second finger to a left of the first individual finger into contact with the first individual finger, and wherein to execute further comprises execution of a user control action proximate to the correlated location of the cursor rendered on the display.

7. The one or more non-transitory computer-readable media of claim 1, wherein the at least one movement comprises swirling the first individual finger of the hand, extending beyond the remaining fingers of the hand, in a clockwise or counter-clockwise direction, and wherein to execute further comprises execution of a user control action proximate to the correlated location of the cursor rendered on the display, the user control action based on the direction of the swirling of the individual finger.

8. The one or more non-transitory computer-readable media of claim 1, wherein the user control action is a mouse click of an item proximate to the correlated location of the cursor rendered on the display.

9. The one or more non-transitory computer-readable media of claim 1, wherein the 3-D interaction region coincides or intersects with a floating display.

10. A computing device for human computer interaction comprising:

a plurality of sensors, including a plurality of light sources and a camera, to create a three-dimensional (3-D) interaction region in which light emitted from the plurality of light sources intersects a viewing angle of the camera and is within a viewing area of a display, within which to track individual finger positions of a user of the computing device, wherein the plurality of light sources and the camera are complementarily disposed to enable the camera to capture the individual finger positions in three dimensions;

a 3-D interaction module coupled with the plurality of sensors, the 3-D interaction module to analyze the individual finger positions, in relation to one another, only within the 3-D interaction region such that the individual finger positions within the viewing angle of the camera and outside of the 3-D interaction region are not to be included as a gesture of the user, captured by the camera, to detect a gesture based on a result of the analysis, and to execute a user control action corresponding to the gesture detected; and

a display coupled with the 3-D interaction module, wherein the 3-D interaction module is to further determine a location of the individual fingers within the 3-D interaction region, and to correlate the location of the individual fingers, within the 3-D interaction region, with a location of a cursor rendered on the display,

wherein the detected gesture comprises an individual finger of the individual fingers to be extended beyond remaining fingers of the individual fingers in a fixed position within the 3-D interaction region for a first predetermined period of time to cause the correlated location of the cursor to freeze for a second predetermined period of time, and the gesture detected further comprises a forward or downward movement of a tip of the individual finger within the second predetermined period of time, and wherein the 3-D interaction module is to execute a user control action at the correlated location of the cursor rendered on the display.

11. The computing device of claim 10, wherein the gesture detected comprises holding the individual finger, extending beyond the remaining fingers of the individual fingers, in a fixed location for the first predetermined period of time within the 3-D interaction region, and wherein the 3-D interaction module is to execute a user control action at the correlated location of the cursor rendered on the display.

12. The computing device of claim 10, wherein the individual finger is a first individual finger of the individual fingers and the gesture detected comprises holding the first individual finger in a fixed location within the 3-D interaction region and bringing a second finger of the individual fingers into contact with the first individual finger, and wherein the 3-D interaction module is to execute a user control action at the correlated location of the cursor rendered on the display.

13. The computing device of claim 10, wherein the individual finger is a first individual finger of the individual fingers and the gesture detected comprises holding the first individual finger of the individual fingers in a fixed location, and wherein the gesture detected further comprises bringing a second individual finger of the individual fingers, to the right of the first individual finger, into contact with the first individual finger, and wherein the 3-D interaction module is to execute a user control action at the correlated location of the cursor rendered on the display.

14. The computing device of claim 10, wherein the individual finger is a first individual finger of the individual fingers and the gesture detected comprises holding the first individual finger of the individual fingers in a fixed location, and wherein the gesture detected further comprises bringing a second individual finger to a left of the first individual finger into contact with the first individual finger, and wherein the 3-D interaction module is to execute a user control action, at the correlated location of the cursor rendered on the display.

15. The computing device of claim 10, wherein the gesture detected comprises swirling the individual finger, extending beyond the remaining fingers of a hand, in a clockwise or counter-clockwise direction, and wherein the 3-D interaction module is to execute a predetermined user control action, proximate to the correlated location of the cursor rendered on the display, based on the direction of the swirling of the individual finger.

16. The computing device of claim 10, wherein the display is a floating display projected near or intersecting with the 3-D interaction region and the plurality of light sources are light emitting diodes (LEDs) and wherein to be complementarily disposed further comprises disposition of the LEDs such that light from the LEDs intersects the viewing angle of the camera where the intersection between the light and the viewing angle defines the 3-D interaction region.

17. The computing device of claim 10, wherein the user control action is one of a left or right mouse click.

18. The computing device of claim 10, wherein at least the camera of the plurality of sensors and the 3-D interaction module are to further cooperate to detect one of heart rate, blood pressure, or blood oxygenation of the user within the 3-D interaction region.

19. The computing device of claim 10, wherein the plurality of sensors and the 3-D interaction module are to cooperate to detect a heart rate of the user, and the 3-D interaction module is to further perform the analysis, detection of finger movements, and execution only when presence of the heart rate of the user is detected within the 3-D interaction region.

20. A computer-implemented method for human computer interaction comprising:

analyzing, by a three-dimensional (3-D) interaction module of a computing device, sensor data associated with individual finger positions of a user of the computing device, only within a 3-D interaction region, sensed by a plurality of sensors of the computing device that include a plurality of light sources and a camera, wherein the plurality of light sources and the camera are complementarily disposed for the camera to capture the individual finger positions in three dimensions, and wherein the 3-D interaction region is a region in which light emitted from the plurality of light sources intersects a viewing angle of the camera and is within a viewing area of a display such that the individual finger positions within the viewing angle of the camera and outside of the 3-D interaction region are not to be included as a gesture of the user;

detecting, by the 3-D interaction module, a gesture of the user, based at least in part on a result of the analysis, wherein the gesture comprises at least an individual finger of a hand hovering in a same position for a predetermined period of time and at least one of a pecking motion in a z-direction of the 3-D interaction region or a tapping motion in a y-direction of the 3-D interaction region; and

executing, by the 3-D interaction module, a user control action corresponding to the gesture detected, to control the computing device.

21. The computer-implemented method of claim 20, further comprising analyzing the individual finger movements to detect a mouse click gesture based on the individual finger movements and executing a mouse click in response to detection of the mouse-click gesture.