Depth-based user interface gesture control

• US 9,389,779 B2
• Filed: 03/14/2013
• Issued: 07/12/2016
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Anderson, Glen J., Reif, Dror, Hurwitz, Barak, Kamhi, Gila
• Primary Examiner(s): Pan, Yongjia
• Application Number:
  US13/976,036
• Publication Number:
  US 20140282278A1
• Time in Patent Office: 1,216 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Technologies for depth-based gesture control include a computing device having a display and a depth sensor. The computing device is configured to recognize an input gesture performed by a user, determine a depth relative to the display of the input gesture based on data from the depth sensor, assign a depth plane to the input gesture as a function of the depth, and execute a user interface command based on the input gesture and the assigned depth plane. The user interface command may control a virtual object selected by depth plane, including a player character in a game. The computing device may recognize primary and secondary virtual touch planes and execute a secondary user interface command for input gestures on the secondary virtual touch plane, such as magnifying or selecting user interface elements or enabling additional functionality based on the input gesture. Other embodiments are described and claimed.

Claims

1. A computing device for depth-based gesture control, the computing device comprising:

a display to define a surface normal;

a depth sensor to;

generate depth sensor data indicative of a depth relative to the display of an input gesture performed by a user of the computing device in front of the display;

generate second depth sensor data indicative of a second depth relative to the display of a second input gesture performed by a second user of the computing device in front of the display; and

generate third depth sensor data indicative of a third depth relative to the display of a third input gesture performed by the second user of the computing device in front of the display;

a gesture recognition module to recognize the input gesture, the second input gesture, and the third input gesture;

a depth recognition module to;

receive the depth sensor data, the second depth sensor data, and the third depth sensor data from the depth sensor;

determine the depth of the input gesture as a function of the depth sensor data;

determine the second depth of the second input gesture as a function of the second depth sensor data;

determine the third depth of the third input gesture as a function of the third depth sensor data;

assign a depth plane to the input gesture as a function of the depth of the input gesture;

assign a second depth plane different from the depth plane to the second input gesture as a function of the second depth of the second input gesture; and

assign a third depth plane different from the second depth plane to the third input gesture as a function of the third depth of the third input gesture;

wherein each depth plane is positioned parallel to the display and intersects the surface normal; and

a user command module to;

designate the second depth plane as an accessible depth plane for the second user;

execute a user interface command based on the input gesture and the assigned depth plane;

execute a second user interface command based on the second input gesture and the assigned second depth plane;

determine whether the third depth is associated with the accessible depth plane for the second user; and

reject the third input gesture in response to a determination that the third depth is not associated with the accessible depth plane for the second user,

wherein to execute the second user interface command comprises to;

determine whether the second assigned depth plane comprises a secondary virtual touch plane of the computing device; and

execute a secondary user interface command in response to a determination that the assigned second depth plane comprises the secondary virtual touch plane, wherein to execute the secondary user interface command comprises to display a contextual command menu on the display.

2. The computing device of claim 1, wherein the depth sensor comprises one of:

a stereo depth camera;

a structured light camera;

a plurality of cameras, wherein each camera of the plurality of cameras is aimed perpendicular to the surface normal of the display and perpendicular to another camera of the plurality of cameras;

or

a proximity sensor.

3. The computing device of claim 1, wherein:

the depth sensor comprises a camera;

to receive the depth sensor data comprises to receive image data from the camera; and

to determine the depth of the input gesture comprises to determine the depth of the input gesture as a function of a position of a shadow cast by the user captured in the received image data.

4. The computing device of claim 1, wherein:

the depth sensor comprises a radio receiver;

the depth recognition module is further to receive, using the radio receiver, a signal transmitted by a transmitter manipulated by the user; and

to receive the depth sensor data comprises to receive signal strength data associated with the received signal from the radio receiver.

5. The computing device of claim 1, wherein to execute the user interface command based on the assigned depth plane comprises:

to select a virtual object as a function of the assigned depth plane; and

to control the virtual object based on the input gesture.

6. The computing device of claim 1, wherein to execute the user interface command comprises one of:

to magnify a user interface element displayed on the display;

to activate a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display;

or

to assign a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

7. The computing device of claim 1, further comprising a user recognition module to:

identify the user of the computing device; and

assign the input gesture to the user;

wherein to execute the user interface command further comprises to execute the user interface command as a function of the user assigned to the input gesture.

8. The computing device of claim 7, wherein the user recognition module further comprises a skeletal tracking module to:

track a skeletal feature of the user;

track a skeletal feature of the second user;

wherein to identify the user of the computing device comprises to distinguish between the skeletal feature of the user and the skeletal feature of the second user.

9. The computing device of claim 1, wherein to execute the secondary user interface command comprises to magnify a user interface element displayed on the display.

10. The computing device of claim 1, wherein to execute the secondary user interface command comprises to activate a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display.

11. The computing device of claim 1, wherein to execute the secondary user interface command comprises to assign a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

12. A method for depth-based gesture control, the method comprising:

recognizing, on a computing device, an input gesture performed by a user of the computing device in front of a display of the computing device;

recognizing, on a computing device, a second input gesture performed by a second user of the computing device in front of the display of the computing device;

recognizing, on a computing device, a third input gesture performed by the second user of the computing device in front of the display of the computing device;

receiving, on the computing device, depth sensor data indicative of a depth relative to the display of the input gesture from a depth sensor of the computing device;

receiving, on the computing device, second depth sensor data indicative of a second depth relative to the display of the second input gesture from the depth sensor;

receiving, on the computing device, third depth sensor data indicative of a third depth relative to the display of the third input gesture from the depth sensor;

determining, on the computing device, the depth of the input gesture as a function of the depth sensor data;

determining, on the computing device, the second depth of the second input gesture as a function of the second depth sensor data;

determining, on the computing device, the third depth of the third input gesture as a function of the third depth sensor data;

assigning, on the computing device, a depth plane to the input gesture as a function of the depth of the input gesture;

assigning, on the computing device, a second depth plane different from the depth plane to the second input gesture as a function of the second depth of the second input gesture;

assigning, on the computing device, a third depth plane different from the second depth plane to the third input gesture as a function of the third depth of the third input gesture;

wherein each depth plane is parallel to the display and intersects a surface normal of the display;

designating, on the computing device, the second depth plane as an accessible depth plane for the second user;

executing, on the computing device, a user interface command based on the input gesture and the assigned depth plane;

executing, on the computing device, a second user interface command based on the input gesture and the assigned second depth plane;

determining, on the computing device, whether the third depth is associated with the accessible depth plane for the second user; and

rejecting, on the computing device, the third input gesture in response to a determination that the third depth is not associated with the accessible depth plane for the second user,

wherein executing the second user interface command comprises;

determining whether the assigned second depth plane comprises a secondary virtual touch plane of the computing device; and

executing a secondary user interface command in response to determining the assigned second depth plane comprises the secondary virtual touch plane, wherein executing the secondary user interface command comprises displaying a contextual command menu.

13. The method of claim 12, wherein executing the user interface command comprises one of:

magnifying a user interface element displayed on the display of the computing device;

activating a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display;

or

assigning a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

14. The method of claim 12, further comprising:

identifying, on the computing device, the user of the computing device; and

assigning, on the computing device, the input gesture to the identified user;

wherein

executing the user interface command further comprises executing the user interface command as a function of the user assigned to the input gesture.

15. One or more non-transitory, machine readable storage media comprising a plurality of instructions that in response to being executed cause a computing device to:

recognize an input gesture performed by a user of the computing device in front of a display of the computing device;

recognize a second input gesture performed by a second user of the computing device in front of a display of the computing device;

recognize a third input gesture performed by the second user of the computing device in front of the display of the computing device;

receive depth sensor data indicative of a depth relative to the display of the input gesture from a depth sensor of the computing device;

receive second depth sensor data indicative of a second depth relative to the display of the second input gesture from the depth sensor;

receive third depth sensor data indicative of a third depth relative to the display of the third input gesture from the depth sensor;

determine the depth of the input gesture as a function of the depth sensor data;

determine the second depth of the second input gesture as a function of the second depth sensor data;

determine the third depth of the third input gesture as a function of the third depth sensor data;

assign a depth plane to the input gesture as a function of the depth of the input gesture,

assign a second depth plane different from the depth plane to the second input gesture as a function of the second depth of the second input gesture;

assign a third depth plane different from the second depth plane to the third input gesture as a function of the third depth of the third input gesture;

wherein each depth plane is parallel to the display and intersects a surface normal of the display;

designate the second depth plane as an accessible depth plane for the second user;

execute a user interface command based on the input gesture and the assigned depth plane;

execute, on the computing device, a second user interface command based on the input gesture and the assigned second depth plane;

determine whether the third depth is associated with the accessible depth plane for the second user; and

reject the third input gesture in response to a determination that the third depth is not associated with the accessible depth plane for the second user,

wherein to execute the second user interface command comprises to;

determine whether the assigned second depth plane comprises a secondary virtual touch plane of the computing device; and

execute a secondary user interface command in response to a determination that the assigned depth plane comprises the secondary virtual touch plane, wherein to execute the secondary user interface command comprises to display a contextual command menu on the display.

16. The non-transitory, machine readable media of claim 15, further comprising a plurality of instructions that in response to being executed cause the computing device to allow access to the secondary user interface command only through the secondary virtual touch plane.

17. The non-transitory, machine readable media of claim 15, wherein to execute the secondary user interface command comprises one of to:

magnify a user interface element displayed on the display of the computing device;

activate a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display;

or

assign a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

18. The non-transitory, machine readable media of claim 15, further comprising a plurality of instructions that in response to being executed cause the computing device to:

identify the user of the computing device; and

assign the input gesture to the identified user;

wherein

to execute the user interface command further comprises to execute the user interface command as a function of the user assigned to the input gesture.