Natural user interface scrolling and targeting

• US 9,342,230 B2
• Filed: 03/13/2013
• Issued: 05/17/2016
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bastien, David, Murillo, Oscar, Schwesinger, Mark
• Primary Examiner(s): Nabi, Reza
• Assistant Examiner(s): Rodriguez, Daniel
• Application Number:
  US13/802,234
• Publication Number:
  US 20140282223A1
• Time in Patent Office: 1,161 Days
• Field of Search: 715/784
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A user interface is output to a display device. If an element of a human subject is in a first conformation, the user interface scrolls responsive to movement of the element. If the element is in a second conformation, different than the first conformation, objects of the user interface are targeted responsive to movement of the element without scrolling the user interface.

Claims

1. A method for presenting a user interface on a display device pursuant to movement of an element of a human subiect, the method comprising:

outputting the user interface to the display device, the user interface including a cursor that moves as a targeting function of a position of the element, where movement of the cursor to a boundary of the user interface is recognized as a transition of the element between different first and second conformations of the element;

when the element is in the first conformation, scrolling the user interface responsive to movement of the element;

when the element is in the second conformation, targeting one or more objects of the user interface responsive to movement of the element, without scrolling the user interface; and

inertially scrolling the user interface as a scrolling function of a velocity of the element during a time period that includes a transition of the element from the first conformation, the user interface inertially scrolling with an amplified speed in a current scrod direction responsive to subsequent movement of the element in a direction similar to the current scroll direction, wherein user interface inertial scrolling speeds are not limited to the velocity of the element.

2. The method of claim 1, where scrolling the user interface responsive to movement of the element includes scrolling the user interface in three dimensions responsive to movement of the element in three dimensions.

3. The method of claim 1 further comprising selecting an object of the user interface responsive to movement of the element over a threshold distance in the direction of the display device, when the cursor engages the object.

4. The method of claim 1, further comprising:

generating an inertial vector as an inertial function of movement of the element during a time period that includes a transition of the element from the first conformation; and

inertially scrolling the user interface in the direction of the inertial vector.

5. The method of claim 1, further including reducing the speed of inertial scrolling proportionately to a distance the element moves along an axis perpendicular to the display device.

6. A storage machine holding instructions executable by a logic machine to:

output a user interface to a display device;

receive a virtual skeleton of a human subject modeled from a depth image from a depth camera, the virtual skeleton including a virtual hand joint modeling a three-dimensional position of a hand of the human subject;

use the virtual hand joint to determine whether the hand of the human subject is in a closed-grip conformation or in an open-grip conformation;

when the hand is in the closed-grip conformation, scrolling the user interface in each of three dimensions responsive to movement of the virtual hand joint in each of three dimensions;

when the hand is in the open-grip conformation, target objects of the user interface responsive to movement of the virtual hand joint without scrolling the user interface;

inertially scroll the user interface in three dimensions as a scrolling function of a velocity of the hand of the human subject and as an inertial function of a direction of movement of the hand of the human subject in three dimensions during a time period that includes a transition of the hand from the the closed grip conformation; and

decrease the speed of inertial scrolling as a continuously variable function of a distance the hand of the human subject moves along an axis perpendicular to the display device.

7. The storage machine of claim 6, where using the virtual hand joint to determine if the hand of the human subject is in a closed grip conformation or open grip conformation includes analyzing a portion of the depth image corresponding to a position of the virtual hand joint.

8. The storage machine of claim 7, where analyzing a portion of the depth image corresponding to the position of the virtual hand joint includes comparing the portion of the depth image to machine-learned depth images corresponding to known hand joint conformations.

9. The storage machine of claim 7, where determining whether the hand of the human subject is in a closed grip conformation or in an open grip conformation further includes determining the conformation of the hand by recognizing a transition of the hand from one conformation to another conformation.

10. The storage machine of claim 9, where recognizing the transition of the hand from one conformation to another conformation includes determining that the hand has reached a virtual boundary.

11. The storage machine of claim 9, where the user interface includes a cursor that moves as a targeting function of a position of the hand of the human subject, and where movement to a boundary of the user interface is recognized as a transition of the hand from one conformation to another conformation.

12. The storage machine of claim 7, further holding instructions to:

generate an inertial vector as an inertial function of a direction of movement of the hand of the human subject during a time period that includes a transition of the hand from the first conformation; and

inertially scroll the user interface in the direction of the inertial vector.

13. The storage machine of claim 7, further holding instructions to inertially scroll the user interface with increased speed in a current scroll direction responsive to subsequent movements of the hand of the human subject over a threshold distance in a direction that is substantially similar to the current scroll direction.

14. A method for presenting a user interface, comprising:

outputting a user interface to a display device;

receiving a virtual skeleton of a human subject modeled from a depth image from a depth camera, the virtual skeleton including a virtual hand joint modeling a three-dimensional position of a hand of the human subject;

analyzing a portion of the depth image corresponding to a position of the virtual hand joint to determine whether the hand of the human subject is in a closed-grip conformation or in an open-grip conformation;

when the hand is in the closed-grip conformation, scrolling the user interface in each of three dimensions responsive to movement of the virtual hand joint in each of three dimensions;

when the hand is in the open-grip conformation, targeting objects of the user interface responsive to movement of the virtual hand joint without scrolling the user interface;

initiate inertial scrolling of the user interface as a scrolling function of a velocity of the hand of the human subject during a time period that includes a transition of the hand from the closed-grip conformation; and

inertially scrolling the user interface in three dimensions in a direction of an inertial vector generated as an inertial function of a direction of movement of the hand of the human subject in three dimensions during the time period that includes a transition of the hand from the closed-grip conformation.

15. The method of claim 14, wherein the inertial vector is constrained to a projection along one or more dimensions during a condition where the user interface is constrained to inertial scrolling in one or two dimensions.

16. The method of claim 14, further comprising:

if the hand is in the closed-grip conformation, simultaneously scrolling the user interface in two or more of three dimensions responsive to movement of the virtual hand joint in a corresponding two or more of three dimensions.