Ergonomic physical interaction zone cursor mapping

• US 9,342,160 B2
• Filed: 05/26/2015
• Issued: 05/17/2016
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bailey, Richard, Bastien, David, Schwesinger, Mark, Yang, Emily, Smith, Adam, Murillo, Oscar, Franklin, Tim, Andersen, Jordan, Klein, Christian
• Primary Examiner(s): HICKS, CHARLES V
• Application Number:
  US14/721,471
• Publication Number:
  US 20150370349A1
• Time in Patent Office: 357 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Users move their hands in a three dimensional (“3D”) physical interaction zone (“PHIZ”) to control a cursor in a user interface (“UI”) shown on a computer-coupled 2D display such as a television or monitor. The PHIZ is shaped, sized, and positioned relative to the user to ergonomically match the user'"'"'s natural range of motions so that cursor control is intuitive and comfortable over the entire region on the UI that supports cursor interaction. A motion capture system tracks the user'"'"'s hand so that the user'"'"'s 3D motions within the PHIZ can be mapped to the 2D UI. Accordingly, when the user moves his or her hands in the PHIZ, the cursor correspondingly moves on the display. Movement in the z direction (i.e., back and forth) in the PHIZ allows for additional interactions to be performed such as pressing, zooming, 3D manipulations, or other forms of input to the UI.

Claims

1. A computer-implemented method for moving a cursor on a two-dimensional (“

2D”

) display in response to motion of a user'"'"'s hands, the method comprising the steps of;

capturing locations of each of the user'"'"'s hands within respective monitored physical interaction zones (PHIZs), each of the PHIZs being ergonomically matched to the user'"'"'s natural range of motions;

tracking the locations for each of the user'"'"'s hands within each PHIZ to determine motion of each of the user'"'"'s hands; and

mapping the tracked hand locations from either one of the PHIZs to the display so that motion of either of the user'"'"'s hands in a respective PHIZ results in a corresponding motion of the cursor on the display.

2. The computer-implemented method of claim 1 further including a step of generating a plurality of tuning parameters, the tuning parameters enabling a size, shape, or location of the PHIZ to be tuned to a given configuration of the user within an area of captured motion.

3. The computer-implemented method of claim 2 in which the configuration of the user is one of standing, seated, lying down, having unconstrained whole arm motion, having constrained whole arm motion, having constrained forearm motion, or having constrained wrist motion.

4. The computer-implemented method of claim 2 in which the tuning parameters include at least one of adjustment for horizontal centering of the PHIZ relative to the user, adjustment of vertical centering of the PHIZ relative to the user, adjustment for horizontal reach of the user within the PHIZ, adjustment for vertical reach of the user within the PHIZ, adjustment of the user'"'"'s reach when at a middle location within a vertical dimension of the PHIZ, adjustment of a backplane of the PHIZ, or adjustment of a forward plane of the PHIZ.

5. The computer-implemented method of claim 2 further including a step of dynamically adjusting the tuning parameters based on context.

6. The computer-implemented method of claim 5 in which the context includes one of system context or application context, the application context including at least one of a quantity, type, or position of user-accessible controls displayed on a user interface (“

UI”

), aspect ratio, supported area of cursor movement within the UI, or application behavior in response to user input.

7. The computer-implemented method of claim 1 further including a step of utilizing a whole arm PHIZ in which motion of the user'"'"'s hand is determined relative to a shoulder joint of the user.

8. The computer-implemented method of claim 1 further including a step of utilizing a forearm PHIZ in which motion of the user'"'"'s hand is determined relative to an elbow joint of the user.

10. The computer-implemented method of claim 1 further including the steps of detecting hand motion in a z direction in the PHIZ and utilizing the detected motion as an indication of a press gesture.

11. The computer-implemented method of claim 1 further including a step of dynamically selecting among a plurality of PHIZs, each PHIZ in the plurality being shaped for tracking ergonomic motion of the user'"'"'s hand about a different arm joint, the arm joint comprising one of shoulder, elbow, or wrist.

12. The computer-implemented method of claim 11 further including a step of transitioning from hand motion data from one PHIZ in the plurality to hand motion data from another PHIZ in either a discrete or a continuous manner.

13. The computer-implemented method of claim 11 further including the steps of obtaining data describing hand motion from two or more of the PHIZs in the plurality and weighting the obtained data to determine a final cursor position on the display in which the weighting comprises observing changes in hand position in each of x and y directions over a time interval to determine a normalized PHIZ velocity for each PHIZ in the plurality and giving greater weight to a PHIZ having a higher velocity.

9. The computer-implemented method of claim l further including a step of utilizing a hand PHIZ in which motion of the user'"'"'s hand or fingers is determined relative to a wrist joint of the user.

14. One or more computer-readable storage media storing instructions which, when executed on one or more processors disposed in a computing device perform a method for controlling a movement of a cursor on a user interface (“

UI”

), the method comprising the steps of;

using an optical sensor to detect a position of a hand or fingers of a subject within one or more of a plurality of three-dimensional (“

3D”

) physical interaction zones (“

PHIZs”

) located in a real world space, each PHIZ being shaped, sized, and located relative to the subject to enable ergonomic motion of the subject'"'"'s hand or fingers throughout the PHIZ'"'"'s volume;

dynamically selecting one of the plurality of PHIZs, each PHIZ in the plurality being shaped, sized, and located relative to the subject for tracking ergonomic motion of the subject'"'"'s hand or fingers about different arm joints, the arm joints comprising one of shoulder, elbow, or wrist;

mapping the detected hand or finger position to a cursor location in a user interface (“

UI”

) supported by a two-dimensional (“

2D”

) display; and

moving the cursor in the UI in correspondence with the subject'"'"'s hand or finger motion within the selected PHIZ.

15. The one or more computer-readable storage media of claim 14 in which the method further includes a step of calibrating the space by observing the subject performing a press at a given location on the UI, the press being performed by motion of the subject'"'"'s hand or finger in a z direction within the selected PHIZ.

16. The one or more computer-readable storage media of claim 14 further including dynamically applying tuning parameters to the selected PHIZ to adjust at least one of shape of the PHIZ, size of the PHIZ, or location of the PHIZ relative to the subject, the dynamic application being dependent on one of the subject'"'"'s position in the space, orientation in the space, distance from the optical sensor, past behavior, or UI context.

17. A system for mapping motion of a user'"'"'s hand in a three-dimensional (“

3D”

) space to motion of a cursor in a two-dimensional (“

2D”

) user interface (UI), comprising;

at least one processor;

a display configured to support at least a visible portion of the UI;

an optical sensor configured to capture an orientation of the user within a physical space and for capturing locations of the user'"'"'s hand within a 3D physical interaction zone (“

PHIZ”

) that is configured so that the user is able to move the hand to reach all points within the PHIZ in an ergonomic manner; and

memory bearing executable instructions that, when executed by the at least one processor, perform a method comprising the steps of;

using the optical sensor, determining the location of the hand within the PHIZ relative to a known point on the user'"'"'s body using a 3D coordinate system,

mapping the hand location in 3D coordinates in the PHIZ to 2D coordinates in the UI wherein the 2D coordinates include non-visible portions of the UI, and

repeating the determining and mapping steps to move the cursor in the UI at a 2D coordinate to correspond with motion of the user'"'"'s hand.

18. The system of claim 17 in which the method further includes a step of determining motion of the user'"'"'s hand in a z direction in the PHIZ to enable interaction with a user interface (“

UI”

) on the display, the interacting including one of pressing, zooming, 3D manipulation, or input to the UI.

19. The system of claim 17 in which the non-visible portions of the UI are associated with an object that is larger than the display.

20. The system of claim 17 further comprising tracking a distance of a 2D coordinate from an edge of the display and enabling panning of the object by the user in which panning velocity is dependent on the distance.