SKELETAL CONTROL OF THREE-DIMENSIONAL VIRTUAL WORLD

• US 20120157203A1
• Filed: 12/21/2010
• Published: 06/21/2012
• Est. Priority Date: 12/21/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Latta, Stephen, Geisner, Kevin, Markovic, Relja, Bennett, Darren
• Granted Patent: US 8,994,718 B2
• Time in Patent Office: Days
• US Class Current: 463/32
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

A virtual skeleton includes a plurality of joints and provides a machine readable representation of a human target observed with a three-dimensional depth camera. A relative position of a hand joint of the virtual skeleton is translated as a gestured control, and a three-dimensional virtual world is controlled responsive to the gestured control.

Claims

1. A data holding device holding instructions executable by a logic subsystem to:

render a three-dimensional virtual gaming world for display on a display device;

receive a virtual skeleton, including a plurality of joints, the plurality of joints including a left hand joint and a right hand joint, the virtual skeleton providing a machine readable representation of a human target observed with a three-dimensional depth camera;

render a control cursor in the three-dimensional virtual gaming world for display on the display device, a screen space position of the control cursor tracking a position of the left hand joint or the right hand joint of the virtual skeleton as modeled from a world space position of a corresponding hand of the human target;

lock the control cursor to an object in the three-dimensional virtual gaming world if a grab threshold of the object is overcome;

when the control cursor is locked to the object, move the object with the control cursor such that the world space position, of the corresponding hand of the human target moves the object in the three-dimensional virtual gaming world; and

unlock the control cursor from the object at a release position of the object within the three-dimensional virtual gaming world if a release threshold of the object is overcome.

2. The data holding device of claim 1, where world space parameters of the corresponding hand overcome the grab threshold of the object if the corresponding hand is closed by the human target.

3. The data holding device of claim 1, where world space parameters of the corresponding hand overcome the grab threshold of the object if tire screen space position of the corresponding hand is within a threshold distance of the object for a duration threshold.

4. The data holding device of claim 1, where world space parameters of the corresponding hand overcome the grab threshold of the object if the screen space position of the corresponding hand is within a threshold distance of the object and a speed of the corresponding hand is less than a speed threshold for a duration threshold.

5. The data holding device of claim 1, where world space parameters of the corresponding hand overcome the release threshold if the corresponding hand is opened by the human target.

6. The data holding device of claim 1, further holding instructions executable by the logic subsystem to:

when the control cursor is locked to the object, rotate the object in the three-dimensional virtual gaming world responsive to a rotation gesture of the virtual skeleton such that a world space gesture of the human target rotates the object in the three-dimensional virtual gaming world.

7. The data holding device of claim 1, further holding instructions executable by the logic subsystem to:

when the control, cursor is locked to the object, scale the object in the three-dimensional virtual gaming world responsive to a scale gesture of the virtual skeleton such that a world space gesture of the human target scales the object in the three-dimensional virtual gaming world.

8. The data holding device of claim 1, further holding instructions executable by the logic subsystem to:

select a plurality of objects in the three-dimensional virtual gaming world responsive to a selection gesture of the virtual skeleton;

lock the control cursor to the plurality of objects if a grab threshold of tire plurality of objects is overcome;

when the control cursor is locked to the plurality of objects, move the plurality of objects with the control cursor such that the world space position of the corresponding hand of the human target moves the plurality of objects in the three-dimensional virtual gaming world; and

unlock the control cursor from the plurality of objects at a release position of the plurality of objects within the three-dimensional virtual gaming world if a release threshold of the plurality of objects is overcome.

9. The data holding device of claim 8, further holding instructions executable by the logic subsystem:

when the control cursor is locked to the plurality of objects, rotate the plurality of objects in the three-dimensional virtual gaming world responsive to a rotation gesture of the virtual skeleton such that a world space gesture of the human target rotates the plurality of objects in the three-dimensional virtual gaming world.

10. The data holding device of claim 8, further holding instructions executable by the logic subsystem to:

when the control cursor is locked to the plurality of objects, scale the plurality of objects in the three-dimensional virtual gaming world responsive to a scale gesture of the virtual skeleton such that a world space gesture of the human target scales the plurality of objects in the three-dimensional virtual gaming world.

11. A data holding device holding instructions executable by a logic subsystem to:

render a three-dimensional virtual gaming world for display on a display device;

receive a virtual skeleton including a plurality of joints, the plurality of joints including a left hand joint and a right hand joint, the virtual skeleton providing a machine readable representation of a human target observed with a three-dimensional depth camera;

render a control cursor in the three-dimensional virtual gaming world for display on the display device, a screen space position of the control cursor tracking a position of the left hand joint or the right hand joint of the virtual skeleton as modeled from a world space position of a corresponding hand of the human target; and

responsive to the screen space position of the control cursor reaching a scrolling threshold at a visible edge of the three-dimensional virtual gaming world, scrolling the three-dimensional virtual gaming world such that a previously hidden portion of the three-dimensional virtual gaming world adjacent to the visible edge of the three-dimensional virtual gaming world becomes unhidden.

12. The data holding device of claim 11, further holding instructions executable by the logic subsystem to:

translate a zoom-in gesture into a gestured magnify view control; and

magnify a view of the three-dimensional virtual gaming world responsive to the gestured magnify view control,

13. The data holding device of claim 12, further holding instructions executable by the logic subsystem to:

render a left control cursor in the three-dimensional virtual gaming world for display on the display device, a screen space position of the left-control cursor franking a position of the left hand joint of the virtual skeleton as modeled from a world space position of a left hand of the human, target;

render a right control cursor in the three-dimensional virtual gaming world for display on the display device, a screen space position of the right control cursor tracking a position of the right hand joint of the virtual skeleton as modeled from a world space position of a right hand of the human target;

responsive to the zoom-in gesture, the screen space position of the left control cursor and the screen space position, of the right control cursor moving apart as the view of the three-dimensional virtual gaming world magnifies.

14. The data holding device of claim 11, further holding instructions executable by the logic subsystem to:

translate a zoom-out gesture into a gestured shrink view control; and

shrink a view of the three-dimensional virtual gaming world responsive to the gestured shrink view control.

15. The data holding device of claim 14, further holding instructions executable by the logic subsystem to:

render a left control cursor in the three-dimensional virtual gaming world for display on the display device, a screen space position of the left control cursor tracking a position of the left hand joint of the virtual skeleton as modeled from a world space position of a left hand of the human target;

render a right control cursor in the three-dimensional, virtual gaming world for display on the display device, a screen space position of the right control cursor tracking a position of the right hand joint of the virtual skeleton as modeled from a world space position of a right hand of the human target;

responsive to the zoom-out gesture, the screen space position of the left control cursor and the screen space position of the right control cursor moving together as the view of the three-dimensional virtual gaming world shrinks.

16. The data holding device of claim 11, farther holding instructions executable by the logic subsystem to:

lock the control cursor to an object in the three-dimensional virtual gaming world if a grab threshold of the object is overcome;

when the control cursor is locked to the object, move the object with the control cursor such that the world space position of the corresponding hand of the human target moves the object in the three-dimensional virtual gaming world; and

unlock the control cursor from the object at a release position of the object within the three-dimensional virtual gaming world if a release threshold of the object is overcome.

17. The data holding device of claim 18, where world space parameters of the corresponding hand overcome the grab threshold of the object if the corresponding hand is closed by the human target,

18. The data holding device of claim 16, where world space parameters of the corresponding hand overcome the grab threshold of the object if the screen space position of the corresponding hand is within a threshold distance of the object for a duration threshold.

19. The data holding device of claim 16, where world space parameters of the corresponding hand overcome the release threshold if the corresponding hand is opened by the human target.

20. A data holding device holding instructions executable by a logic subsystem to:

render a three-dimensional virtual gaming world for display on a display device;

receive a virtual skeleton including a plurality of joints, the virtual skeleton providing a machine readable representation of a human target observed with a three-dimensional depth camera;

responsive to recognizing a first mode initiation gesture of the virtual skeleton, setting a first input modality;

while the first input modality is set, interpreting a specific action gesture of the virtual skeleton, as a first gesture control;

responsive to recognizing a second mode initiation gesture of the virtual skeleton, setting a second input modality; and

while the second input modality is set, interpreting the specific action gesture of the virtual skeleton as a second gesture control, the first gesture control and the second gesture control producing different controls of the three-dimensional virtual world.