SYSTEMS AND METHODS OF CREATING A REALISTIC GRAB EXPERIENCE IN VIRTUAL REALITY/AUGMENTED REALITY ENVIRONMENTS

• US 20160239080A1
• Filed: 02/19/2015
• Published: 08/18/2016
• Est. Priority Date: 02/13/2015
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: MARCOLINA, Alex, DURDIK, Paul A., HOLZ, David S.
• Granted Patent: US 9,696,795 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

The technology disclosed relates to a method of realistic rotation of a virtual object for an interaction between a control object in a three-dimensional (3D) sensory space and the virtual object in a virtual space that the control object interacts with. In particular, it relates to detecting free-form gestures of a control object in a three-dimensional (3D) sensory space and generating for display a 3D solid control object model for the control object during the free-form gestures, including sub-components of the control object and in response to detecting a two e sub-component free-form gesture of the control object in the 3D sensory space in virtual contact with the virtual object, depicting, in the generated display, the virtual contact and resulting rotation of the virtual object by the 3D solid control object model.

Claims

1. A method of manipulating virtual objects using real motions of one or more hands in a three-dimensional (3D) sensory space, the method including:

capturing an image of the hands in the a three-dimensional (3D) sensory space and sensing a location of the hands;

incorporating the image of the hands into a virtual reality scene; and

outlining a modeled position of the location of the hands and incorporating the outline into the virtual reality scene.

2. The method of claim 1, further including changing an appearance of the outline upon detection of a discrepancy between the image of the hands and the outline.

3. The method of claim 1, further including changing an appearance of the image of the hands upon detection of a discrepancy between the image of the hands and the outline.

17. A system implementing the method of claim 1, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

capture an image of the hands in the a three-dimensional (3D) sensory space and sense a location of the hands;

incorporate the image of the hands into a virtual reality scene; and

outline a modeled position of the location of the hands and incorporate the outline into the virtual reality scene.

18. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 1, the method including:

capturing an image of the hands in the a three-dimensional (3D) sensory space and sensing a location of the hands;

incorporating the image of the hands into a virtual reality scene; and

outlining a modeled position of the location of the hands and incorporating the outline into the virtual reality scene.

4. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:

capturing an image of at least one hand in a three-dimensional (3D) sensory space and sensing a location of a first hand;

incorporating the image of the first hand into a virtual reality scene; and

sensing a pinch action between a thumb and first finger of the first hand and rendering a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.

5. The method of claim 4, further including:

sensing a pinch action between a thumb and first finger of a second hand and rendering a second virtual pinch force image positioned between the thumb and the first finger while the pinch action continues;

sensing a movement of at least one of the first and second hands that increases a separation distance between the first and second hands, while sensing continuing pinching actions of both the first and second hands; and

rendering a new virtual object between the first and second pinch force images, responsive to the increased separation distance, wherein at least a size of the new virtual object is responsive to the separation distance.

6. The method of claim 5, wherein an orientation of the new virtual object is responsive to positions of the first and second hands.

19. A system implementing the method of claim 4, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

capture an image of at least one hand in a three-dimensional (3D) sensory space and sense a location of a first hand;

incorporate the image of the first hand into a virtual reality scene; and

sense a pinch action between a thumb and first finger of the first hand and render a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.

20. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 4, the method including:

capturing an image of at least one hand in a three-dimensional (3D) sensory space and sensing a location of a first hand;

incorporating the image of the first hand into a virtual reality scene; and

sensing a pinch action between a thumb and first finger of the first hand and rendering a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.

7. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:

capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;

incorporating the image of the first hand into a virtual reality scene that includes a grabable virtual object;

sensing a gesture of the first hand and determining whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determining further including taking into account at least;

an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;

fingertip separations between the fingertips of the first hand and the proximate surface;

a gesture rate at which the first hand closes on the virtual object;

a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and

linear velocity of the palm of the first hand relative to the virtual object; and

responsive to determining that a gesture of the first hand is intended to grab the virtual object, linking motion of the first hand to manipulation of the virtual object.

8. The method of claim 7, further including taking into account when determining whether the gesture is intended to interact with the grabable virtual object a maximum separation between any of the fingertips of the first hand and the proximate surface.

9. The method of claim 7, further including taking into account when determining whether the gesture is intended to interact with the grabable virtual object a rotational velocity of the palm of the first hand in the 3D sensory space.

21. A system implementing the method of claim 7, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of a first hand;

incorporate the image of the first hand into a virtual reality scene that includes a grabable virtual object;

sense a gesture of the first hand and determine whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determination further including taking into account at least;

an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;

fingertip separations between the fingertips of the first hand and the proximate surface;

a gesture rate at which the first hand closes on the virtual object;

a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and

linear velocity of the palm of the first hand relative to the virtual object; and

responsive to the determination that a gesture of the first hand is intended to grab the virtual object, link motion of the first hand to manipulation of the virtual object.

22. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 7, the method including:

capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;

incorporating the image of the first hand into a virtual reality scene that includes a grabable virtual object;

sensing a gesture of the first hand and determining whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determining further including taking into account at least;

an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;

fingertip separations between the fingertips of the first hand and the proximate surface;

a gesture rate at which the first hand closes on the virtual object;

a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and

linear velocity of the palm of the first hand relative to the virtual object; and

responsive to determining that a gesture of the first hand is intended to grab the virtual object, linking motion of the first hand to manipulation of the virtual object.

10. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:

capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;

incorporating the image of the first hand into a virtual reality scene that includes a pushable virtual object;

sensing a gesture of the first hand and determining whether the gesture is intended to interact with the pushable virtual object by pushing the virtual object, the determining further including taking into account at least;

an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;

fingertip separations between the fingertips of the first hand and the proximate surface;

a hand posture, whether suitable for pushing the virtual object or incompatible with pushing; and

linear velocity of the palm of the first hand relative to the virtual object; and

responsive to determining that a gesture of the first hand is intended to push the virtual object, linking motion of the first hand to manipulation of the virtual object.

11. The method of claim 10, further including taking into account when determining whether the gesture is intended to interact with the pushable virtual object a gesture rate at which the first hand closes on the virtual object.

12. The method of claim 10, further including taking into account when determining whether the gesture is intended to interact with the pushable virtual object a maximum separation between any of the fingertips of the first hand and the proximate surface.

13. The method of claim 10, further including taking into account when determining whether the gesture is intended to interact with the pushable virtual object a rotational velocity of the palm of the first hand in the 3D sensory space.

23. A system implementing the method of claim 10, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of a first hand;

incorporate the image of the first hand into a virtual reality scene that includes a pushable virtual object;

sense a gesture of the first hand and determine whether the gesture is intended to interact with the pushable virtual object by pushing the virtual object, the determination further including taking into account at least;

an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;

fingertip separations between the fingertips of the first hand and the proximate surface;

a hand posture, whether suitable for pushing the virtual object or incompatible with pushing; and

linear velocity of the palm of the first hand relative to the virtual object; and

responsive to the determination that a gesture of the first hand is intended to push the virtual object, link motion of the first hand to manipulation of the virtual object.

24. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 10, the method including:

capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;

incorporating the image of the first hand into a virtual reality scene that includes a pushable virtual object;

sensing a gesture of the first hand and determining whether the gesture is intended to interact with the pushable virtual object by pushing the virtual object, the determining further including taking into account at least;

an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;

fingertip separations between the fingertips of the first hand and the proximate surface;

a hand posture, whether suitable for pushing the virtual object or incompatible with pushing; and

linear velocity of the palm of the first hand relative to the virtual object; and

responsive to determining that a gesture of the first hand is intended to push the virtual object, linking motion of the first hand to manipulation of the virtual object.

14. A method of manipulating virtual objects using real motions of one or more hands in a three-dimensional (3D) sensory space, the method including:

capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of the hands;

incorporating at least part the image of the hands into a virtual reality scene;

outlining a modeled position of the location of the hands and incorporating the outline into the virtual reality scene;

detecting that at least part of the hands is obscured by a virtual object in virtual reality scene; and

rendering one of the outline and the image but not both where the hands are obscured.

15. The method of claim 14, further including changing an appearance of the outline as the part of the hands obscured changes.

16. The method of claim 14, further including changing an appearance of the image of the hands as the part of the hands obscured changes.

25. A system implementing the method of claim 14, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of the hands;

incorporate at least part the image of the hands into a virtual reality scene;

outline a modeled position of the location of the hands and incorporate the outline into the virtual reality scene;

detect that at least part of the hands is obscured by a virtual object in virtual reality scene; and

render one of the outline and the image but not both where the hands are obscured.

26. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 14, the method including:

capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of the hands;

incorporating at least part the image of the hands into a virtual reality scene;

outlining a modeled position of the location of the hands and incorporating the outline into the virtual reality scene;

detecting that at least part of the hands is obscured by a virtual object in virtual reality scene; and

rendering one of the outline and the image but not both where the hands are obscured.