Systems and methods of rerendering image hands to create a realistic grab experience in virtual reality/augmented reality environments

• US 10,607,413 B1
• Filed: 09/02/2016
• Issued: 03/31/2020
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Marcolina, Alex, Holz, David
• Primary Examiner(s): He, Weiming
• Application Number:
  US15/256,446
• Time in Patent Office: 1,306 Days
• Field of Search: None
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed relates to a method of realistic rendering of a real object as a virtual object in a virtual space using an offset in the position of the hand in a three-dimensional (3D) sensory space. An offset between expected positions of the eye(s) of a wearer of a head mounted device and a sensor attached to the head mounted device for sensing a position of at least one hand in a three-dimensional (3D) sensory space is determined. A position of the hand in the three-dimensional (3D) sensory space can be sensed using a sensor. The sensed position of the hand can be transformed by the offset into a re-rendered position of the hand as would appear to the wearer of the head mounted device if the wearer were looking at the actual hand. The re-rendered hand can be depicted to the wearer of the head mounted device.

Claims

1. A system of realistic displacement of a virtual object to render a realistic representation of a hand in a three-dimensional (3D) sensory space as a virtual object in a virtual space, the system including:

a head mounted device depicting at least a visual presentation to a wearer;

a hand-gesture sensor attached to the head mounted device; and

a processor and a computer readable storage medium storing computer instructions configured for performing;

determining an offset between expected positions of one or more eyes of a wearer of a head mounted device and an actual position of a hand-gesture sensor attached to the head mounted device;

using the hand-gesture sensor, obtaining a sequence of images correlated to construct a 3-D solid model of the hand, including its position and shape, and using the 3-D solid model of the hand computing a sensed position of hand in a three-dimensional (3D) sensory space, wherein the computing includes determining positions of fingers, thumb and palm of the hand;

calculating a transformation matrix using the determined offset, such that the transformation matrix, when applied to the sensed position of the hand, decreases a rendered size of the hand in accordance with the determined offset;

applying the transformation matrix to the sensed position of the hand to transform the sensed position of the hand into a re-rendered position of the hand, the re-rendered position of the hand placing in the 3D sensory space the positions of the fingers, thumb and palm of the re-rendered position of the hand at the offset with respect to the sensed position of the hand such that the wearer of the head mounted device perceives the re-rendered position of the hand situated in the 3D sensory space as expected by the wearer when the wearer is looking at the actual hand in the 3D sensory space; and

depicting to the wearer of the head mounted device a re-rendered image hand using the re-rendered position of the hand.

2. The system of claim 1, further configured to perform:

sensing one or more additional positions of the hand and determining therefrom motion or gesture; and

applying the offset to the additional positions of the hand to transform the sensed additional positions to re-rendered positions of the hand as seen by the wearer, thereby providing the re-rendered image hand having the motion or gesture of the hand.

3. The system of claim 2, further configured to perform:

sensing an indication of quality of tracking for the hand-gesture sensor, the indication of quality of tracking indicating whether the hand-gesture sensor is tracking position or motion of the hand; and

adding to the re-rendered image hand a visual indication corresponding to the indication of quality of tracking;

thereby providing the wearer with a visual cue of how well the hand-gesture sensor is tracking the hand.

4. The system of claim 3, wherein the indication of quality of tracking includes an indication that the hand-gesture sensor is able to recognize the hand in captured images.

5. The system of claim 4, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is able to recognize the hand in captured images.

6. The system of claim 4, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is unable to recognize the hand in captured images.

7. The system of claim 1, wherein the transformation matrix is an affine matrix.

8. A method of realistic displacement of a virtual object to render a realistic representation of a hand in a three-dimensional (3D) sensory space as a virtual object in a virtual space, the method including:

determining an offset between expected positions of one or more eyes of a wearer of a head mounted device and an actual position of a hand-gesture sensor attached to the head mounted device;

using the hand-gesture sensor, obtaining a sequence of images correlated to construct a 3-D solid model of the hand, including its position and shape, and using the 3-D solid model of the hand computing a sensed position of at least one hand in a three-dimensional (3D) sensory space, wherein the computing includes determining positions of fingers, thumb and palm of the hand;

calculating a transformation matrix using the determined offset, such that the transformation matrix, when applied to the sensed position of the hand, decreases a rendered size of the hand in accordance with the determined offset;

applying the transformation matrix to the sensed position of the hand to transform the sensed position of the hand into a re-rendered position of the hand, the re-rendered position of the hand placing in the 3D sensory space the positions of the fingers, thumb and palm of the re-rendered position of the hand at the offset with respect to the sensed position of the hand such that the wearer of the head mounted device perceives the re-rendered position of the hand situated in the 3D sensory space as expected by the wearer when the wearer is looking at the actual hand in the 3D sensory space; and

depicting to the wearer of the head mounted device a re-rendered image hand using the re-rendered position of the hand.

9. The method of claim 8, further including:

sensing one or more additional positions of the hand and determining therefrom motion or gesture; and

applying the offset to the additional positions of the hand to transform the sensed additional positions to re-rendered positions of the hand as seen by the wearer, thereby providing the re-rendered image hand having the motion or gesture of the hand.

10. The method of claim 9, further including:

sensing an indication of quality of tracking for the hand-gesture sensor, the indication of quality of tracking indicating whether the hand-gesture sensor is tracking position or motion of the hand; and

adding to the re-rendered image hand a visual indication corresponding to the indication of quality of tracking;

thereby providing the wearer with a visual cue of how well the hand-gesture sensor is tracking the hand.

11. The method of claim 10, wherein the indication of quality of tracking includes an indication that the hand-gesture sensor is able to recognize the hand in captured images.

12. The method of claim 11, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is able to recognize the hand in captured images.

13. The method of claim 11, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is unable to recognize the hand in captured images.

14. One or more non-transitory computer readable media having instructions stored thereon for performing realistic displacement of a virtual object to render a realistic representation of a hand in a three-dimensional (3D) sensory space as a virtual object in a virtual space, the performing including:

determining an offset between expected positions of one or more eyes of a wearer of a head mounted device and an actual position of a hand-gesture sensor attached to the head mounted device;

using the hand-gesture sensor, obtaining a sequence of images correlated to construct a 3-D solid model of the hand, including its position and shape, and using the 3-D solid model of the hand computing a sensed position of at least one hand in a three-dimensional (3D) sensory space, wherein the computing includes determining positions of fingers, thumb and palm of the hand;

calculating a transformation matrix using the determined offset, such that the transformation matrix, when applied to the sensed position of the hand, decreases a rendered size of the hand in accordance with the determined offset;

applying the transformation matrix to the sensed position of the hand to transform the sensed position of the hand into a re-rendered position of the hand, the re-rendered position of the hand placing in the 3D sensory space the positions of the fingers, thumb and palm of the re-rendered position of the hand at the offset with respect to the sensed position of the hand such that the wearer of the head mounted device perceives the re-rendered position of the hand situated in the 3D sensory space as expected by the wearer when the wearer is looking at the actual hand in the 3D sensory space; and

depicting to the wearer of the head mounted device a re-rendered image hand using the re-rendered position of the hand.

15. The non-transitory computer readable media of claim 14, the instructions further implementing performing:

sensing one or more additional positions of the hand and determining therefrom motion or gesture; and

applying the offset to the additional positions of the hand to transform the sensed additional positions to re-rendered positions of the hand as seen by the wearer, thereby providing the re-rendered image hand having the motion or gesture of the hand.

16. The non-transitory computer readable media of claim 15, the instructions further implementing performing:

sensing an indication of quality of tracking for the hand-gesture sensor, the indication of quality of tracking indicating whether the hand-gesture sensor is tracking position or motion of the hand; and

adding to the re-rendered image hand a visual indication corresponding to the indication of quality of tracking;

thereby providing the wearer with a visual cue of how well the hand-gesture sensor is tracking the hand.

17. The non-transitory computer readable media of claim 16, wherein the indication of quality of tracking includes an indication that the hand-gesture sensor is able to recognize the hand in captured images.

18. The non-transitory computer readable media of claim 17, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is able to recognize the hand in captured images.

19. The non-transitory computer readable media of claim 17, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is unable to recognize the hand in captured images.

20. The non-transitory computer readable media of claim 17, wherein the visual indication corresponding to the indication of quality of tracking includes coloring the re-rendered image hand a different color when the hand-gesture sensor is able to recognize the hand in captured images than when the hand-gesture sensor is unable to recognize the hand in captured images.

21. The non-transitory computer readable media of claim 15, the instructions further implementing performing:

sensing an indication of whether the hand has grabbed a virtual object; and

adding to the re-rendered image hand a visual indication corresponding to the indication of whether the hand has grabbed the virtual object;

thereby providing the wearer with a visual cue of whether the virtual object is grabbed.

22. The non-transitory computer readable media of claim 21, the instructions further implementing performing:

depicting a stretched version of the virtual object grabbed.