Systems and methods of locating a control object appendage in three dimensional (3D) space

• US 9,945,660 B2
• Filed: 05/27/2015
• Issued: 04/17/2018
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S.
• Primary Examiner(s): Moyer, Andrew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US14/723,370
• Publication Number:
  US 20150287204A1
• Time in Patent Office: 1,056 Days
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: G01B 11/2433 for measuring outlines by shadow casting
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/507 from shading G06T7/586 takes precedence
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods and systems for capturing motion and/or determining the shapes and positions of one or more objects in 3D space utilize cross-sections thereof. In various embodiments, images of the cross-sections are captured using a camera based on reflections therefrom or shadows cast thereby.

Claims

1. A system of locating a control object appendage in three dimensional (3D) space, the system including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

recording images of a control object appendage in 3D space using at least two geometrically distinct predetermined vantages;

calculating by a processor four co planar tangents to observed edges of the control object appendage from the recorded images;

fitting an ellipse by a processor to a cross-section of the control object appendage by selecting the ellipse from a family of ellipses that fit the four co planar tangents using an assumed parameter;

repeatedly fitting contiguous cross-sections to the control object appendage; and

constructing a model of the control object appendage in 3D space from the contiguous cross-sections.

2. The system of claim 1, further configured to substitute one or more fitted parameters of a first fitted ellipse for one or more of the four co planar tangents and the assumed parameter when fitting an adjacent second ellipse.

3. The system of claim 1, further configured to use a first fitted ellipse to filter fits of additional ellipses to the contiguous cross-sections.

4. The system of claim 2, further including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

for a complex control object model that includes a palm and multiple fingers, applying the actions of claim 2 to construct multiple fingers of control object appendages; and

fitting cross sections of a palm to observed edges from the images in positions correlated with the multiple fingers.

5. The system of claim 1, further including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

repeatedly applying the actions of claim 1 over time; and

calculating motion of the control object appendage over time based on differences between modeled locations of the control object appendage over time.

6. The system of claim 3, further including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

repeatedly applying the actions of claim 3 over time; and

calculating motion of a complex control object over time based on differences between modeled locations of a complex control object over time.

7. The system of claim 1, further configured to:

determine that only three co planar tangents are available in portions of the recorded images and fitting a circle instead of an ellipse for the portions of the recorded images, including;

calculating by a processor the three co planar tangents to observed edges of the control object appendage from the recorded images; and

fitting a circle by a processor to the cross-section of the control object appendage using at least the three co planar tangents.

8. The system of claim 7, further including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

for a complex control object model that includes a palm and multiple fingers, applying the actions of claim 7 to construct multiple fingers of control object appendages; and

fitting cross sections of a palm to observed edges from the images in positions correlated with the multiple fingers.

9. The system of claim 7, further including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

repeatedly applying the actions of claim 7 over time; and

calculating motion of the control object appendage over time based on differences between modeled locations of the control object appendage over time.

10. The system of claim 8, further including:

one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions including;

repeatedly applying a actions of claim 8 over time; and

calculating motion of a complex control object over time based on differences between modeled locations of the complex control object over time.

11. A non-transitory computer readable medium storing a plurality of instructions for programming one or more processors to locate a control object appendage in three dimensional (3D) space, the instructions, when executed on the processors, implementing actions including:

recording images of a control object appendage in 3D space using at least two geometrically distinct predetermined vantages;

calculating by a processor four co planar tangents to observed edges of the control object appendage from the recorded images;

fitting an ellipse by a processor to a cross-section of the control object appendage by selecting the ellipse from a family of ellipses that fit the four co planar tangents using an assumed parameter;

repeatedly fitting contiguous cross-sections to the control object appendage; and

constructing a model of the control object appendage in 3D space from the contiguous cross-sections.

12. The non-transitory computer readable medium of claim 11, further configured to substitute one or more fitted parameters of a first fitted ellipse for one or more of the four co planar tangents and the assumed parameter when fitting an adjacent second ellipse.

13. The non-transitory computer readable medium of claim 11, further configured to use a first fitted ellipse to filter fits of additional ellipses to the contiguous cross-sections.

14. The non-transitory computer readable medium of claim 12, further including storing a plurality of instructions for programming one or more processors to locate a complex control object in 3D space, the instructions, when executed on the processors, implementing actions including:

for a complex control object model that includes a palm and multiple fingers, applying the actions of claim 12 to construct multiple fingers of control object appendages; and

fitting cross sections of a palm to observed edges from the images in positions correlated with the multiple fingers.

15. The non-transitory computer readable medium of claim 11, further including storing a plurality of instructions for programming one or more processors to track motion of a control object appendage in 3D space, the instructions, when executed on the processors, implementing actions including:

repeatedly applying the actions of claim 11 over time; and

calculating motion of the control object appendage over time based on differences between modeled locations of the control object appendage over time.

16. The non-transitory computer readable medium of claim 13, further including storing a plurality of instructions for programming one or more processors to locate a complex control object in 3D space, the instructions, when executed on the processors, implementing actions including:

repeatedly applying the actions of claim 13 over time; and

calculating motion of a complex control object over time based on differences between modeled locations of a complex control object over time.

17. The non-transitory computer readable medium of claim 11, further configured to:

determine that only three co planar tangents are available in portions of the recorded images and fitting a circle instead of an ellipse for the portions of the recorded images, including;

calculating by a processor the three co planar tangents to observed edges of the control object appendage from the recorded images; and

fitting a circle by a processor to the cross-section of the control object appendage using at least the three co planar tangents.

18. The non-transitory computer readable medium of claim 17, further including storing a plurality of instructions for programming one or more processors to locate a complex control object in 3D space, the instructions, when executed on the processors, implementing actions including:

for a complex control object model that includes a palm and multiple fingers, applying the actions of claim 17 to construct multiple fingers of control object appendages; and

fitting cross sections of a palm to observed edges from the images in positions correlated with the multiple fingers.

19. The non-transitory computer readable medium of claim 17, further including storing a plurality of instructions for programming one or more processors to track motion of a control object appendage in 3D space, the instructions, when executed on the processors, implementing actions including:

repeatedly applying the actions of claim 17 over time; and

calculating motion of the control object appendage over time based on differences between modeled locations of the control object appendage over time.

20. The non-transitory computer readable medium of claim 18, further including storing a plurality of instructions for programming one or more processors to track motion of a complex control object appendage in 3D space, the instructions, when executed on the processors, implementing actions including:

repeatedly applying the actions of claim 18 over time; and

calculating motion of a complex control object over time based on differences between modeled locations of a complex control object over time.