Systems and methods of object shape and position determination in three-dimensional (3D) space

• US 9,881,386 B1
• Filed: 08/18/2017
• Issued: 01/30/2018
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US15/681,279
• Publication Number:
  US 20180033159A1
• Time in Patent Office: 165 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: G06V 40/11 : Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/117 : Biometrics derived from hands
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Methods and systems for capturing motion and/or determining the shapes and positions of one or more objects in 3D space utilize cross-sections thereof. In various embodiments, images of the cross-sections are captured using a camera based on edge points thereof.

Claims

1. A smart phone having an interface that identifies a position and a shape of a portion of a human hand moving in a three-dimensional (3D) space, the smart phone comprising:

a camera having a particular vantage point configured to view the human hand; and

a fixed function logic circuit storing instructions that, when executed, implement actions including;

capturing, with the camera from the particular vantage point, two or more images of a portion of a human hand moving in the 3D space;

analyzing the two or more images captured by the camera from the particular vantage point to computationally represent the portion of the human hand, as captured, as one or more mathematically represented 3D surfaces, each 3D surface corresponding to a cross-section of the portion of the human hand, based at least in part on a plurality of edge points of the portion of the human hand in the image, tangent lines extending from the camera to at least two edge points of the plurality of edge points, and a centerline corresponding to the tangent lines; and

reconstructing the position of, and the shape fitting, at least the portion of the human hand in the 3D space based at least in part on the plurality of edge points and the centerline.

2. The smart phone of claim 1, further including:

at least one source that casts an output onto the portion of the human hand.

3. The smart phone of claim 1, further including transmitting to at least one further process, a signal that includes at least one selected from (i) trajectory information determined from the reconstructed position of, and the shape fitting, the at least a portion of the human hand that the at least one further process interprets, and (ii) gesture information interpreted from trajectory information for the portion of the human hand by the smart phone.

4. The smart phone of claim 1, further comprising a time-of-flight camera, and wherein a plurality of points on at least one surface of the portion of the human hand are determined from time-of-flight data.

5. The smart phone of claim 4, wherein the fixed function logic circuit further implements actions that combine time-of-flight data with tangent-line information to provide a more detailed model of the shape of at least the portion of the human hand.

6. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that analyze the one or more surfaces to produce a 3D model.

7. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that refine a 3D model by correlating a cross-section of a human hand portion represented by a first surface with a cross-section of at least one different human hand portion represented by at least one different 3D surface.

8. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that refines a 3D model based on at least one of a human hand type of a human hand comprising the portion of the human hand and characteristics of the human hand type.

9. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that refines a 3D model by informing results of the analyzing with results of such analyzing as conducted on images obtained at a different time and correspondingly modifying the 3D model.

10. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that refine a position of the portion of the human hand based upon informing results of the analyzing with results of such analyzing as conducted on images obtained at a different time.

11. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that determine constraints on at least one of a number and position of one or more human hand portions according a human hand type.

12. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that resolve an at least partially occluded portion of the human hand by conducting spatial correlations across surfaces.

13. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that capture one or more additional image sets of two or more images, and resolve an at least partially occluded portion of the human hand by conducting temporal correlations across image sets.

14. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that resolve an at least partially occluded portion of the human hand by applying physical constraints of a human hand type.

15. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that refine a 3D model formed by the reconstructing.

16. The smart phone of claim 15, wherein the 3D model represents a generic hand and the refining of the 3D model comprises texture mapping details from image capture onto a generic hand model.

17. The smart phone of claim 15, wherein the fixed function logic circuit further implements actions that morph the 3D model according to detected human hand orientation and motion.

18. The smart phone of claim 15, wherein the refining of the 3D model includes refining the 3D model based at least in part upon one of continuity in motion and continuity in deformation.

19. The smart phone of claim 1, wherein the fixed function logic circuit further implements actions that (i) refine a 3D model formed by the reconstructing by comparing a reconstructed model of the portion of the human hand to human hand types in a library of human hand types and characteristic parameters for human hands in one or more possible poses, and (ii) assign matching human hand types to the 3D model when a match based on characteristic parameters is found.

20. The smart phone of claim 19, wherein a human hand type includes a hand, a pose includes fingers extended and palm facing the camera, and the characteristic parameters include five fingers.

21. The smart phone of claim 19, wherein the refining of the 3D model further comprises applying a constraint to the 3D model based on characteristics of a human hand type.

22. The smart phone of claim 21, wherein a human hand type includes a hand, and the constraint includes positions and angles of fingers relative to each other and to a palm portion of a hand.

23. A method of using a smart phone to identify a position and a shape of a human hand moving in a three-dimensional (3D) space, the method comprising:

configuring a camera of the smart phone to have a particular vantage point to view the human hand; and

executing instructions on a fixed function logic circuit of the smart phone to implement actions including;

capturing, with the camera from the particular vantage point, two or more images of a portion of a human hand moving in the 3D space;

analyzing the two or more images captured by the camera from the particular vantage point to computationally represent the portion of the human hand, as captured, as one or more mathematically represented 3D surfaces, each 3D surface corresponding to a cross-section of the portion of the human hand, based at least in part on a plurality of edge points of the portion of the human hand in the image, tangent lines extending from the camera to at least two edge points of the plurality of edge points, and a centerline corresponding to the tangent lines; and

reconstructing the position of, and the shape fitting, at least the portion of the human hand in the 3D space based at least in part on the plurality of edge points and the centerline.

24. A head mounted device, comprising:

an optical assembly to display a displayed environment including at least one of a surrounding three-dimensional (3D) environment and a virtual environment, the surrounding 3D environment including at least a portion of a human hand; and

a smart phone having an interface that identifies a position and a shape of a portion of the human hand moving in the surrounding 3D environment, the smart phone including;

a camera having a particular vantage point configured to view the human hand; and

a fixed function logic circuit storing instructions that, when executed, implement actions including;

capturing, with the camera from the particular vantage point, two or more images of a portion of a human hand moving in the surrounding 3D environment;

analyzing the two or more images captured by the camera from the particular vantage point to computationally represent the portion of the human hand, as captured, as one or more mathematically represented 3D surfaces, each 3D surface corresponding to a cross-section of the portion of the human hand, based at least in part on a plurality of edge points of the portion of the human hand in the image, tangent lines extending from the camera to at least two edge points of the plurality of edge points, and a centerline corresponding to the tangent lines; and

reconstructing the position of, and the shape fitting, at least the portion of the human hand in the surrounding 3D environment based at least in part on the plurality of edge points and the centerline.