Systems and methods of object shape and position determination in three-dimensional (3D) space

• US 9,741,136 B2
• Filed: 12/21/2016
• Issued: 08/22/2017
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US15/387,353
• Publication Number:
  US 20170103545A1
• Time in Patent Office: 244 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: G06V 40/11 : Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/117 : Biometrics derived from hands
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Methods and systems for capturing motion and/or determining the shapes and positions of one or more objects in 3D space utilize cross-sections thereof. In various embodiments, images of the cross-sections are captured using a camera based on edge points thereof.

Claims

1. A system of identifying a position and a shape of a portion of an object in a three-dimensional (3D) space, the system comprising:

one or more processors coupled to a memory, the memory loaded with computer instructions that, when executed on the one or more processors, implement a method including;

capturing, with a single camera having a particular vantage point, two or more images generated by casting an output from at least one source onto the portion of the object;

analyzing the two or more images captured by the camera from the particular vantage point to computationally represent the portion of the object, as captured, as one or more mathematically represented 3D surfaces, each 3D surface corresponding to a cross-section of the portion of the object, based at least in part on a plurality of edge points of the portion of the object in the image, tangent lines extending from the camera to at least two edge points of the plurality of edge points and a centerline corresponding to the tangent lines; and

reconstructing the position of and the shape fitting at least the portion of the object in the 3D space based at least in part on the plurality of edge points and the centerline.

2. The system of claim 1, wherein at least a portion of at least one of the 3D surfaces is determined from a correlation of slices that represent all or part of the object.

3. The system of claim 2, wherein at least one slice of the slices is tilted with respect to one or more other slices of the slices.

4. The system of claim 2, wherein the correlating comprises defining a 3D surface that best fits the slices.

5. The system of claim 1, wherein:

the analyzing comprises determining the centerline by identifying diagonal line segments that connect opposite corners of an intersection region formed from an intersection of two pairs of intersecting tangent lines selected from the tangent lines, identifying midpoints of the diagonal line segments, and identifying a line segment joining the midpoints as the centerline; and

the reconstructing comprises fitting at least one of the 3D surfaces to a point at which at least one of the tangent lines is substantially tangent to the object, as represented, and positioned about the centerline.

6. The system of claim 5, wherein the reconstructing further comprises stacking two or more of the 3D surfaces.

7. The system of claim 1, wherein at least one of the 3D surfaces approximates a contour of the cross-section of the portion of the object.

8. The system of claim 1, wherein at least one of the 3D surfaces is symmetrical in at least one plane.

9. The system of claim 8, wherein at least one of the 3D surfaces is selected from a group consisting of ellipsoids, hyperboloids, cylinders and other quadratic surfaces.

10. The system of claim 1, wherein the analyzing comprises revising characteristic parameters of at least one of the 3D surfaces based upon results of the analyzing as conducted on images obtained at a different time.

11. The system of claim 1, wherein:

at least one of the 3D surfaces is defined according to characteristic parameters defining characteristics of the at least one of the 3D surfaces; and

the analyzing comprises assuming a characteristic parameter of the at least one of the 3D surfaces.

12. The system of claim 11, wherein the assuming is conducted according to a type of the object.

13. The system of claim 1, wherein:

at least one of the 3D surfaces is defined according to characteristic parameters defining characteristics of the at least one of the 3D surfaces; and

the analyzing comprises assuming a typical characteristic parameter of the at least one of the 3D surfaces.

14. The system of claim 1, wherein the object comprises multiple members.

15. The system of claim 14, wherein one or more of the multiple members is separately articulating.

16. The system of claim 15, wherein spatial relationships between multiple members change as a subject comprising the multiple members moves.

17. The system of claim 1, wherein the portion of the object is selected from a group consisting essentially of fingers, finger segments, thumb segments, hands, palms, arms, legs, a core of a person and a trunk of a person.

18. The system of claim 1, wherein the analyzing comprises analyzing additional images and determining an object motion of the object.

19. The system of claim 18, wherein the object motion determines a pose.

20. The system of claim 18, wherein the determining of the object motion of the object comprises determining a movement of digits indicated by a sequence of models captured across time.

21. The system of claim 20, wherein the object is any type of object.

22. The system of claim 20,

wherein the object is a hand, and

wherein motion of the hand as captured is used to control equipment.

23. The system of claim 1, wherein the capturing comprises capturing images of a viewed portion of the object.

24. The system of claim 1, wherein the method further comprises analyzing one or more of the 3D surfaces to produce a 3D model.

25. The system of claim 1, wherein the method further comprises refining a 3D model by correlating a cross-section of an object portion represented by a first surface with a cross-section of at least one different object portion represented by at least one different 3D surface.

26. The system of claim 1, wherein the method further comprises refining a 3D model by informing results of the analyzing with results of such analyzing as conducted on images obtained at a different time and correspondingly modifying the 3D model.

27. The system of claim 1, wherein the method further comprises refining a position of an object portion based upon informing results of the analyzing with results of such analyzing as conducted on images obtained at a different time.

28. The system of claim 1, wherein the method further comprises determining constraints on at least one of a number and position of one or more object portions according an object type.

29. The system of claim 1, wherein the method further comprises resolving an at least partially occluded object portion by conducting spatial correlations across surfaces.

30. The system of claim 1, wherein the method further comprises:

capturing one or more additional image sets of two or more images; and

resolving an at least partially occluded object portion by conducting temporal correlations across image sets.

31. The system of claim 1, wherein the method further comprises resolving an at least partially occluded object portion by applying physical constraints of an object type.

32. The system of claim 1, wherein the method further comprises refining a 3D model formed by the reconstructing.

33. The system of claim 32, wherein the 3D model represents a generic hand, the refining further comprising texture mapping details from image capture onto a generic hand model.

34. The system of claim 32, wherein the method further comprises morphing the 3D model according to detected object orientation and motion.

35. The system of claim 32, wherein the refining includes refining the 3D model based at least in part upon one of continuity in motion and continuity in deformation.

36. The system of claim 1, wherein the method further comprises refining a 3D model formed by the reconstructing by comparing a reconstructed model of the portion of the object to object types in a library of object types and characteristic parameters for objects in one or more possible poses;

and assigning matching object types to the 3D model when a match based on characteristic parameters is found.

37. The system of claim 36, wherein the object type includes a hand, the pose includes fingers extended and palm facing the camera, and the characteristic parameters include five fingers.

38. The system of claim 36, wherein the refining of the 3D model further comprises applying a constraint to the 3D model based on characteristics of the object type.

39. The system of claim 38, wherein the object type includes a hand, and the constraint includes positions and angles of fingers relative to each other and to a palm portion of a hand.

40. A non-transitory computer readable medium storing a plurality of instructions for programming one or more processors to identify a position and a shape of a portion of an object in a three-dimensional (3D) space, the instructions, when executed on the one or more processors, implementing a method including:

capturing, with a single camera having a particular vantage point, two or more images generated by casting an output from at least one source onto the portion of the object;

analyzing the two or more images captured by the camera from the particular vantage point to computationally represent the portion of the object, as captured, as one or more mathematically represented 3D surfaces, each 3D surface corresponding to a cross-section of the portion of the object, based at least in part on a plurality of edge points of the portion of the object in the image, tangent lines extending from the camera to at least two edge points of the plurality of edge points and a centerline corresponding to the tangent lines; and

reconstructing the position of and the shape fitting at least the portion of the object in the 3D space based at least in part on the plurality of edge points and the centerline.