Systems and methods for capturing motion in three-dimensional space

• US 9,153,028 B2
• Filed: 12/13/2013
• Issued: 10/06/2015
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US14/106,140
• Publication Number:
  US 20140139641A1
• Time in Patent Office: 662 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: G06V 40/11 : Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/117 : Biometrics derived from hands
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Methods and systems for capturing motion and/or determining the shapes and positions of one or more objects in 3D space utilize cross-sections thereof. In various embodiments, images of the cross-sections are captured using a camera based on edge points thereof.

Claims

1. A method of identifying a position and shape of at least a portion of an object (object portion) in three-dimensional (3D) space, the method comprising:

capturing, with a plurality of cameras, each camera having a particular vantage point, two or more images generated by casting an output from at least one source onto the object portion;

analyzing the two or more images captured by the cameras from the particular vantage points to computationally slice the object portion, as captured, into a plurality of two-dimensional (2D) slices, each slice corresponding to a cross-section of the object portion, based at least in part on a plurality of edge points of the object portion in the image; and

reconstructing the position and shape of at least a portion of the object portion in 3D space based at least in part on a plurality of identified cross-sectional positions and sizes.

2. The method of claim 1, wherein the vantage points are on the same side of the object.

3. The method of claim 1, wherein the cameras are arranged such that the vantage points lie in a single line.

4. The method of claim 1, wherein the fields of view of the cameras overlap throughout an area where motion of interest is expected to occur.

5. The method of claim 1, wherein optical axes of the cameras are parallel.

6. The method of claim 1, wherein the two or more images comprise images captured as the object moves and articulates.

7. The method of claim 1, wherein the two or more images are captured such that a time between images is sufficiently small so as to avoid quality degradation due to object movement.

8. The method of claim 1, wherein the capturing comprises controlling camera settings of the cameras.

9. The method of claim 8, wherein the camera settings are selected from a group including resolution, depth of field, sensitivity, light frequency and frame rate.

10. The method of claim 8, wherein the controlling camera settings is conducted according to object parameters assumed to correspond to at least a portion of a type of object.

11. The method of claim 1, wherein the at least one source comprises one or more light emitting diodes.

12. The method of claim 1, wherein the at least one source comprises one or more light sources and one or more of the light sources are placed on either side of a camera.

13. The method of claim 1, wherein the at least one source comprises one or more light sources and the one or more light sources are placed to create a contrast between at least a portion of the object and a background.

14. The method of claim 1, wherein the at least one source comprises one or more light sources and the one or more light sources are placed to illuminate a region on the at least a portion of the object.

15. The method of claim 1, wherein the at least one source comprises one or more light sources and at least a subset of the light sources is operable to emit at different wavelengths.

16. The method of claim 1, wherein the at least one source comprises one or more light sources, at least a subset of the light sources is operable in sequence or simultaneously and one or more of the light sources is operable in a pulsed fashion.

17. The method of claim 1, wherein the at least one source comprises one or more light sources and at least a subset of the light sources is operable at different brightnesses.

18. The method of claim 1, wherein the at least one source comprises one or more light sources selected from a group including point light sources, arrays and bar lights.

19. The method of claim 1, wherein the analyzing comprises applying a pixel threshold to determine portions of the image as corresponding to at least a portion of the object or background.

20. The method of claim 1, wherein the edge points define a viewed portion of the cross-section such that the portion of the cross-section is within a field of view of at least one of the cameras from at least one of the particular vantage points.

21. The method of claim 20, wherein the computationally slicing comprises fitting a closed curve to the viewable portion of the object portion, the closed curve defining a 3D model of the object portion.

22. The method of claim 21, wherein the closed curve is an ellipse.

23. The method of claim 21, wherein the fitting a closed curve comprises:

determining a family of closed curves corresponding to the viewable portion of the object portion;

selecting a closed curve from the family of closed curves; and

selecting a rotation angle for the closed curve that corresponds with a determined contour of the object portion.

24. The method of claim 21, wherein the computationally slicing further comprises approximating a contour of the portion of the cross-section within the field of view of at least one of the cameras.

25. The method of claim 1, wherein light rays cast from the edge points to the image-capturing device are tangent to the cross-section.

26. The method of claim 1, wherein the at least one of the edge points is defined by a boundary between a shadow region and an illuminated region on the cross-section of the object.

27. The method of claim 1, wherein the reconstructing comprises assembling the 2D slices to form a 3D model.

28. The method of claim 27, further comprising using correlation to refine the 3D model.

29. The method of claim 28, wherein the correlation comprises at least one of:

correlating among the 2D slices; and

correlating a 3D model resulting from stacking the 2D slices to preserve continuity with characteristic information of a corresponding object.

30. The method of claim 29, wherein the characteristic information includes at least one of continuity in motion and continuity in deformation.

31. The method of claim 1, wherein:

the casting an output comprises casting an output that causes an illuminated part of the object and a shadow region; and

the analyzing comprises detecting a shadow edge point of the shadow region and determining characteristic parameters of a closed curve corresponding to the shadow edge point.

32. The method of claim 1, wherein the analyzing comprises determining closed curve characteristics as corresponding to an object type that corresponds to the object portion of the object.

33. The method of claim 32, wherein the determining closed curve characteristics is conducted by comparing the object to object types in a library including object types.

34. The method of claim 1, further comprising determining characteristics of an object model determined from the reconstructing;

and, according to the characteristics of the object model, at least one of;

revising characteristic parameters in an object library and

defining characteristic parameters in an object library.

35. The method of claim 32, wherein the closed curve characteristics are selected from a group including closed curve shapes, 3D closed curves and closed curve rotation angle.

36. The method of claim 32, further comprising modeling at least a portion of the object with a 3D volume.

37. The method of claim 36, wherein the 3D volume is selected from a group including ellipsoids, hyperboloids and cylinders.

38. The method of claim 36, wherein the modeling at least a portion of the object comprises modeling at least one of a fingertip, a thumb and one or more finger segments with an ellipsoid.

39. The method of claim 1, wherein the object portion is moving in 3D space;

and the two or more images capture a motion of the object portion.

40. The method of claim 39, further comprising interpreting the captured motion as user input.

41. The method of claim 40, wherein the motion of the object portion comprises one or more hand gestures.

42. The method of claim 1, further comprising:

comparing a reconstructed position and shape of the object portion with characteristic parameters for the object to determine one or more object poses; and

interpreting the one or more object poses as user input.

43. The method of claim 1, further comprising:

determining that a reconstructed shape of the object portion matches an object model in an object library; and

interpreting the reconstructed shape as user input.

44. The method of claim 1, wherein the object comprises a hand and the at least a portion of the object comprises at least a portion of a finger.

45. The method of claim 1, wherein the cameras have overlapping fields of view.

46. The method of claim 1, wherein the cameras are directed in an upward direction.

47. A system for identifying a position and shape of at least a portion of an object (object portion) in three-dimensional (3D) space, the system comprising:

a plurality of cameras, each camera having a particular vantage point, and oriented toward a field of view;

at least one source to direct illumination onto the object portion in the field of view; and

an image analyzer coupled to the cameras and the at least one source and configured to;

capture two or more images generated by casting an output from at least one source onto the object;

analyze the two or more images captured by the cameras from the particular vantage points to computationally slice the object portion, as captured, into a plurality of two-dimensional (2D) slices, each slice corresponding to a cross-section of the object portion, based at least in part on a plurality of edge points in the image; and

reconstruct the position and shape of at least a portion of the object portion in 3D space based at least in part on a plurality of the identified cross-sectional positions and sizes.

48. The system of claim 47, wherein the cameras are arranged such that the vantage points lie in a single line.