Gesture based control using three-dimensional information extracted over an extended depth of field

• US 9,075,441 B2
• Filed: 04/02/2009
• Issued: 07/07/2015
• Est. Priority Date: 02/08/2006
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: St. Hilaire, Pierre, Underkoffler, John S.
• Primary Examiner(s): Haley, Joseph
• Assistant Examiner(s): ILUYOMADE, IFEDAYO B
• Application Number:
  US12/417,252
• Publication Number:
  US 20090231278A1
• Time in Patent Office: 2,287 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods are described for gesture-based control using three-dimensional information extracted over an extended depth of field. The system comprises a plurality of optical detectors coupled to at least one processor. The optical detectors image a body. At least two optical detectors of the plurality of optical detectors comprise wavefront coding cameras. The processor automatically detects a gesture of the body, wherein the gesture comprises an instantaneous state of the body. The detecting comprises aggregating gesture data of the gesture at an instant in time. The gesture data includes focus-resolved data of the body within a depth of field of the imaging system. The processor translates the gesture to a gesture signal, and uses the gesture signal to control a component coupled to the processor.

Claims

1. A system comprising:

a plurality of optical detectors, wherein at least two optical detectors of the plurality of optical detectors comprise wavefront coded cameras, wherein the plurality of optical detectors image a body; and

a processor coupled to the plurality of optical detectors, the processor automatically detecting a gesture of a body, wherein the gesture comprises an instantaneous state of the body, wherein the detecting comprises aggregating only gesture data of the gesture and precluding use of background data in the detecting of the gesture, the gesture data comprising absolute three-space focus-resolved location data of an instantaneous state of the body at a point in time and space, the processor translating the gesture to a gesture signal and using the gesture signal to control a component coupled to the processor.

2. The system of claim 1, wherein the wavefront coded cameras include a wavefront coding optical element.

3. The system of claim 1, wherein the imaging comprises generating wavefront coded images of the body.

4. The system of claim 1, wherein the wavefront coded cameras include a phase mask that increases a depth of focus of the imaging.

5. The system of claim 1, wherein the gesture data comprises focus-resolved range data of the body within the depth of field.

6. The system of claim 5, wherein the focus-resolved range data of the body within the depth of field is derived from an output of the wavefront coded cameras.

7. The system of claim 1, wherein the gesture data comprises focus-resolved position data of the body within the depth of field.

8. The system of claim 7, wherein the focus-resolved position data of the body within the depth of field is derived from an output of the wavefront coded cameras.

9. The system of claim 1, comprising modulation transfer functions and point spread functions that are invariant to a distance between the body and the imaging system.

10. The system of claim 1, comprising modulation transfer functions and point spread functions that are invariant with respect to defocus.

11. The system of claim 1, wherein the processor generates intermediate images by coding images gathered by the wavefront coded cameras.

12. The system of claim 11, wherein the intermediate images are blurred.

13. The system of claim 11, wherein the intermediate images are insensitive to changes in the body or the plurality of optical detectors that include defocus aberrations.

14. The system of claim 1, wherein the gesture data is three-space location data representing the gesture.

15. The system of claim 1, wherein the detecting includes at least one of detecting a location of the body, detecting an orientation of the body, and detecting includes detecting motion of the body.

16. The system of claim 1, wherein the detecting comprises identifying the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

17. The system of claim 1, wherein the detecting includes detecting at least one of a first set of appendages and a second set of appendages of the body.

18. The system of claim 1, wherein the detecting includes dynamically detecting a position of at least one tag.

19. The system of claim 18, wherein the detecting includes detecting position of a set of tags coupled to a part of the body.

20. The system of claim 19, wherein each tag of the set of tags includes a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

21. The system of claim 1, wherein the detecting includes dynamically detecting and locating a marker on the body.

22. The system of claim 21, wherein the detecting includes detecting position of a set of markers coupled to a part of the body.

23. The system of claim 21, wherein the set of markers form a plurality of patterns on the body.

24. The system of claim 21, wherein the detecting includes detecting position of a plurality of appendages of the body using a set of markers coupled to each of the appendages.

25. The system of claim 1, wherein the translating comprises translating information of the gesture to a gesture notation.

26. The system of claim 25, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

27. The system of claim 26, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

28. The system of claim 26, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

29. The system of claim 26, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

30. The system of claim 26, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

31. The system of claim 30, wherein the kinematic linkage is at least one first appendage of the body.

32. The system of claim 31, comprising assigning each position in the string to a second appendage, the second appendage connected to the first appendage.

33. The system of claim 32, comprising assigning characters of a plurality of characters to each of a plurality of positions of the second appendage.

34. The system of claim 33, wherein the plurality of positions is established relative to a coordinate origin.

35. The system of claim 34, comprising establishing the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

36. The system of claim 33, comprising assigning characters of the plurality of characters to each of a plurality of orientations of the first appendage.

37. The system of claim 31, wherein the detecting comprises detecting when an extrapolated position of the body intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the computer.

38. The system of claim 37, wherein controlling the component comprises controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

39. The system of claim 38, wherein controlling the component comprises controlling a position of the virtual object in the virtual space in response to the extrapolated position in the virtual space.

40. The system of claim 38, wherein controlling the component comprises controlling attitude of the virtual object in the virtual space in response to the gesture.

41. The system of claim 1, comprising controlling scaling of the detecting and controlling to generate coincidence between virtual space and physical space, wherein the virtual space comprises space depicted on a display device coupled to the processor, wherein the physical space comprises space inhabited by the body.

42. The system of claim 41, comprising controlling at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.

43. The system of claim 1, wherein the controlling includes at least one of controlling a function of an application hosted on the processor and controlling a component displayed on the processor.

44. A method comprising:

imaging a body with an imaging system, the imaging comprising generating wavefront coded images of the body;

automatically detecting a gesture of a body, wherein the gesture comprises an instantaneous state of the body, wherein the detecting comprises aggregating only gesture data of the gesture and precluding use of background data in the detecting of the gesture, the gesture data comprising absolute three-space focus-resolved location data of an instantaneous state of the body at a point in time and space;

translating the gesture to a gesture signal; and

controlling a component coupled to a computer in response to the gesture signal.

45. The method of claim 44, wherein the imaging system comprises a plurality of optical detectors, wherein at least two of the optical detectors are wavefront coded cameras comprising a wavefront coding optical element.

46. The method of claim 44, wherein the imaging comprises generating wavefront coded images of the body.

47. The method of claim 44, wherein the imaging system comprises a plurality of optical detectors, wherein at least two of the optical detectors are wavefront coded cameras comprising a phase mask that increases a depth of focus of the imaging.

48. The method of claim 44, wherein the gesture data comprises focus-resolved range data of the body within the depth of field.

49. The method of claim 48, wherein the focus-resolved range data of the body within the depth of field is derived from an output of the imaging system.

50. The method of claim 44, wherein the gesture data comprises focus-resolved position data of the body within the depth of field.

51. The method of claim 50, wherein the focus-resolved position data of the body within the depth of field is derived from an output of the imaging system.

52. The method of claim 44, comprising generating modulation transfer functions and point spread functions that are invariant to a distance between the body and the imaging system.

53. The method of claim 44, comprising generating modulation transfer functions and point spread functions that are invariant with respect to defocus.

54. The method of claim 44, comprising generating intermediate images by coding images gathered by the wavefront coded cameras.

55. The method of claim 54, wherein the intermediate images are blurred.

56. The method of claim 54, wherein the intermediate images are insensitive to changes in the body or a plurality of optical detectors of the imaging system that include defocus aberrations.

57. The method of claim 44, wherein the gesture data is three-space location data representing the gesture.

58. The method of claim 44, wherein the detecting includes detecting a location of the body.

59. The method of claim 44, wherein the detecting includes detecting an orientation of the body.

60. The method of claim 44, wherein the detecting includes detecting motion of the body.

61. The method of claim 44, wherein the detecting comprises identifying the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

62. The method of claim 44, wherein the detecting includes detecting at least one of a first set of appendages and a second set of appendages of the body.

63. The method of claim 44, wherein the detecting includes dynamically detecting a position of at least one tag.

64. The method of claim 63, wherein the detecting includes detecting position of a set of tags coupled to a part of the body.

65. The method of claim 64, wherein each tag of the set of tags includes a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

66. The method of claim 44, wherein the detecting includes dynamically detecting and locating a marker on the body.

67. The method of claim 66, wherein the detecting includes detecting position of a set of markers coupled to a part of the body.

68. The method of claim 66, wherein the set of markers form a plurality of patterns on the body.

69. The method of claim 66, wherein the detecting includes detecting position of a plurality of appendages of the body using a set of markers coupled to each of the appendages.

70. The method of claim 44, wherein the translating comprises translating information of the gesture to a gesture notation.

71. The method of claim 70, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

72. The method of claim 71, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

73. The method of claim 71, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

74. The method of claim 71, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

75. The method of claim 71, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

76. The method of claim 75, wherein the kinematic linkage is at least one first appendage of the body.

77. The method of claim 76, comprising assigning each position in the string to a second appendage, the second appendage connected to the first appendage.

78. The method of claim 77, comprising assigning characters of a plurality of characters to each of a plurality of positions of the second appendage.

79. The method of claim 78, wherein the plurality of positions is established relative to a coordinate origin.

80. The method of claim 79, comprising establishing the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

81. The method of claim 78, comprising assigning characters of the plurality of characters to each of a plurality of orientations of the first appendage.

82. The method of claim 76, wherein the detecting comprises detecting when an extrapolated position of the body intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the computer.

83. The method of claim 82, wherein controlling the component comprises controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

84. The method of claim 83, wherein controlling the component comprises controlling a position of the virtual object in the virtual space in response to the extrapolated position in the virtual space.

85. The method of claim 83, wherein controlling the component comprises controlling attitude of the virtual object in the virtual space in response to the gesture.

86. The method of claim 44, comprising controlling scaling of the detecting and controlling to generate coincidence between virtual space and physical space, wherein the virtual space comprises space depicted on a display device coupled to the processor, wherein the physical space comprises space inhabited by the body.

87. The method of claim 86, comprising translating scale, angle, depth, and dimension between the virtual space and the physical space as appropriate to at least one application coupled to the processor.

88. The method of claim 86, comprising controlling at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.

89. The method of claim 44, wherein the controlling includes controlling a function of an application hosted on the processor.

90. The method of claim 44, wherein the controlling includes controlling a component displayed on the processor.