Gestural control of autonomous and semi-autonomous systems

• US 9,910,497 B2
• Filed: 09/10/2009
• Issued: 03/06/2018
• Est. Priority Date: 02/08/2006
• Status: Expired due to Fees

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, White, Tom, Kramer, Mattie Ruth
• Primary Examiner(s): LEE, NICHOLAS J
• Application Number:
  US12/557,464
• Publication Number:
  US 20100066676A1
• Time in Patent Office: 3,099 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods are described for controlling a remote system. The controlling involves detecting a gesture of a body from gesture data received via a detector. The gesture comprises an instantaneous state of the body. The gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and physical space. The detecting aggregates only the gesture data of the gesture at an instant in time and identifies the gesture using the gesture data. The controlling then translates the gesture to a gesture signal, and controls a component of the remote system in response to the gesture signal.

Claims

1. A method for controlling a remote system, the method comprising:

detecting a first selection gesture of a body from first gesture data received via a detector, the first selection gesture indicating selection of a remote system located in a physical space that includes the body, wherein the first gesture data is absolute three-space location data of an instantaneous state of the body at a first point in time and physical space,

selecting the remote system for control in response to the detection of the first selection gesture;

detecting a first control gesture of the body from second gesture data received via the detector, wherein the first control gesture comprises an instantaneous state of the body, wherein the second gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and physical space of the body, the detecting comprising aggregating only the second gesture data of the first control gesture at an instant in time and identifying the first control gesture using only the second gesture data and excluding use of reference data and background;

translating the first control gesture to a first control gesture signal; and

controlling a component of the selected remote system in response to the first control gesture signal,

wherein the remote system is one of a robot and a vehicle.

2. The method of claim 1, comprising:

detecting a first pose of the body;

recording a first position at which the first pose is entered, wherein the first position is a three-space hand position;

setting the first position as a neutral point.

3. The method of claim 1, comprising forming a gestural interaction space comprising an active zone and a dead zone, wherein the dead zone is formed around a neutral point such that the neutral point is surrounded spatially and angularly by the dead zone.

4. The method of claim 3, wherein the active zone is adjacent the dead zone.

5. The method of claim 3, wherein movement of the body within the dead zone has no effect.

6. The method of claim 3, wherein the controlling of the component in response to the first control gesture signal is activated in response to the first control gesture when the first control gesture is detected in the active zone.

7. The method of claim 3, comprising providing a feedback indicator that indicates feedback of the controlling.

8. The method of claim 7, wherein the feedback indicator provides feedback indicating the body is in one of the active zone and the dead zone.

9. The method of claim 7, wherein the feedback indicator comprises at least one of a visual indicator and an aural indicator.

10. The method of claim 3, wherein the component comprises at least one of a propulsion component, a navigation component, and an orientation component of the remote system.

11. The method of claim 10, wherein the detecting of the first control gesture comprises:

detecting a gesture that is a movement of the body relative to a body axis and beyond a boundary of the dead zone;

effecting movement of the remote system relative to a remote system axis, wherein the remote system is capable of movement relative to the remote system axis, wherein the movement of the remote system is analogous to the gesture, wherein the remote system axis is analogous to the body axis.

12. The method of claim 11, wherein the movement of the remote system is translational movement in a first direction along the remote system axis.

13. The method of claim 11, wherein the movement of the remote system is translational movement in a second direction along the remote system axis.

14. The method of claim 11, wherein the movement of the remote system is rotational movement around the remote system axis.

15. The method of claim 11, comprising mapping the first control gesture to the component.

16. The method of claim 15, wherein the component is a control system of the remote system.

17. The method of claim 15,

wherein the control system is a steering system of the remote system,

wherein the method further comprises controlling movement of the steering system around a steering system axis in response to the mapping, and

wherein the steering system axis is analogous to the body axis, wherein the remote system axis is an axis about which the remote system is stationary.

18. The method of claim 10, wherein the detecting of the first control gesture comprises detecting a gesture that is a movement of the body in a first direction along a body axis and beyond a boundary of the dead zone, wherein the body axis is formed with reference to the body.

19. The method of claim 18, wherein, in response to the movement of the body in the first direction, the controlling comprises instructing movement of the remote system in the first direction along a remote system axis, wherein the remote system axis is formed with reference to the remote system, wherein the remote system axis is analogous to the body axis.

20. The method of claim 19, wherein the movement of the remote system in the first direction along the remote system axis is at a speed proportional to a distance of the body from the neutral point.

21. The method of claim 20, wherein the detecting of the first control gesture comprises detecting a gesture that is a movement of the body in a second direction along the body axis and beyond a boundary of the dead zone.

22. The method of claim 21, wherein, in response to the movement of the body in the second direction, the controlling comprises instructing movement of the remote system in the second direction along the remote system axis.

23. The method of claim 22, wherein the movement of the remote system in the second direction along the remote system axis is at a speed proportional to a distance of the body from the neutral point.

24. The method of claim 23, wherein the body axis lies in a horizontal plane relative to the body.

25. The method of claim 24, wherein the first direction is forward and the second direction is rearward.

26. The method of claim 24, wherein the first direction is left and the second direction is right.

27. The method of claim 23, wherein the body axis lies in a vertical plane relative to the body, wherein the first direction is up and the second direction is down.

28. The method of claim 10, wherein the detecting of the first control gesture comprises detecting a gesture that is a rotation of the body in a first direction around a body axis.

29. The method of claim 28, wherein, in response to the rotation of the body in the first direction, the controlling comprises instructing rotation of the remote system in the first direction around a remote system axis, wherein the remote system axis is analogous to the body axis.

30. The method of claim 29, wherein the detecting of the first control gesture comprises detecting a gesture that is a rotation of the body in a second direction around the body axis.

31. The method of claim 30, wherein, in response to the rotation of the body in the second direction, the controlling comprises instructing rotation of the remote system in the second direction around the remote system axis.

32. The method of claim 28, wherein the remote system axis is an axis about which the remote system moves.

33. The method of claim 1, wherein the component comprises an operational mode controller of the remote system, wherein the operational mode controller comprises at least one operational mode.

34. The method of claim 33, wherein the controlling comprises selecting an operational mode.

35. The method of claim 33, wherein the controlling comprises activating an operational mode.

36. The method of claim 33, wherein the controlling comprises deactivating an operational mode.

37. The method of claim 33, wherein the controlling comprises resetting an operational mode.

38. The method of claim 33, wherein the controlling comprises parameterizing an operational mode.

39. The method of claim 1, wherein the controlling comprises selecting the component from among a plurality of components.

40. The method of claim 39, comprising selecting an additional component from among a plurality of components.

41. The method of claim 40, wherein the controlling comprises forming a group to include the component and the additional component.

42. The method of claim 1, comprising controlling scaling of the detecting and controlling to generate coincidence between the physical space and component space inhabited by the component.

43. The method of claim 42, comprising a mapping between a physical distance of displacement of the body during a gesture and a physical dimension between successive points in the component space, wherein the mapping controls translation of the component along a dimension of the component space.

44. The method of claim 42, comprising a mapping between a physical distance of displacement of the body during a gesture and a virtual dimension of the component space, wherein the mapping controls a virtual aspect of the component in the virtual dimension of the component space.

45. The method of claim 1, comprising aligning a control axis of the component with a dimension of the physical space.

46. The method of claim 45, wherein the dimension is a depth dimension.

47. The method of claim 45, wherein the dimension is a horizontal dimension.

48. The method of claim 45, wherein the dimension is a vertical dimension.

49. The method of claim 45, wherein the dimension is a lateral dimension.

50. The method of claim 45, wherein the navigating comprises motion along the dimension to effect a translation along the control axis.

51. The method of claim 1, wherein the detector comprises a plurality of optical sensors.

52. The method of claim 1, comprising positioning the detector on the remote system.

53. The method of claim 1, comprising positioning the detector on the body.

54. The method of claim 1, comprising positioning the detector in the physical space.

55. The method of claim 1, wherein the second gesture data comprises a three-dimensional data point cloud representative of the body, and wherein the first gesture data comprises a three-dimensional data point cloud representative of the body.

56. The method of claim 55, wherein a resolved depth accuracy of data points in the three-dimensional data point cloud is approximately 1 millimeter (mm).

57. The method of claim 1, wherein the detecting of the first control gesture includes detecting an evolving position of the body.

58. The method of claim 57, wherein the body is a hand of an operator.

59. The method of claim 58, wherein the position comprises a position of the hand.

60. The method of claim 58, wherein the position comprises a position of a set of fingers of the hand.

61. The method of claim 1, wherein the detecting of the first control gesture comprises extracting positions of the body.

62. The method of claim 61, wherein the detecting of the first control gesture comprises recovering quantized poses and orientation vectors of the body.

63. The method of claim 1, wherein the detecting of the first control gesture includes detecting an evolving orientation of the body.

64. The method of claim 1, wherein the detecting of the first control gesture includes detecting an evolving pose of the body, wherein the pose is a geometric disposition of a part of the body relative to at least one other part of the body.

65. The method of claim 1, wherein the detecting of the first control gesture includes detecting evolving motion of the body.

66. The method of claim 1, comprising analyzing the first control gesture into a sequence of gestural events.

67. The method of claim 1, wherein the detecting of the first control gesture includes detecting at least one of a first set of appendages and a second set of appendages of the body.

68. The method of claim 1, wherein the detecting of the first control gesture includes dynamically detecting a position of at least one tag coupled to the body.

69. The method of claim 68, wherein the detecting of the first control gesture includes detecting position of a set of tags coupled to the body.

70. The method of claim 69, wherein each tag of the set of tags includes a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

71. The method of claim 1, wherein the detecting of the first control gesture includes dynamically detecting and locating a marker on the body.

72. The method of claim 71, wherein the detecting of the first control gesture includes detecting position of a set of markers coupled to the body.

73. The method of claim 71, wherein the set of markers form a plurality of patterns on the body.

74. The method of claim 71, wherein the detecting of the first control gesture includes detecting position of a plurality of appendages of the body using a set of markers coupled to each of the appendages.

75. The method of claim 1, wherein the translating comprises translating information of the second gesture data to a gesture notation.

76. The method of claim 75, wherein the gesture notation represents a gesture vocabulary, and the first control gesture signal comprises communications of the gesture vocabulary.

77. The method of claim 76, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

78. The method of claim 76, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

79. The method of claim 76, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

80. The method of claim 76, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

81. The method of claim 80, wherein the kinematic linkage is at least one first appendage of the body.

82. The method of claim 81, comprising assigning each position in the string to a second appendage, the second appendage connected to the first appendage.

83. The method of claim 82, comprising assigning characters of a plurality of characters to each of a plurality of positions of the second appendage.

84. The method of claim 83, wherein the plurality of positions is established relative to a coordinate origin.

85. The method of claim 84, comprising establishing the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

86. The method of claim 83, comprising assigning characters of the plurality of characters to each of a plurality of orientations of the first appendage.

87. The method of claim 1, comprising imaging the body with the detector, wherein the second gesture data is generated from the imaging, and wherein the first gesture data is generated from the imaging.

88. The method of claim 87, wherein the imaging comprises generating wavefront coded images of the body.

89. The method of claim 88, wherein the second gesture data comprises focus-resolved data of the body within a depth of field of the imaging system.

90. The method of claim 89, comprising generating intermediate images by coding images gathered by the imaging system.

91. The method of claim 90, wherein the intermediate images are blurred.

92. The method of claim 90, wherein the intermediate images are insensitive to changes in at least one of the body and the detector that include defocus aberrations.

93. The method of claim 89, wherein the second gesture data comprises focus-resolved range data of the body within the depth of field.

94. The method of claim 93, wherein the focus-resolved range data of the body within the depth of field is derived from an output of the detector.

95. The method of claim 89, wherein the second gesture data comprises focus-resolved position data of the body within the depth of field.

96. The method of claim 95, wherein the focus-resolved position data of the body within the depth of field is derived from an output of the detector.

97. The method of claim 87, wherein the detector comprises a plurality of cameras.

98. The method of claim 97, wherein at least two of the cameras are wavefront coded cameras comprising a wavefront coding optical element.

99. The method of claim 97, wherein at least two of the cameras are wavefront coded cameras comprising a phase mask that increases a depth of focus of the imaging.

100. The method of claim 87, comprising generating modulation transfer functions and point spread functions that are invariant to a distance between the body and the detector.

101. The method of claim 87, comprising generating modulation transfer functions and point spread functions that are invariant with respect to defocus.

102. The method of claim 1, wherein the remote system is a vehicle.

103. The method of claim 1, wherein the remote system is a robot.

104. The method of claim 1, further comprising:

detecting a second selection gesture of the body from third gesture data received via the detector, the second selection gesture indicating selection of a second remote system located in the physical space that includes the body, wherein the third gesture data is absolute three-space location data of an instantaneous state of the body at a first point in time and physical space,

selecting the second remote system for control in response to the detection of the second selection gesture;

detecting a second control gesture of the body from fourth gesture data received via the detector;

translating the second control gesture to a second control gesture signal; and

controlling a component of the selected second remote system in response to the second control gesture signal,

wherein the second remote system is one of a robot and a vehicle, and

wherein the second remote system is remote from the remote system that is controlled by the first control gesture.

105. The method of claim 104,

wherein the controlling of the remote system comprises simultaneously using a plurality of gesture modes,

wherein controlling a component of the remote system in response to the gesture signal comprises:

controlling movement of the remote system and controlling at least one operational mode of the remote system,

wherein the detecting comprises detecting a pose of the body and detecting at least one of;

an evolving position of the body, an evolving orientation of the body, and an evolving motion of the body,

wherein the controlling of the remote system comprises controlling of the remote system based on the pose of the body and at least one of;

the evolving position of the body, the evolving orientation of the body, and the evolving motion of the body,

wherein the pose includes a geometric disposition of a part of a hand of the body relative to at least one other part of the hand of the body.

106. A system comprising:

a detector for receiving gesture data that represents a gesture made by a body; and

a processor coupled to the detector,

wherein the processor is constructed to;

detect a first selection gesture of a body from first gesture data received via the detector, the first selection gesture indicating selection of a remote system located in a physical space that includes the body, wherein the first gesture data is absolute three-space location data of an instantaneous state of the body at a first point in time and physical space,

select the remote system for control in response to the detection of the first selection gesture;

detect a control gesture of the body from second gesture data received via the detector, wherein the control gesture comprises an instantaneous state of the body, wherein the second gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and physical space of the body, the detecting comprising aggregating only the second gesture data of the control gesture at an instant in time and identifying the control gesture using only the second gesture data and excluding use of reference data and background;

translate the control gesture to a control gesture signal; and

control a component of the selected remote system in response to the control gesture signal,

wherein the remote system is one of a robot and a vehicle.