Gesture-Based Control System For Vehicle Interfaces

• US 20090278915A1
• Filed: 06/18/2009
• Published: 11/12/2009
• Est. Priority Date: 02/08/2006
• Status: Abandoned Application

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Application Number:
  US12/487,623
• Publication Number:
  US 20090278915A1
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/595 Data transfer involving internal databases
• CPC Class Codes: B60K 2370/146 Input by gesture
• CPC Class Codes: B60K 2370/1464 3D-gesture
• CPC Class Codes: B60K 2370/595 Internal database involved
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60K 37/06 of controls, e.g. controls knobs
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and method for controlling vehicle subsystems using gestures are described. The controlling comprises receiving an image of an object via sensors in a vehicle and outputting gesture data representing a gesture made by the object. The object comprises at least one hand and/or finger of a user. The gesture data is an instantaneous state of the object at a point in time in space. The controlling comprises automatically detecting the gesture of the object from the gesture data by extracting positions of the object, recovering quantized poses and orientation vectors of the object, and translating the gesture data to a gesture signal. The controlling comprises managing user interactions with the vehicle subsystems in response to the gesture signal.

Claims

1. A system comprising:

a plurality of sensors in a vehicle, wherein the plurality of sensors image an object and output gesture data representing a gesture made by the object, wherein the gesture data is an instantaneous state of the object at a point in time in space, wherein the object comprises at least one of a hand and a finger of a user;

a processor coupled to the plurality of sensors, the processor automatically detecting the gesture of the object from the gesture data, the detecting comprising extracting positions of the object, the detecting comprising recovering quantized poses and orientation vectors of the object, the detecting comprising translating the gesture data to a gesture signal; and

a controller coupled to the processor and to a plurality of vehicle subsystems of the vehicle, the controller managing user interactions with the plurality of vehicle subsystems in response to the gesture signal.

2. The system of claim 1, wherein the plurality of sensors are positioned to view a passenger compartment of the vehicle.

3. The system of claim 1, wherein the plurality of sensors comprise a plurality of cameras.

4. The system of claim 3, wherein the plurality of cameras comprise a stereo-vision system.

5. The system of claim 1, wherein the gesture data comprises a three-dimensional data point cloud representative of the object.

6. The system of claim 5, wherein a resolved depth accuracy of data points in the three-dimensional data point cloud is approximately 1 millimeter (mm).

7. The system of claim 1, wherein the positions of the object comprise position of a hand.

8. The system of claim 1, wherein the positions of the object comprise position of a set of fingers.

9. The system of claim 1, wherein the quantized poses of the object comprise quantized poses of at least one hand.

10. The system of claim 1, wherein the orientation vectors of the object comprise continuous orientation vectors of at least one hand.

11. The system of claim 1, wherein the orientation vectors of the object comprise continuous orientation vectors of a set of fingers.

12. The system of claim 1, wherein the translating comprises translating information of the gesture to a gesture notation.

13. The system of claim 12, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

14. The system of claim 13, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the object.

15. The system of claim 13, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the object.

16. The system of claim 13, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the object.

17. The system of claim 13, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the object.

18. The system of claim 1, comprising a feedback device coupled to the processor, the feedback device providing to the user indications corresponding to the gesture.

19. The system of claim 18, wherein the indications comprise an interpretation of the gesture.

20. The system of claim 18, wherein the indications comprise an effect realized on the plurality of vehicle subsystems in response to the gesture.

21. The system of claim 18, wherein the feedback device is a visual device.

22. The system of claim 18, wherein the feedback device is an aural device.

23. The system of claim 18, wherein the feedback device is an audio-visual device.

24. The system of claim 18, wherein the feedback device is a head-up display of the vehicle.

25. The system of claim 18, wherein the feedback device is a pixel-based display of the vehicle.

26. The system of claim 18, wherein the feedback device is at least one color-changing lighted indicator.

27. The system of claim 18, wherein the feedback device is a tactile display.

28. The system of claim 1, wherein the controller manages the user interactions using a first interactive mode, the first interactive mode comprising controlling the interactions with the plurality of vehicle subsystems in response to the gesture, wherein the gesture comprises finger motions of the user.

29. The system of claim 28, wherein the gesture comprises a grip-and-extend-again motion of two fingers of a hand, wherein the controller cycles through a plurality of control elements of the vehicle subsystems in response to the gesture, wherein each control element allows the user to control a parameter of a vehicle subsystem.

30. The system of claim 29, wherein the gesture comprises a grip-and-extend-again motion of a finger of a hand of the user, wherein the controller incrementally changes a setting of the parameter in response to the gesture.

31. The system of claim 29, wherein the gesture comprises holding a first finger down and extending a second finger, wherein the controller changes a setting of the parameter at a constant rate in response to the gesture.

32. The system of claim 28, wherein the gesture comprises extension of at least one finger of a first hand of the user, wherein the processor activates the controller in response to the gesture.

33. The system of claim 32, wherein hands of the user remain in contact with a guidance mechanism of the vehicle during the gesture.

34. The system of claim 32, wherein the at least one finger is at least one of an index finger, a middle finger, and a thumb of the user.

35. The system of claim 32, wherein the gesture comprises a grip-and-extend-again motion of two fingers of the first hand, wherein the controller cycles through a plurality of control modes of the vehicle subsystems in response to the gesture, wherein each control mode of the plurality of control modes corresponds to a vehicle subsystem.

36. The system of claim 35, wherein the gesture comprises a grip-and-extend-again motion of two fingers of a second hand, wherein the controller cycles through a plurality of control options of each control mode in response to the gesture, wherein each control option of the plurality of control options corresponds to a vehicle subsystem.

37. The system of claim 36, wherein the gesture comprises a grip-and-extend-again motion of a finger of a hand of the user, wherein the controller controls a setting of a control in response to the gesture.

38. The system of claim 37, wherein the gesture comprises a grip-and-extend-again motion of a first finger of a hand of the user, wherein the controller incrementally changes the setting by stepping the setting in a first direction in response to the gesture.

39. The system of claim 38, wherein the gesture comprises a grip-and-extend-again motion of a second finger of a hand of the user, wherein the controller incrementally changes the setting by stepping the setting in a second direction in response to the gesture.

40. The system of claim 39, wherein the gesture comprises holding the first finger down and extending the second finger, wherein the controller changes the setting at a constant rate in response to the gesture.

41. The system of claim 28, comprising a graphical display coupled to the processor, the graphical display providing to the user indications corresponding to the gesture, wherein the gesture controls position of a display element on the graphical display.

42. The system of claim 41, wherein the finger motions effect non-linear selection of a control element of the vehicle subsystems via the display element, wherein each control element allows the user to control a parameter of a vehicle subsystem.

43. The system of claim 42, wherein the gesture comprises extension of at least one finger of a first hand of the user, wherein the processor activates the controller and causes the display element to be displayed in response to the gesture.

44. The system of claim 43, wherein the gesture comprises relative movement of a finger tip, wherein the processor controls movement of the display element on the graphical display in response to the relative movement of the finger tip.

45. The system of claim 44, wherein interface graphics of the display are selected by proximity of the display element, wherein the display element is a pointer.

46. The system of claim 45, wherein the interface graphics correspond to control elements of the vehicle subsystems, wherein an interface graphic of each control element allows the user to control a parameter of a vehicle subsystem.

47. The system of claim 46, wherein the gesture comprises a grip-and-extend-again motion of a finger of a hand of the user, wherein the controller changes a setting of the parameter via the interface graphic in response to the gesture.

48. The system of claim 46, wherein the gesture comprises holding a first finger down and extending a second finger, wherein the controller changes a setting of the parameter via the interface graphic in response to the gesture.

49. The system of claim 41, wherein the display element comprises a toggle switch icon, wherein the gesture comprises a grip-and-extend-again motion of a finger of a hand of the user, wherein the gesture controls a setting of the toggle switch.

50. The system of claim 41, wherein the display element comprises a slider icon, wherein the gesture comprises holding a first finger down and extending a second finger, wherein the controller changes a setting of the slider at a constant rate in response to the gesture.

51. The system of claim 41, wherein the display element comprises a slider icon, wherein the gesture comprises a grip-and-extend-again motion of a finger of a hand of the user, wherein the controller incrementally changes a setting of the slider in response to the gesture.

52. The system of claim 1, wherein the controller manages the user interactions using a second interactive mode, the second interactive mode comprising controlling the interactions with the plurality of vehicle subsystems in response to the gesture, wherein the gesture comprises hand motions and finger motions of the user.

53. The system of claim 52, wherein the controller cycles through a plurality of control elements of the vehicle subsystems in response to the gesture, wherein each control element allows the user to control a parameter of a vehicle subsystem.

54. The system of claim 52, wherein the gesture comprises a first finger of a hand extended in a first direction.

55. The system of claim 54, wherein the gesture comprises a second finger of the hand extended in a second direction.

56. The system of claim 55, wherein the gesture comprises a vertical orientation of a combination of the hand, the first finger and the second finger, wherein the vertical orientation places the hand, the first finger and the second finger in a gravity-aligned plane.

57. The system of claim 55, wherein the gesture comprises a horizontal orientation of a combination of the hand, the first finger and the second finger, wherein the horizontal orientation places the hand, the first finger and the second finger in a gravity-perpendicular plane.

58. The system of claim 55, wherein the gesture comprises a third finger of the hand extended in the first direction and approximately aligned with the first finger.

59. The system of claim 55, wherein the second finger when extended is approximately perpendicular to the first finger.

60. The system of claim 55, wherein the second finger when extended is approximately parallel to the first finger.

61. The system of claim 55, wherein the gesture comprises the hand rotated into a position in which the first finger is pointing approximately medially.

62. The system of claim 52, comprising a graphical display coupled to the processor, the graphical display providing control to the user via the gesture, wherein the controller cycles through a plurality of control elements of the vehicle subsystems in response to the gesture, wherein each control element allows the user to control a parameter of a vehicle subsystem.

63. The system of claim 62, wherein the gesture controls position of a display element on the graphical display.

64. The system of claim 63, wherein the gesture comprises a first finger of a hand extended in a first direction.

65. The system of claim 64, wherein the display element comprises a pointer, wherein the position of the pointer is controlled by at least one of movement and aim of the first finger.

66. The system of claim 65, wherein the gesture controls a selection of a control element when the pointer is placed within a control element using the aim of the first finger and a second finger of the hand is moved from a first position to a second position, wherein the first position is extended approximately perpendicular to the first finger and the second position is extended approximately parallel to the first finger.

67. The system of claim 66, wherein the gesture comprises aiming the first finger at a position off the graphical display, wherein the controller presents a menu list.

68. The system of claim 67, wherein the gesture comprises aiming the first finger at a menu item and moving the second finger to the first position, wherein the controller selects the menu item as an active control element.

69. The system of claim 62, wherein the gesture comprises a vertical orientation of a combination of the hand and at least one finger of the hand, wherein the vertical orientation places the hand, the first finger and the second finger in a gravity-aligned plane, wherein the controller presents a first set of control elements.

70. The system of claim 69, wherein the gesture comprises a horizontal orientation of a combination of the hand and at least one finger of the hand, wherein the horizontal orientation places the hand, the first finger and the second finger in a gravity-perpendicular plane, wherein the controller presents a second set of control elements different from the first set of control elements.

71. The system of claim 1, wherein the vehicle subsystems comprise at least one of climate control devices, multi-function display devices, audio devices, radio devices, digital audio devices, telephones, navigation devices, cruise control devices, autopilot devices, automated guidance devices, and networked devices.

72. The system of claim 1, wherein the plurality of sensors comprises a plurality of time-of-flight depth-sensing cameras.

73. The system of claim 1, wherein the plurality of sensors comprises a plurality of structured light three-dimensional sensors.

74. A system comprising:

a plurality of sensors, wherein the plurality of sensors image an object and output gesture data representing a gesture, wherein the gesture data is an instantaneous state of the object at a point in time in space, wherein the object comprises at least one of a hand and a finger; and

a processor coupled to the plurality of sensors, the processor automatically detecting the gesture of the object from the gesture data, the detecting comprising extracting positions of the object, the detecting comprising recovering quantized poses and orientation vectors of the object, the detecting comprising translating the gesture data to a gesture signal and using the gesture signal to control a vehicle subsystem coupled to the processor.

75. A system comprising:

a plurality of sensors installed in a vehicle, wherein the plurality of sensors image an object and output gesture data representing a gesture made by the object, wherein the gesture data is an instantaneous state of the object at a point in time in space;

a processor coupled to the plurality of sensors, the processor automatically detecting the gesture of the object from the gesture data and translating the gesture data to a gesture signal;

a controller coupled to the processor and to vehicle subsystems of the vehicle, the controller controlling the vehicle subsystems in response to the gesture signal; and

a feedback device coupled to the processor, the feedback device providing at least one of visual and audio feedback corresponding to the gesture.

76. A method comprising:

receiving an image of an object via a plurality of sensors in a vehicle and outputting gesture data representing a gesture made by the object, wherein the gesture data is an instantaneous state of the object at a point in time in space, wherein the object comprises at least one of a hand and a finger of a user;

automatically detecting the gesture of the object from the gesture data by extracting positions of the object, recovering quantized poses and orientation vectors of the object, and translating the gesture data to a gesture signal; and

managing user interactions with a plurality of vehicle subsystems in response to the gesture signal.

77. The method of claim 76, comprising positioning the plurality of sensors to view a passenger compartment of the vehicle, wherein the plurality of sensors comprises a plurality of cameras.

78. The method of claim 76, wherein the gesture data comprises a three-dimensional data point cloud representative of the object.

79. The method of claim 76, wherein the positions of the object comprise position of a hand.

80. The method of claim 76, wherein the positions of the object comprise position of a set of fingers.

81. The method of claim 76, wherein the quantized poses of the object comprise quantized poses of at least one hand.

82. The method of claim 76, wherein the orientation vectors of the object comprise continuous orientation vectors of at least one hand.

83. The method of claim 76, wherein the orientation vectors of the object comprise continuous orientation vectors of a set of fingers.

84. The method of claim 76, wherein the translating comprises translating information of the gesture to a gesture notation, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

85. The method of claim 84, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the object.

86. The method of claim 84, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the object.

87. The method of claim 84, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the object.

88. The method of claim 76, comprising providing to the user indications corresponding to the gesture, wherein the indications comprise at least one of an interpretation of the gesture and an effect realized on the plurality of vehicle subsystems in response to the gesture.

89. The method of claim 76, wherein the managing of the user interactions comprises using a first interactive mode, the first interactive mode comprising controlling the interactions with the plurality of vehicle subsystems in response to the gesture, wherein the gesture comprises finger motions of the user.

90. The method of claim 89, wherein the gesture comprises a grip-and-extend-again motion of two fingers of a hand, wherein the managing user interactions comprises cycling through a plurality of control elements of the vehicle subsystems in response to the gesture, wherein each control element allows the user to control a parameter of a vehicle subsystem.

91. The method of claim 90, wherein the gesture comprises a grip-and-extend-again motion of a finger of a hand of the user, wherein the managing user interactions comprises incrementally changing a setting of the parameter in response to the gesture.

92. The method of claim 90, wherein the gesture comprises holding a first finger down and extending a second finger, wherein the managing user interactions comprises changing a setting of the parameter at a constant rate in response to the gesture.

93. The method of claim 89, wherein hands of the user remain in contact with a guidance mechanism of the vehicle during the gesture.

94. The method of claim 89, comprising providing to the user indications corresponding to the gesture, wherein the gesture controls position of a display element on the graphical display.

95. The method of claim 94, wherein the finger motions effect non-linear selection of a control element of the vehicle subsystems via the display element, wherein each control element allows the user to control a parameter of a vehicle subsystem.

96. The method of claim 95, wherein the gesture comprises relative movement of a finger tip, wherein the managing user interactions comprises controlling movement of the display element on the graphical display in response to the relative movement of the finger tip.

97. The method of claim 96, comprising selecting interface graphics of the display by proximity of the display element, wherein the display element is a pointer.

98. The method of claim 97, wherein the interface graphics correspond to control elements of the vehicle subsystems, wherein an interface graphic of each control element allows the user to control a parameter of a vehicle subsystem.

99. The method of claim 76, wherein the managing of the user interactions comprises using a second interactive mode, the second interactive mode comprising controlling the interactions with the plurality of vehicle subsystems in response to the gesture, wherein the gesture comprises hand motions and finger motions of the user.

100. The method of claim 99, comprising cycling through a plurality of control elements of the vehicle subsystems in response to the gesture, wherein each control element allows the user to control a parameter of a vehicle subsystem.

101. The method of claim 99, wherein the gesture comprises a first finger of a hand extended in a first direction.

102. The method of claim 101, wherein the gesture comprises a second finger of the hand extended in a second direction.

103. The method of claim 102, wherein the gesture comprises a vertical orientation of a combination of the hand, the first finger and the second finger, wherein the vertical orientation places the hand, the first finger and the second finger in a gravity-aligned plane.

104. The method of claim 102, wherein the gesture comprises a horizontal orientation of a combination of the hand, the first finger and the second finger, wherein the horizontal orientation places the hand, the first finger and the second finger in a gravity-perpendicular plane.

105. The method of claim 102, wherein the gesture comprises a third finger of the hand extended in the first direction and approximately aligned with the first finger.

106. The method of claim 102, wherein the second finger when extended is approximately perpendicular to the first finger.

107. The method of claim 102, wherein the second finger when extended is approximately parallel to the first finger.

108. The method of claim 102, wherein the gesture comprises the hand rotated into a position in which the first finger is pointing approximately medially.

109. The method of claim 99, comprising providing a display output that provides control to a user via the gesture, wherein the managing user interactions comprises cycling through a plurality of control elements of the vehicle subsystems in response to the gesture, wherein each control element allows the user to control a parameter of a vehicle subsystem.

110. The method of claim 109, wherein the gesture controls position of a display element on the display output.

111. The method of claim 110, wherein the gesture comprises a first finger of a hand extended in a first direction.

112. The method of claim 111, wherein the display element comprises a pointer, wherein the position of the pointer is controlled by at least one of movement and aim of the first finger.

113. The method of claim 112, wherein the gesture controls a selection of a control element when the pointer is placed within a control element using the aim of the first finger and a second finger of the hand is moved from a first position to a second position, wherein the first position is extended approximately perpendicular to the first finger and the second position is extended approximately parallel to the first finger.

114. The method of claim 113, wherein the gesture comprises aiming the first finger at a position off the graphical display, wherein the controller presents a menu list.

115. The method of claim 114, wherein the gesture comprises aiming the first finger at a menu item and moving the second finger to the first position, wherein the controller selects the menu item as an active control element.