Gesture-based automotive controls

• US 8,942,881 B2
• Filed: 04/02/2012
• Issued: 01/27/2015
• Est. Priority Date: 04/02/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hobbs, Nicholas Kenneth, Chi, Liang-Yu (Tom)
• Primary Examiner(s): To, Tuan C.
• Assistant Examiner(s): Kan, Yuri
• Application Number:
  US13/437,730
• Publication Number:
  US 20130261871A1
• Time in Patent Office: 1,030 Days
• Field of Search: 701/1, 701/2, 701/23, 701/26, 701/28, 701/45, 701/47, 700/245, 715/852, 340/435, 340/457.1, 345/158, 348/48
• US Class Current: 701/28
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods and apparatuses for gesture-based controls are disclosed. In one aspect, a method is disclosed that includes maintaining a correlation between a plurality of predetermined gestures, in combination with a plurality of predetermined regions of a vehicle, and a plurality of functions. The method further includes recording three-dimensional images of an interior portion of the vehicle and, based on the three-dimensional images, detecting a given gesture in a given region of the vehicle, where the given gesture corresponds to one of the plurality of predetermined gestures and the given region corresponds to one of the plurality of predetermined regions. The method still further includes selecting, based on the correlation, a function associated with the given gesture in combination with the given region and initiating the function in the vehicle.

Claims

1. A method comprising:

maintaining, by a computer system, a correlation between a plurality of predetermined gestures, in combination with a plurality of predetermined regions of a vehicle, and a plurality of functions, such that each gesture in the plurality of predetermined gestures, in combination with a particular region of the plurality of predetermined regions, is associated with a particular function in the plurality of functions, wherein the plurality of predetermined regions comprise a first region associated with a first controllable component of the plurality of controllable components and a second region associated with a second controllable component of the plurality of controllable components;

recording three-dimensional images of an interior portion of the vehicle;

based on the three-dimensional images, detecting, by the computer system, a given gesture and a position of the given gesture, wherein the given gesture corresponds to one of the plurality of predetermined gestures;

determining, by the computer system, that the position of the given gesture corresponds to the first region rather than the second region;

responsive to determining that the position of the given gesture corresponds to the first region rather than the second region, selecting, by the computer system and based on the correlation, a function associated with the given gesture in combination with the first region; and

controlling, by the computer system, the first controllable component of the vehicle based on the function.

2. The method of claim 1, wherein the three-dimensional images are recorded using at least one of a depth camera and a three-dimensional laser scanner.

3. The method of claim 1, further comprising, in response to detecting the given gesture, triggering at least one of an audible feedback, a visual feedback, and a haptic feedback.

4. The method of claim 1, wherein the three-dimensional images are recorded from a ceiling of the vehicle.

5. The method of claim 1, further comprising:

determining an extent of the given gesture; and

determining an operational parameter related to the function based on the extent,

wherein controlling the first controllable component of the vehicle based on the function comprises controlling the first controllable component of the vehicle based on the function and the operational parameter.

6. The method of claim 1, wherein recording three-dimensional images of an interior portion of the vehicle comprises recording three-dimensional images of the interior portion of the vehicle from a first position and recording three-dimensional images of the interior portion of the vehicle from a second position.

7. The method of claim 1, wherein the first region of the vehicle includes an interior surface of the vehicle.

8. The method of claim 7, wherein the interior surface comprises at least one of a dashboard of the vehicle, a steering wheel of the vehicle, an air-conditioning vent, a seat of the vehicle, a headrest of the vehicle, an arm rest of the vehicle, a rear-view mirror of the vehicle, a window of the vehicle, a driver of the vehicle, and a passenger of a vehicle.

9. The method of claim 1, wherein the first controllable component comprises a controllable component selected from the group consisting of:

an air conditioning system, heater, climate control system, cruise control system, audio system, seat, window, windshield wipers, entertainment system, display, or sunscreen in the vehicle.

10. A vehicle comprising:

a camera configured to record three-dimensional images of an interior portion of the vehicle;

a plurality of controllable components;

at least one processor; and

data storage comprising;

(i) a correlation between a plurality of predetermined gestures, in combination with a plurality of predetermined regions of the vehicle, and a plurality of functions, such that each gesture in the plurality of predetermined gestures, in combination with a particular region of the plurality of predetermined regions, is associated with a particular function in the plurality of functions, wherein the plurality of predetermined regions comprise a first region associated with a first controllable component of the plurality of controllable components and a second region associated with a second controllable component of the plurality of controllable components, and

(ii) instructions executable by the at least one processor to;

(a) record three-dimensional images of an interior portion of the vehicle,

(b) based on the three-dimensional images, detect a given gesture and a position of the given gesture, wherein the given gesture corresponds to one of the plurality of predetermined gestures,

(c) determine that the position of the given gesture corresponds to the first region rather than the second region,

(d) responsive to determining that the position of the given gesture corresponds to the first region rather than the second region, select, based on the correlation, a function associated with the given gesture in combination with the first region, and

(e) control the first controllable component based on the function.

11. The vehicle of claim 10, wherein the camera comprises at least one of a depth camera and a three-dimensional laser scanner.

12. The vehicle of claim 10, further comprising at least one feedback mechanism configured to provide feedback in response to detecting the given gesture.

13. The vehicle of claim 12, wherein the at least one feedback mechanism comprises at least one of an audible feedback mechanism, a visual feedback mechanism, and a haptic feedback mechanism.

14. The vehicle of claim 10, wherein the camera is positioned on a ceiling of the vehicle.

15. The vehicle of claim 10, wherein the first controllable component comprises a controllable component selected from the group consisting of:

an air conditioning system, heater, climate control system, cruise control system, audio system, seat, window, windshield wipers, and a sunscreen.

16. The vehicle of claim 10, further comprising an additional camera configured to record additional three-dimensional images of an additional interior portion of the vehicle.

17. The vehicle of claim 16, wherein detecting the gesture is further based on the additional three-dimensional images.

18. The vehicle of claim 10, wherein the vehicle is configured to operate in an autonomous mode.

19. A non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform the functions of:

maintaining a correlation between a plurality of predetermined gestures, in combination with a plurality of predetermined regions of a vehicle, and a plurality of functions, such that each gesture in the plurality of predetermined gestures, in combination with a particular region of the plurality of predetermined regions, is associated with a particular function in the plurality of functions, wherein the plurality of predetermined regions comprise a first region associated with a first controllable component of the plurality of controllable components and a second region associated with a second controllable component of the plurality of controllable components;

recording three-dimensional images of an interior portion of the vehicle;

based on the three-dimensional images, detecting a given gesture and a position of the given gesture, wherein the given gesture corresponds to one of the plurality of predetermined gestures and the given region corresponds to one of the plurality of predetermined regions;

determining that the position of the given gesture corresponds to the first region rather than the second region;

responsive to determining that the position of the given gesture corresponds to the first region rather than the second region, selecting, based on the correlation, a function associated with the given gesture in combination with the first region; and

controlling the first controllable component of the vehicle based on the function.

20. The non-transitory computer-readable medium of claim 19, wherein the computing device is in the vehicle.