System, components and methodologies for gaze dependent gesture input control

• US 9,244,527 B2
• Filed: 03/26/2013
• Issued: 01/26/2016
• Est. Priority Date: 03/26/2013
• Status: Active Grant

• Current Assignee: Audi AG (Volkswagen AG), Volkswagen AG
• Original Assignee: Audi AG (Volkswagen AG), Volkswagen AG
• Inventors: Lathrop, William Brian, Mejia Gonzalez, Maria Esther, Maiwand, Heiko, Grant, Bryan
• Primary Examiner(s): DINH, DUC Q
• Application Number:
  US13/850,668
• Publication Number:
  US 20140292665A1
• Time in Patent Office: 1,036 Days
• Field of Search: 345 7- 9
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers

Abstract

A control system, components and methodologies enable control of vehicle and in-vehicle system functionality via gesture recognition. In illustrative embodiments, the control system, constituent components and methodologies determine what functionality a user wishes to control by tracking the user'"'"'s gaze to identify a selected function or function set. In combination with identifying the selected function and function set, one or more sensors installed in the vehicle detect one or more user gestures initiated by a user to interact with and/or control the selected function/function set.

Claims

1. A control system comprising:

at least one user interface that outputs data to a user;

at least one gaze tracking sensor that tracks a gaze of the user and outputs data indicating a gaze direction of the user;

a processor that receives the data indicating the gaze direction of the user; and

a plurality of gesture recognition sensors that configured to detect at least one gesture made by the user, on a plurality of disjunct surfaces included in an automotive vehicle in which the user is sitting and outputs data indicating the at least one gesture,

wherein the processor includes a means for determining the functionality a user wishes to control by tracking the user'"'"'s gaze using the data indicating the gaze direction of the user and identifying a selected area of interest displayed on the at least one user interface based on the data indicating the gaze direction of the user,

wherein the processor also includes a means for recognizing at least one gesture based on the data indicating the at least one gesture made by the user and for controlling at least one operation associated with the selected area of interest based on the data indicating the at least one gesture made by the user, and

wherein the control system controls operation of the vehicle or in-vehicle control system for the vehicle in which the user is sitting.

2. The control system of claim 1, wherein the at least one user interface includes a touch screen display that both outputs data to the user and receives data from the user.

3. The control system of claim 1, wherein the system further comprises a plurality of user interfaces including the at least one user interface that outputs data to the user, wherein at least one other user interface included in that plurality includes a touch surface that receives data from the user in the form of a gesture data and provides the at least one gesture recognition sensor.

4. The control system of claim 3, wherein data input to the touch surface is received from a user by received data indicating one or more positions on the touch surface where the user has drawn.

5. The control system of claim 1, wherein the system controls in-vehicle infotainment features and/or driver assistance systems within the automotive vehicle.

6. The control system of claim 1, wherein the means for determining and identifying the selected area of interest alters display of the selected area of interest on the at least one user interface once the selected area of interest is identified.

7. The control system of claim 1, wherein the means for recognizing at least one gesture and the at least one gesture recognition sensor cooperate to recognize the user'"'"'s at least one gesture made in free space.

8. The control system of claim 1, wherein the means for recognizing at least one gesture and the at least one gesture recognition sensor cooperate to recognize the user'"'"'s at least one gesture made on a surface within arm'"'"'s reach of the user.

9. The control system of claim 1, wherein the at least one gesture recognition sensor is a camera-based hand and finger gesture tracking sensor and/or tactile control surface sensor.

10. The control system of claim 1, wherein the at least one gaze tracking sensor is a camera-based eye tracking sensor.

23. The control system of claim 1, wherein the at least one user interface includes at least two displays at two locations in the vehicle.

26. The method of claim 23, wherein the at least one gesture controls a first functionality when the gaze data indicates a gaze direction of a user to the first display and the at least one gesture controls a second functionality when the gaze data indicates a gaze direction of a user to the second display.

11. A method for receiving at least one control command from a user using a control system, the method comprising:

tracking a gaze of the user using at least one gaze tracking sensor and outputting data indicating a gaze direction of the user;

receiving, at a processor, the data indicating the gaze direction of the user;

detecting via a plurality of gesture recognition sensors at least one gesture performed by the user on a plurality of disjunct surfaces included in an automotive vehicle in which the user is sitting using at least one gesture recognition sensor and outputting data indicating the at least one gesture to the processor;

determining functionality a user wishes to control by tracking the user'"'"'s gaze using the data indicating the gaze direction of the user and identifying a selected area of interest displayed on at least one user interface based on the data indicating the gaze direction of the user, and

recognizing at least one gesture based on the data indicating the at least one gesture made by the user; and

controlling at least one operation associated with the selected area of interest based on the data indicating the at least one gesture made by the user

wherein the at least one operation controlled by the method is an operation of the vehicle or in-vehicle control system for the vehicle in which the user is sitting.

12. The method of claim 11, further comprising outputting data to the user via the at least one user interface prior to tracking the gaze direction of the user.

13. The method of claim 12, wherein a plurality of user interfaces are utilized that include the at least one user interface that outputs data to the user, wherein at least one other user interface included in that plurality includes a touch surface that receives data from the user in the form of a gesture data and provides the at least one gesture recognition sensor.

14. The method of claim 13, wherein data input to the touch surface is received from a user by received data indicating one or more positions on the touch surface where the user has drawn.

15. The method of claim 11, wherein the at least one user interface includes a touch screen display that both outputs data to the user and receives data from the user.

16. The method of claim 11, wherein the at least one operation controls in-vehicle infotainment features and/or driver assistance systems within the automotive vehicle.

17. The method of claim 11, further comprising altering display of the selected area of interest on the at least one user interface once the selected area of interest is identified.

18. The method of claim 11, wherein the recognizing of the at least one gesture is performed using at least one gesture recognition sensor that recognizes the user'"'"'s at least one gesture made in free space.

19. The method of claim 11, wherein the recognizing of the at least one gesture is performed using at least one gesture recognition sensor that recognizes the user'"'"'s at least one gesture made on a surface within arm'"'"'s reach of the user.

20. The method of claim 11, wherein the at least one gesture recognition sensor is a camera-based hand and finger gesture tracking sensor and/or tactile control surface sensor.

21. The method of claim 11, wherein the at least one gaze tracking sensor is a camera-based eye tracking sensor.

24. The method of claim 11, wherein the at least one user interface includes at least two displays at two locations in the vehicle.

25. The control system of claim 24, wherein the at least one gesture controls a first functionality when the gaze data indicates a gaze direction of a user to the first display and the at least one gesture controls a second functionality when the gaze data indicates a gaze direction of a user to the second display.

22. A transportation system comprising:

at least one user interface that outputs data to a user;

at least one gaze tracking sensor that tracks a gaze of the user and outputs data indicating a gaze direction of the user;

a passenger vehicle including the at least one user interface,

a processor that receives the data indicating the gaze direction of the user; and

a plurality of camera-based gesture recognition sensors configured to detect at least one gesture made by the user on a plurality of disjunct surfaces included in an automotive vehicle in which the user is sitting and outputs data indicating the at least one gesture,

wherein the processor includes a means for determining the functionality a user wishes to control by tracking the user'"'"'s gaze using the data indicating the gaze direction of the user and identifying a selected area of interest displayed on the at least one user interface based on the data indicating the gaze direction of the user, and

wherein the processor includes software that recognizes at least one gesture based on the data indicating the at least one gesture made by the user and controls at least one operation associated with the selected area of interest based on the data indicating the at least one gesture made by the user,

wherein the at least one operation is an operation of the vehicle or in-vehicle control system for the vehicle in which the user is sitting.