Method and control system for operating a motor vehicle

• US 9,493,169 B2
• Filed: 07/08/2013
• Issued: 11/15/2016
• Est. Priority Date: 07/06/2012
• Status: Active Grant

• Current Assignee: Audi AG (Volkswagen AG)
• Original Assignee: Audi AG (Volkswagen AG)
• Inventors: Ebner, Andr
• Primary Examiner(s): FREJD, RUSSELL WARREN
• Application Number:
  US14/412,213
• Publication Number:
  US 20150336588A1
• Time in Patent Office: 1,226 Days
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60R 16/037 for occupant comfort , e.g. for automatic adjustment of appliances according to personal settings, e.g. seats, mirrors, steering wheel
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2556/00 Input parameters relating to data
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G05B 13/0265 the criterion being a learning criterion
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

A method and control system for operating a motor vehicle, sense a gesture directed at an object from a vehicle occupant using a gesture recognition unit, interpret the gesture and identify of the object using a data processing unit and generate a corresponding control command on the basis of the interpreted gesture and the identified object. An assistance system of the motor vehicle is actuated in accordance with the control command using the data processing unit. A voice command, uttered by the vehicle occupant is sensed before, during or after the sensing of the gesture, using a voice recognition unit. A combined interpretation of the gesture, the identified object and the voice command is performed, and the control command is generated on the basis of the combined interpretation, using the data processing unit.

Claims

1. A method for operating a motor vehicle, comprising:

sensing a gesture of a vehicle occupant, the gesture being a finger pointing toward an object, the gesture being sensed by a gesture recognition unit;

interpreting the gesture and identifying the object, using a data processing unit;

sensing a voice command using a voice recognition unit, the voice command being uttered by the vehicle occupant before, during or after the gesture is sensed;

using the data processing unit to perform a combined interpretation of the gesture, the object and the voice command and to generate a control command based on the combined interpretation; and

using the data processing unit to actuate an assistance system of the motor vehicle in accordance with the control command,

wherein

both a gesture directed at an outside object arranged outside the motor vehicle and a gesture directed at an inside object arranged inside the motor vehicle prompt sensing of the gesture, interpreting the gesture and identifying the object.

2. The method as claimed in claim 1, wherein

for identifying the object, a motor vehicle position and a motor vehicle orientation are ascertained and then the object is identified by a comparison with digital map information.

3. The method as claimed in claim 1, wherein

the gesture is directed at the outside object arranged outside the motor vehicle, and

for identifying the object, a motor vehicle position and a motor vehicle orientation are ascertained and then the object is identified by a comparison with digital map information.

4. The method as claimed in claim 1, wherein

the gesture is directed at the outside object arranged outside the motor vehicle, and

for identifying the object, an exterior camera is mounted on the motor vehicle.

5. The method as claimed in claim 1, wherein

combined interpretation of the gesture, the object and the voice command to generate the control command is performed only if the gesture and the voice command are sensed within a prescribed period.

6. The method as claimed in claim 1, wherein

the gesture is a gesture directed at a further motor vehicle,

the voice command is a request to setup a communication link to a communication device of the further motor vehicle,

the gesture is sensed and interpreted in combination with the voice command, and

the control command actuates a communication device of the assistance system to set up the communication link.

7. The method as claimed in claim 6, wherein

the further motor vehicle is highlighted in a contact-analogous head-up display as soon as the gesture has been recognized.

8. The method as claimed in claim 6, wherein

the communication link comprises a chat function or a telephone call.

9. The method as claimed in claim 1, wherein

the gesture is directed at the outside object arranged outside the motor vehicle,

the voice command is an information request,

the gesture is sensed and interpreted in combination with the voice command, and

the control command actuates a navigation unit of the assistance system to provide a piece of information about the outside object.

10. The method as claimed in claim 1, wherein

the gesture is directed at the outside object arranged outside the motor vehicle,

the voice command is a navigation command,

the gesture is sensed and interpreted in combination with the voice command, and

the control command actuates a navigation unit of the assistance system to ascertain a route to the outside object.

11. The method as claimed in claim 10, wherein

the outside object is highlighted in a contact-analogous head-up display as soon as the gesture has been recognized.

12. The method as claimed in claim 1, wherein

the gesture is directed at the outside object arranged outside the motor vehicle, and

the object is highlighted in a contact-analogous head-up display of the motor vehicle as soon as the gesture has been recognized.

13. The method as claimed in claim 1, wherein

the gesture is directed at an air conditioning system, and

the voice command specifies a setting change for the air conditioning system.

14. The method as claimed in claim 1, wherein

the gesture recognition unit identifies the object, and

the voice recognition unit identifies how the object should be actuated.

15. The method as claimed in claim 1, wherein

the gesture recognition unit is a camera that senses movements or gestures of the vehicle occupant.

16. The method as claimed in claim 1, wherein

the gesture is directed at the outside object arranged outside the motor vehicle,

the voice command is an information request,

the gesture is sensed and interpreted in combination with the voice command, and

the control command is output to a communication device of the assistance system which accesses information related to the information request through an internet connection, and the information accessed using the internet connection is provided to the vehicle occupant.

17. A control system for operating a motor vehicle, comprising:

a gesture recognition unit to recognize a gesture of a vehicle occupant, the gesture being a finger pointing directed at an object;

a voice recognition unit to sense a voice command uttered by the vehicle occupant before, during or after the gesture is recognized; and

a data processing unit to interpret the gesture, to identify the object, to perform a combined interpretation of the gesture, the object and the voice command, to generate a control command based on the combined interpretation and to actuate an assistance system of the motor vehicle in accordance with the control command, wherein

the gesture recognition unit recognizes both an outside gesture directed at an object arranged outside the motor vehicle and an inside gesture directed at an object arranged inside the motor vehicle, so that the data processing unit interprets the gesture and identifies the object.

18. The control system as claimed in claim 17, wherein

the gesture recognition unit monitors exclusively a prescribed interior area of the motor vehicle to sense only gestures from a particularly located vehicle occupant arranged in the prescribed interior area.