System and method for controlling a vehicle user interface based on gesture angle

• US 9,346,471 B2
• Filed: 10/14/2014
• Issued: 05/24/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Edge 3 Technologies, LLC, Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Edge Technologies, Inc, Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: El Dokor, Tarek A., Cluster, Jordan, Holmes, James E., Vaghefinazari, Pedram, Yamamoto, Stuart M.
• Primary Examiner(s): NGUYEN, TAN QUANG
• Application Number:
  US14/514,160
• Publication Number:
  US 20150032331A1
• Time in Patent Office: 588 Days
• Field of Search: 701/36, 701/1, 348/48, 348/156
• US Class Current: 1/1
• CPC Class Codes: B60R 1/072 for adjusting the mirror relative to its housing
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An in-vehicle computing system allows a user to control components of the vehicle by performing gestures. The user provides a selecting input to indicate that he wishes to control one of the components. After the component is identified, the user performs a gesture to control the component. The gesture and the component that was previously selected are analyzed to generate a command for the component. Since the command is based on both the gesture and the identified component, the user can perform the same gesture in the same position within the vehicle to control different components.

Claims

1. A vehicle-based computer-implemented method for controlling a component of the vehicle, the method comprising:

receiving, via a sensor within the vehicle, a first selecting input from a user inside the vehicle, wherein the first selecting input represents a directional gesture performed by the user;

processing the first selecting input to identify and select a first component of the vehicle to be controlled by the user, wherein processing the first selecting input includes determining a line that is generated matching a direction of the directional gesture and finding at least one component whose stored three-dimensional coordinates are closest to the line;

providing, to the user, a first feedback signal confirming selection of the first component to be controlled;

subsequent to providing the first feedback signal, receiving a first data signal for controlling the first component, the first data signal representing a the directional gesture that is directed at the first component inside the vehicle; and

processing the first data signal to determine a first command for controlling the first component.

2. The method of claim 1, wherein:

the sensor within the vehicle comprises a camera system including one or more cameras positioned to capture the directional gestures performed in a capture region inside the vehicle;

and

processing the first data signal comprises performing gesture recognition on the directional gesture wherein performing gesture recognition on the directional gesture includes determining a positon of a hand of the user, a plane representing an orientation of the hand of the user, and an angle at which each joint of the hand of the user is bent.

3. The method of claim 1, wherein:

the sensor within the vehicle comprises a microphone configured to detect voice signals within the vehicle;

the first selecting input comprises a voice signal that includes a name of the first component; and

processing the first selecting input comprises performing voice recognition on the voice signal to identify the first component of the vehicle.

4. The method of claim 1, wherein:

the first feedback signal is an audio feedback signal provided through one or more speakers inside the vehicle; and

the audio feedback signal contains a name of the first component and includes audio instructions for controlling the first component.

5. The method of claim 1, wherein:

the first feedback signal is a visual feedback signal provided through one or more displays inside the vehicle; and

the visual feedback signal contains a name or an image of the first component and includes textual, graphical, or animation instructions for controlling the first component.

6. The method of claim 1, further comprising:

subsequent to providing the first feedback signal, providing the user a prompt to verify whether the first identified component was correctly selected based on the first selecting input.

7. The method of claim 6, further comprising:

responsive to receiving, from the user, an indication that the first component was incorrectly selected, providing the user an additional prompt to enable the user to rectify the selection.

8. A vehicle-based computing system for controlling a component of the vehicle, the system comprising:

one or more processors; and

a non-transitory computer-readable storage medium for storing computer program instructions, the program instructions when executed by the processors cause the processors to perform steps including;

receiving, via a sensor within the vehicle, a first selecting input from a user inside the vehicle, wherein the first selecting input represents a directional gesture performed by the user;

processing the first selecting input to identify and select a first component of the vehicle to be controlled by the user, wherein processing the first selecting input includes determining a line that is generated matching a direction of the directional gesture and finding at least one component whose stored three-dimensional coordinates are closest to the line;

providing, to the user, a first feedback signal confirming selection of the first component to be controlled;

subsequent to providing the first feedback signal, receiving a first data signal for controlling the first component, the first data signal representing a the directional gesture that is directed at the first component inside the vehicle; and

processing the first data signal to determine a first command for controlling the first component.

9. The vehicle-based computing system of claim 8, wherein:

the sensor within the vehicle comprises a camera system including one or more cameras positioned to capture the directional gestures performed in a capture region inside the vehicle; and

processing the first data signal comprises performing gesture recognition on the directional gesture wherein performing gesture recognition on the directional gesture includes determining a positon of a hand of the user, a plane representing an orientation of the hand of the user, and an angle at which each joint of the hand of the user is bent.

10. The vehicle-based computing system of claim 8, wherein:

the sensor within the vehicle comprises a microphone configured to detect voice signals within the vehicle;

the first selecting input comprises a voice signal that includes a name of the first component; and

instructions for processing the first selecting input comprise instructions for performing voice recognition on the voice signal to identify the first component of the vehicle.

11. The vehicle-based computing system of claim 8, wherein:

the first feedback signal is an audio feedback signal provided through one or more speakers inside the vehicle; and

the audio feedback signal contains a name of the first component and includes audio instructions for controlling the first component.

12. The vehicle-based computing system of claim 8, wherein:

the first feedback signal is a visual feedback signal provided through one or more displays inside the vehicle; and

the visual feedback signal contains a name or an image of the first component and includes textual, graphical, or animation instructions for controlling the first component.

13. The vehicle-based computing system of claim 8, further comprising instructions for:

subsequent to providing the first feedback signal, providing the user a prompt to verify whether the first component was correctly selected based on the first selecting input.

14. The vehicle-based computing system of claim 13, further comprising instructions for:

responsive to receiving, from the user, an indication that the first component was incorrectly selected, providing the user an additional prompt to enable the user to rectify the selection.