Multi-modal vehicle operating device

• US 8,532,871 B2
• Filed: 03/14/2008
• Issued: 09/10/2013
• Est. Priority Date: 06/05/2007
• Status: Active Grant

• Current Assignee: Nexgen Control Systems LLC
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Okada, Reiko, Matsutani, Kiyoshi, Kohno, Atsushi, Sato, Fumitaka, Kawana, Yuta, Yamazaki, Wataru
• Primary Examiner(s): Jayne, Darnell
• Assistant Examiner(s): Varghese, Sasha T
• Application Number:
  US12/599,545
• Publication Number:
  US 20110218696A1
• Time in Patent Office: 2,006 Days
• Field of Search: 701/431, 701/432, 701/436, 701443-444, 701/36, 704/200, 704/246, 345 7- 9
• US Class Current: 701/36
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A vehicle operating device includes: a voice operation unit (3) for recognizing an uttered voice and outputting a voice recognition result; a spatial operation unit (2) for recognizing a movement performed within a predetermined space and outputting a spatial recognition result; a main processing unit (4) for executing a processing corresponding to the voice recognition result and the spatial recognition result; and a display unit (5) for displaying an image generated in accordance with an instruction from the main processing unit, the image being superimposed on an actual scene that can be viewed through a windshield.

Claims

1. A vehicle operating device, comprising:

a voice operation unit for recognizing an uttered voice and outputting a voice recognition result;

a spatial operation unit for recognizing a movement performed within a predetermined space and outputting a spatial recognition result;

a main processing unit for executing a specific processing corresponding to said voice recognition result and executing a separate specific processing corresponding to said spatial recognition result;

a display unit for displaying an image generated in accordance with an instruction from said main processing unit, said image being superimposed on an actual scene that can be viewed through a windshield; and

an individual authentication processing unit for authenticating an individual,

wherein said main processing unit causes said display unit to display a menu corresponding to said voice recognition result, and executes processing corresponding to said spatial recognition result to select a single item from said menu, and

said spatial operation unit obtains a recognition mode set in relation to said individual authenticated by said individual authentication unit, and when an item displayed on said display unit is selected, said spatial operation unit recognizes said movement performed within said space by a relative position thereof when said recognition mode is a relative mode, and by an absolute position thereof when said recognition mode is an absolute mode.

2. The vehicle operating device according to claim 1, wherein said spatial operation unit begins spatial operation recognition by detecting a movement performed in a specific position of said space.

3. The vehicle operating device according to claim 1, wherein said spatial operation unit recognizes said movement performed in said space three-dimensionally.

4. The vehicle operating device according to claim 1, wherein said voice operation unit recognizes an utterance of a vocabulary as an operation subject,

said spatial operation unit recognizes said movement performed within said space, and

said main processing unit causes said display unit to display information based on said voice recognition result, determines operation content detail on the basis of said spatial recognition result, and then executes processing corresponding to a result of said determination.

5. The vehicle operating device according to claim 1, wherein said display unit generates an image conforming to an operation method recognized by said spatial operation unit in accordance with an instruction from said main processing unit, and displays said generated image, said image being superimposed on said actual scene that can be viewed through said windshield.

6. The vehicle operating device according to claim 1, wherein said display unit displays an icon using an image expressing a spatial operation that can be recognized in said spatial operation unit.

7. The vehicle operating device according to claim 1, wherein when an item displayed on said display unit is selected, said spatial operation unit recognizes said movement performed within said space by a relative position thereof.

8. The vehicle operating device according to claim 1, wherein when an item displayed on said display unit is selected, said spatial operation unit recognizes said movement performed within said space by an absolute position thereof.

9. The vehicle operating device according to claim 1, wherein when said recognition mode is said absolute mode, said spatial operation unit modifies a position in which said movement performed within said space is recognized in accordance with said individual authenticated by said individual authentication unit.

10. The vehicle operating device according to claim 1, further comprising a voice processing unit for performing voice guidance relating to an item displayed on said display unit when said item is selected using said spatial operation unit.

11. The vehicle operating device according to claim 1, further comprising a voice processing unit for performing voice guidance corresponding to a spatial operation when said spatial operation is recognized by said spatial operation unit.

12. The vehicle operating device according to claim 1, wherein said main processing unit comprises a vehicle information acquisition unit for obtaining vehicle information, and modifies a content to be displayed on said display unit in accordance with said vehicle information obtained by said vehicle information acquisition unit.

13. The vehicle operating device according to claim 1, wherein said main processing unit comprises a proficiency determination unit for determining a spatial operation proficiency level, and modifies a spatial operation method to be recognized by said spatial operation unit or a content to be displayed on said display unit in accordance with said proficiency level determined by said proficiency determination unit.

14. The vehicle operating device according to claim 13, further comprising:

an individual authentication processing unit for authenticating an individual; and

a proficiency management unit for managing said proficiency level determined by said proficiency determination unit for each individual who is authenticated by said individual authentication processing unit, wherein

said main processing unit obtains a proficiency level of an individual authenticated by said individual authentication processing unit from said proficiency management unit, and modifies said spatial operation method to be recognized by said spatial operation unit or said content to be displayed on said display unit in accordance with said obtained proficiency level.

15. The vehicle operating device according to claim 1, wherein said voice operation unit begins recognition of said uttered voice when a specific spatial operation is recognized by said spatial operation unit.

16. The vehicle operating device according to claim 1, wherein when said spatial operation unit recognizes that a spatial operation is continued, said voice operation unit continues a voice input.

17. The vehicle operating device according to claim 1, wherein said voice operation unit comprises a plurality of types of recognition dictionaries used in voice recognition, and switches between said recognition dictionaries in accordance with said spatial operation recognized by said spatial operation unit.