NATURAL QUERY INTERFACE FOR CONNECTED CAR

• US 20130030811A1
• Filed: 07/29/2011
• Published: 01/31/2013
• Est. Priority Date: 07/29/2011
• Status: Abandoned Application

• Current Assignee: Godo Kaisha IP Bridge 1 (IP Bridge, Inc.)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Talati, Rohit, Kryze, David, Olleon, Jules, Sugiura, Akihiko
• Application Number:
  US13/193,986
• Publication Number:
  US 20130030811A1
• Time in Patent Office: Days
• US Class Current: 704/267
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2360/113 Scrolling through menu items
• CPC Class Codes: B60K 2360/117 Cursors
• CPC Class Codes: B60K 2360/141 Activation of instrument input devices by approaching fingers or pens
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/149 Instrument input by detecting viewing direction not otherwise provided for
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/169 Remaining operating distance or charge
• CPC Class Codes: B60K 2360/182 Distributing information between displays
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/55 Remote control arrangements
• CPC Class Codes: B60K 2360/566 Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2360/589 Wireless data transfers
• CPC Class Codes: B60K 2360/5894 SIM cards
• CPC Class Codes: B60K 2360/5899 Internet
• CPC Class Codes: B60K 2360/592 Data transfer involving external databases
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 : Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 : characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/29 : Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: B60K 35/80 : Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 : Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/107 : Static hand or arm
• CPC Class Codes: G10L 2015/227 : of the speaker; Human-factor methodology

Abstract

Sensors within the vehicle monitor driver movement, such as face and head movement to ascertain the direction a driver is looking, and gestural movement to ascertain what the driver may be pointing at. This information is combined with video camera data taken of the external vehicle surroundings. The apparatus uses these data to assist the speech dialogue processor to disambiguate phrases uttered by the driver. The apparatus can issue informative responses or control vehicular functions based on queries automatically generated based on the disambiguated phrases.

Claims

1. A natural query processing apparatus for use in a vehicle, comprising:

a driver sensor system disposed inside the vehicle and positioned to monitor physical activity of the driver of the vehicle and to produce driver activity data as a function of time;

a camera system disposed on the vehicle and positioned to collect visual data regarding conditions external to the vehicle as a function of time;

a speech processor that performs speech recognition to convert driver utterances into query data as a function of time;

a correlation processor with non-transitory computer-readable memory that time-correlates and stores said driver activity data, said visual data and said query data;

a disambiguation processor that accesses the computer-readable memory and formulates a computer system query using the stored, time-correlated driver activity data, the visual data and the query data;

the disambiguation processor being operative to ascertain from said stored driver activity data what external condition the driver is referring to during a driver utterance and to formulate the computer system query to correspond to that ascertained external condition.

2. The apparatus of claim 1 wherein said driver sensor system includes at least one camera directed to the driver'"'"'s face and operative to sense the direction the driver is looking as a function of time.

3. The apparatus of claim 1 wherein said driver sensor system includes at least one camera directed to the driver'"'"'s face and operative to extract and compare facial features to a set of predefined models to determine the direction the driver is looking as a function of time.

4. The apparatus of claim 1 wherein said driver sensor system includes at least one camera directed to the driver'"'"'s face and operative to extract and compare facial features to a set of predefined models and thereby determine driver head position and orientation as a function of time.

5. The apparatus of claim 1 wherein said driver sensor system includes at least one camera directed to the driver'"'"'s eyes and operative to sense the direction the driver is looking as a function of time.

6. The apparatus of claim 1 wherein the driver sensor system includes a gesture sensor responsive to gestural movements of the driver'"'"'s arms and hands.

7. The apparatus of claim 1 further comprising an information retrieval system receptive of said computer system query that generates a response retrieved from a source of stored information.

8. The apparatus of claim 7 wherein said response is converted into synthesized or digitally reproduced speech and output to the driver via an infotainment system within the vehicle.

9. The apparatus of claim 7 wherein said response is converted into graphical or textual information displayed on a display apparatus located within the vehicle.

10. The apparatus of claim 1 wherein at least two of said speech processor, correlation processor and disambiguation processor are implemented by a single processor preprogrammed to perform multiple tasks.

11. The apparatus of claim 1 further comprising a vehicle location sensor coupled to said correlation processor and operative to provide vehicle location data as a function of time;

and

wherein said correlation processor time-correlates and stores said vehicle location data in said memory.