Eye gaze enabled navigation system

• US 9,354,073 B2
• Filed: 12/09/2013
• Issued: 05/31/2016
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Censo, Davide Di, Marti, Stefan
• Primary Examiner(s): BUTLER, RODNEY ALLEN
• Application Number:
  US14/100,386
• Publication Number:
  US 20150160033A1
• Time in Patent Office: 904 Days
• Field of Search: 701/300, 701/301, 701/400
• US Class Current: 1/1
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3626 Details of the output of route guidance instructions traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3655 Timing of guidance instructions
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A navigation system is configured to generate a sequence of driving instructions for directing a driver of a vehicle from a current location to a final destination. The navigation system is configured to capture sensor data that reflects the field of view of the driver and to identify objects in that field of view. The navigation system then customizes the driving instructions to reference specific objects, thereby generating context-specific driving instructions. The navigation system is also configured to identify particular objects upon which the driver is focused, and to generate context-specific driving instructions relative to such objects of focus. The navigation system may also provide correction information to the driver upon determining that the driver has incorrectly focused on a different object than an object referenced in a context-specific driving instruction.

Claims

1. A system configured to generate context-specific driving instructions for a driver of a vehicle, the system comprising:

at least one sensor configured to capture sensor data that reflects a field of view of the driver;

an eye-gaze sensor configured to determine an eye-gaze direction of the driver;

a computing device configured to;

process the sensor data to identify, based on the eye-gaze direction of the driver, a first object within the field of view of the driver, and

based on identifying the first object, generate a context-specific driving instruction that references the first object; and

at least one output device configured to output the context-specific driving instruction to the driver.

2. The system of claim 1, wherein the eye-gaze sensor is further configured to determine a depth at which the driver is focusing, and the computing device is further configured to identify the first object by determining, based on the depth at which the driver is focusing, that the driver is focused on the first object.

3. The system of claim 1, wherein the computing device is further configured to:

determine that the driver is focused on a second object within the field of view of the driver instead of the first object based on information received from the eye-gaze sensor;

generate correction information that distinguishes the first object from the second object and indicates that the driver should be focused upon the first object; and

cause the at least one output device to output the correction information to the driver.

4. The system of claim 1 wherein the computing device is further configured to:

receive a query from the driver that references a second object within the field of view of the driver;

generate an error message indicating that the driver is incorrectly focused upon the second object; and

cause the at least one output device to output the error message to the driver.

5. The system of claim 1, wherein the computing device is further configured to:

receive input from the driver specifying a final destination;

determine a current location of the vehicle;

generate a sequence of driving instructions to direct the driver from the current location to the final destination, wherein each driving instruction in the sequence of driving instructions indicates a particular driving action to be performed by the driver;

identify a current driving instruction in the sequence of driving instructions that is associated with the current location of the vehicle; and

generate the context-specific driving instruction by modifying the current driving instruction to include a reference to the first object.

6. The system of claim 1, wherein the computing device is configured to identify the first object by determining, based on the eye-gaze direction of the driver, that the driver is looking towards the first object.

7. The system of claim 2 wherein the computing device is further configured to:

receive a query from the driver that references the first object;

generate a confirmation message indicating that the driver is correctly focused upon the first object; and

cause the at least one output device to output the confirmation message to the driver.

8. The system of claim 3, wherein the computing device is further configured to:

receive a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;

generate a confirmation message confirming that the driver is properly focused upon the first object instead of the second object; and

cause the at least one output device to output the confirmation message to the driver.

cause the at least one output device to output the error message to the driver.

9. A non-transitory computer-readable medium that, when executed by a processing unit, causes the processing unit to generate context-specific driving instructions for a driver of a vehicle, by performing the steps of:

capturing data from a visual panorama that surrounds at least a portion of the vehicle;

receiving an eye-gaze direction of the driver from an eye-gaze sensor;

identifying a first object within the visual panorama based on the eye-gaze direction;

based on identifying the first object, generating a context-specific driving instruction that references the first object; and

outputting the context-specific driving instruction to the driver.

10. The non-transitory computer-readable medium of claim 9, further comprising the step of receiving a depth at which the driver is focusing from the eye-gaze sensor, and wherein identifying the first object further comprises determining, based on the depth at which the driver is focusing, that the driver is focused on the first object.

11. The non-transitory computer-readable medium of claim 9, further comprising the steps of:

determining that the driver is focused on a second object within the visual panorama instead of the first object based on the information received from the eye-gaze sensor;

generating correction information indicating that the driver should be focused upon the first object and should not be focused on the second object; and

outputting the correction information to the driver.

12. The non-transitory computer-readable medium of claim 9, further comprising the step of:

determining a final destination for the vehicle;

determining a current location of the vehicle;

acquiring a sequence of driving instructions to direct the driver from the current location to the final destination, wherein each driving instruction in the sequence of driving instructions indicates a particular driving action to be performed by the driver;

identifying a current driving instruction in the sequence of driving instructions that indicates a subsequent driving action to be performed by the driver; and

generating the context-specific driving instruction by modifying the current driving instruction to include a reference to the first object.

13. The non-transitory computer-readable medium of claim 10, further comprising the steps of:

receiving a query from the driver regarding a second object within the field of view of the driver;

generating an error message indicating that the driver should ignore the second object; and

outputting the error message to the driver.

14. The non-transitory computer-readable medium of claim 10, further comprising the steps of:

receiving a query from the driver regarding the first object;

generating a confirmation message indicating that the driver should continue to focus upon the first object; and

outputting the confirmation message to the driver.

15. The non-transitory computer-readable medium of claim 11, further comprising the steps of:

receiving a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;

generating a confirmation message based on the notification; and

outputting the confirmation message to the driver.

16. A computer-implemented method for generating context-specific driving instructions for a driver of a vehicle, the method comprising:

capturing data that reflects a field of view of the driver;

receiving, via an eye-gaze sensor, an eye-gaze direction of the driver;

processing the data to identify, based on the eye-gaze direction of the driver, a first object within the field of view of the driver;

based on identifying the first object, generating a context-specific driving instruction that references the first object; and

outputting the context-specific driving instruction to the driver.

17. The computer-implemented method of claim 16, further comprising:

determining that the driver is focused on a second object within the field of view of the driver instead of the first object based on information received from the eye-gaze sensor;

generating correction information that distinguishes the first object from the second object and indicates that the driver should be focused upon the first object; and

outputting the correction information to the driver.

18. The computer-implemented method of claim 16, further comprising receiving a depth at which the driver is focusing from the eye-gaze sensor, and wherein identifying the first object further comprises determining, based on the depth at which the driver is focusing, that the driver is focused upon the first object.

19. The computer-implemented method of claim 16, further comprising:

determining a current location of the vehicle;

acquiring a sequence of driving instructions to direct the driver from the current location to the final destination, wherein each driving instruction in the sequence of driving instructions indicates a particular driving action to be performed by the driver;

identifying a current driving instruction in the sequence of driving instructions that indicates a subsequent driving action to be performed by the driver; and

generating the context-specific driving instruction by modifying the current driving instruction to include a reference to the first object.

20. A system configured to generate context-specific driving instructions for a driver of a vehicle, the system comprising:

at least one sensor configured to capture sensor data that reflects a field of view of the driver;

an eye-gaze sensor configured to identify one or more objects within the field of view of the driver upon which the driver is focused;

a computing device configured to;

process the sensor data to identify a first object within the field of view of the driver;

generate a context-specific driving instruction that references the first object;

determine that the driver is focused on a second object within the field of view of the driver instead of the first object based on information received from the eye-gaze sensor; and

generate correction information that distinguishes the first object from the second object and indicates that the driver should be focused upon the first object; and

at least one output device configured to;

output the context-specific driving instruction to the driver; and

output the correction information to the driver.

21. The system of claim 20, wherein the computing device is further configured to:

receive a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;

generate a confirmation message confirming that the driver is properly focused upon the first object instead of the second object; and

cause the at least one output device to output the confirmation message to the driver.

22. A non-transitory computer-readable medium that, when executed by a processing unit, causes the processing unit to generate context-specific driving instructions for a driver of a vehicle, by performing the steps of:

capturing data from a visual panorama that surrounds at least a portion of the vehicle;

identifying a first object within the visual panorama;

generating a context-specific driving instruction that references the first object;

outputting the context-specific driving instruction to the driver;

receiving information from an eye-gaze sensor that indicates one or more objects within the visual panorama upon which the driver is focused;

determining that the driver is focused on a second object within the visual panorama instead of the first object based on the information received from the eye-gaze sensor;

generating correction information indicating that the driver should be focused upon the first object and should not be focused on the second object; and

outputting the correction information to the driver.

23. The non-transitory computer-readable medium of claim 22, further comprising the steps of:

receiving a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;

generating a confirmation message based on the notification; and

outputting the confirmation message to the driver.