Systems, methods, and apparatus for controlling devices based on a detected gaze

• US 9,517,776 B2
• Filed: 12/29/2011
• Issued: 12/13/2016
• Est. Priority Date: 12/29/2011
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Graumann, David L.
• Primary Examiner(s): BELLA, MATTHEW C
• Assistant Examiner(s): Shin, Soo
• Application Number:
  US13/977,516
• Publication Number:
  US 20140348389A1
• Time in Patent Office: 1,811 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/149 Instrument input by detecting viewing direction not otherwise provided for
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 37/00 Dashboards as road-vehicle superstructure sub-units B62D25/14
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/00 Input parameters relating to occupants
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Certain embodiments of the invention may include systems, methods, and apparatus for controlling devices based on a detected gaze. According to an example embodiment of the invention, a method is provided for executing computer executable instructions by one or more processors for controlling one or more devices attached to a vehicle based at least in part on a detected direction of observation of an occupant of the vehicle, the method further includes receiving one or more images from at least one camera attached to the vehicle; locating, from the one or more images, one or more body features associated with the occupant; determining a direction of gaze from the one or more located body features; and controlling the one or more devices based at least in part on the direction of gaze.

Claims

1. A method comprising executing computer executable instructions by one or more processors for controlling at least a first device and a second device attached to a vehicle based at least in part on a detected direction of observation of an occupant of the vehicle, the method further comprising:

receiving one or more images from at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with the first device and a second portion associated with the second device;

locating, from the one or more images, one or more body features associated with the occupant;

determining a direction of gaze towards the object using the one or more located body features; and

controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output.

2. The method of claim 1, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant.

3. The method of claim 1, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant.

4. The method of claim 1, further comprising determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture.

5. The method of claim 4, wherein the gesture comprises one or more of a hand movement or a voice command.

6. The method of claim 1, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information.

7. The method of claim 1, wherein determining the direction of gaze comprises tracking the occupant'"'"'s head attitude duration and stability.

8. The method of claim 1, wherein the first device and the second device comprise an integrated device.

9. A vehicle comprising:

a first device attached to the vehicle;

a second device attached to the vehicle;

at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with the first device and a second portion associated with the second device;

at least one memory for storing data and computer-executable instructions; and

one or more processors configured to access the at least one memory and further configured to execute computer executable instructions for;

receiving one or more images from at least one camera attached to the vehicle;

locating, from the one or more images, one or more body features associated with an occupant of the vehicle;

determining a direction of gaze towards the object using the one or more located body features; and

controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output.

10. The vehicle of claim 9, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant.

11. The vehicle of claim 9, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant.

12. The vehicle of claim 9, wherein the one or more processors are further configured for determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture.

13. The vehicle of claim 12, wherein the gesture comprises one or more of a hand movement or a voice command.

14. The vehicle of claim 9, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information.

15. The vehicle of claim 9, wherein determining the direction of gaze comprises tracking the occupant'"'"'s head attitude duration and stability.

16. The vehicle of claim 9, wherein the at least one of the first device and the second device comprises one or more of a video screen, a mirror with an integrated display, a stereo system, a sun roof, a car window, a lock, a car seat, a climate control system, or a dashboard console.

17. The vehicle of claim 9, wherein the first device and the second device comprise an integrated device.

18. An apparatus comprising:

at least one memory for storing data and computer-executable instructions; and

one or more processors configured to access the at least one memory and further configured to execute computer executable instructions for;

receiving one or more images from at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with a first device and a second portion associated with a second device;

locating, from the one or more images, one or more body features associated with an occupant of the vehicle;

determining a direction of gaze towards the object using the one or more located body features; and

controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output.

19. The apparatus of claim 18, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant.

20. The apparatus of claim 18, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant.

21. The apparatus of claim 18, wherein the one or more processors are further configured for determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture.

22. The apparatus of claim 21, wherein the gesture comprises one or more of a hand movement or a voice command.

23. The apparatus of claim 18, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information.

24. The apparatus of claim 18, wherein determining the direction of gaze comprises tracking the occupant'"'"'s head attitude duration and stability.

25. The apparatus of claim 18, wherein at least one of the first device and the second device comprises one or more of a video screen, a mirror with an integrated display, a stereo system, a sun roof, a car window, a lock, a car seat, a climate control system, or a dashboard console.

26. The apparatus of claim 18, wherein the first device and the second device comprise an integrated device.

27. A computer program product, comprising a non-transitory computer-readable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for controlling at least a first device and a second device attached to a vehicle based at least in part on a detected direction of observation of an occupant of the vehicle, the method further comprising:

receiving one or more images from at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with the first device and a second portion associated with the second device;

locating, from the one or more images, one or more body features associated with the occupant;

determining a direction of gaze towards the object using the one or more located body features; and

controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output.

28. The computer program product of claim 27, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant.

29. The computer program product of claim 27, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant.

30. The computer program product of claim 27, further comprising determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture.

31. The computer program product of claim 30, wherein the gesture comprises one or more of a hand movement or a voice command.

32. The computer program product of claim 27, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information.

33. The computer program product of claim 27, wherein determining the direction of gaze comprises tracking the occupant'"'"'s head attitude duration and stability.

34. The computer program product of claim 27, wherein the first device and the second device comprise an integrated device.