Gaze discriminating electronic control apparatus, system, method and computer program product

• US 7,438,414 B2
• Filed: 05/03/2006
• Issued: 10/21/2008
• Est. Priority Date: 07/28/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Outland Research LLC
• Inventors: Rosenberg, Louis Barry
• Primary Examiner(s): Dang; Hung X
• Application Number:
  US11/381,504
• Publication Number:
  US 20070024579A1
• Time in Patent Office: 902 Days
• Field of Search: 351/209, 351/210, 351/221, 351/246, 351/205, 351/200
• US Class Current: 351/210
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

An electronic control apparatus, system, method and computer program product for selecting and controlling an electronic device from among a plurality of available electronic devices based upon the substantially contemporaneous issuance of a verbal command and device-targeting gaze by a user. The system includes a processor driven controller unit programmed to control a selected device in dependence on generally contemporaneous aural and optical signals. A gaze sensing unit is operatively coupled to the controller unit and is functional to receive the optical signals from a functionally coupled gaze sensor. The optical signals being indicative of a user'"'"'s device-targeting gaze falling upon a device to be controlled. An aural recognition unit is operatively coupled to the controller unit and is functional to receive the aural signals from a functionally coupled aural sensor. The aural signals being indicative of verbal commands appropriate for controlling the selected device and a light source configurable to sufficiently reflect off a pair of eyes of a user and be detected by the gaze sensor.

Claims

1. A gaze-discriminating electronic control apparatus comprising:

a controller unit comprising a processor programmed to control a selected device in dependence on generally contemporaneously received aural and optical signals;

a gaze sensing unit operatively coupled to said controller unit and functional to receive said optical signals from a functionally coupled gaze sensor indicative of a user performing a device-targeting gaze upon said selected device;

an aural recognition unit operatively coupled to said controller unit and functional to receive said aural signals from a functionally coupled aural sensor indicative of a user issuing a verbal command appropriate for controlling said selected device; and

,

a discriminator unit functional to determine if said aural signals and said optical signals are contemporaneous in dependence on at least one gaze determinative variable.

2. The apparatus according to claim 1 further including a proximity detection unit coupled to said controller unit functional to receive permissive activation signals from a functionally coupled proximity sensor disposed in a location to detect a user'"'"'s presence.

3. The apparatus according to claim 1 wherein said controller unit further includes a light source configurable to sufficiently reflect off a pair of eyes of a user and be detected by said gaze sensor.

4. The apparatus according to claim 3 wherein said light source is an infrared light emitting diode.

5. The apparatus according to claim 1 wherein said gaze detection sensor is one of;

a charge coupled device and a complimentary metal-oxide semiconductor.

6. The apparatus according to claim 1 wherein said aural sensor is a microphone.

7. The apparatus according to claim 1 wherein at least said controller unit, said gaze sensing unit and said aural recognition unit are installed in a container affixed to or in proximity to said selected device.

8. The apparatus according to claim 1 wherein said aural recognition unit is configured to recognize aural commands of a specific user.

9. The apparatus according to claim 1 wherein said controller unit includes a gaze timing circuit configured to determine whether said device-targeting gaze has been detected for a sufficient amount of time to allow selection.

10. The apparatus according to claim 1 wherein said controller unit includes a gaze timing circuit for determining whether said device-targeting gaze was detected prior to the inception of said issued verbal command by more than a certain time threshold.

11. A discriminating gaze electronic control system comprising:

a controller unit comprising a processor programmed to control a selected device in dependence on generally contemporaneously received aural and optical signals;

a gaze sensing unit operatively coupled to said controller unit and functional to receive said optical signals from a plurality of functionally coupled gaze sensors indicative of a user performing a device-targeting gaze upon one of a plurality of electronic devices;

an aural recognition unit operatively coupled to said controller unit and functional to receive said aural signals from at least one functionally coupled aural sensor indicative of a user issuing a verbal command appropriate for controlling at least one of a plurality of electronic devices; and

,

a plurality of light sources configurable to sufficiently reflect off a pair of eyes of a user and be detected by at least one of said plurality of gaze sensors.

12. The system according to claim 11 wherein at least a portion of each of said plurality of gaze sensors is disposed in close proximity to one of a plurality of separate devices to be controlled.

13. The system according to claim 12 wherein each of said plurality of gaze sensors is disposed in close proximity to a different one of said plurality of separate devices to be controlled.

14. The system according to claim 11 wherein said gaze sensing unit is operative to determine a characteristic reflection off either a retinal portion or corneal portion of said user'"'"'s eyes when said user is gazing in toward one or more of said plurality of gaze sensors.

15. The system according to claim 11 wherein each of said plurality of light sources emits a distinctive illumination signal assigned to a device to be controlled and distinguishable by said gaze sensing unit.

16. The system according to claim 11 wherein at least said plurality of gaze sensors are functionally coupled to said gaze sensing unit over a communications network.

17. The system according to claim 16 wherein said communications network is one of;

a wired network, a wireless network and an electrical power network.

18. A method for providing a discriminating gaze electronic control apparatus comprising:

providing a controller unit comprising a processor programmed to select and control a device in dependence on generally contemporaneously received aural and optical signals;

providing a gaze sensing unit operatively coupled to said controller unit and functional to receive said optical signals from a functionally coupled gaze sensor indicative of a user performing a device-targeting gaze upon said selected device;

providing an aural recognition unit operatively coupled to said controller unit and functional to receive said aural signals from a functionally coupled aural sensor indicative of a user issuing a verbal command appropriate for controlling said selected device; and

,

providing a discriminator circuit coupled to said processor and functional to determine if said aural signals and optical signals are contemporaneous based upon one or more gaze time-thresholds and/or gaze-synchronization parameters.

19. The method according to claim 18 further comprising providing a light source configurable to sufficiently reflect off a pair of eyes of a user to be detected as a characteristic signal by said gaze sensor only when said user is gazing substantially in a direction of a particular gaze sensor.

20. The method according to claim 18 further comprising disposing at least said controller unit, said gaze sensing unit and said aural recognition unit in a single container.

21. The method according to claim 20 wherein said container is configured to be one of, directly coupled to said device, disposed in close proximity to said device and remotely disposed.

22. The method according to claim 21 wherein said remotely disposed container is configured to be remotely operated by coupling at least said gaze sensor and said aural sensor to a communication network in processing communications with said controller unit.

23. The method according to claim 18 wherein said programmed comprises;

determining a gaze selected target in dependence on said optical signals received from said gaze sensing unit;

determining an appropriate control action in dependence on said aural signals received from said aural recognition unit; and

discriminating against said optical signals or said aural signals which are not generally received contemporaneously.

24. A computer program product embodied in a tangible form executable by a processor associated with a discriminating gaze electronic control apparatus comprising instructions executable by said processor to:

determine a gaze selected target in dependence on optical signals received from a gaze sensing unit;

determine an appropriate control action in dependence on aural signals received from a aural recognition unit;

discriminate against optical signals or aural signals which are not generally received contemporaneously;

else,

send a command to an interface unit associated with a device to be controlled.

25. The computer program product according to claim 24 wherein said appropriate control action is associated with a device to be controlled through an interface unit.

26. The computer program product according to claim 24 wherein said tangible form comprises magnetic media, optical media or logical media.

27. A system for selecting and controlling an electronic device from among a plurality of available electronic devices in dependence on a generally contemporaneous issuance of a verbal command and device-targeting gaze, said system comprising:

a plurality of gaze sensors associated with and proximal to said plurality of available electronic devices and functional to detect whether a user'"'"'s gaze falls proximal to at least one of said plurality of electronic devices in which at least one of said plurality of gaze sensors is associated therewith;

an aural recognition unit functional to interpret a verbal command uttered by a user,

a controller unit coupled to said plurality of gaze sensors and said aural recognition unit and functional to;

evaluate whether said verbal command was uttered in general contemporaneousness with a device-targeting gaze;

determine which of said plurality of available electronic devices said device-targeting gaze was proximal thereto; and

,

control a selected electronic device from said plurality of electronic devices in dependence on whether said uttered verbal command was in general contemporaneousness with said device-targeting gaze.

28. The system according to claim 27 wherein said controller evaluates said general contemporaneousness based at least in part on a gaze time-threshold.

29. The system according to claim 27 wherein said controller unit evaluates said general contemporaneousness based at least in part upon a gaze-synchronization parameter.

30. The system according to claim 27 wherein said controller unit determines whether said device-targeting gaze was proximal to said selected electronic device prior to the inception of said uttered verbal command by more than a certain time threshold.

31. The system according to claim 27 wherein said controller determines whether said device-targeting gaze lingered upon said selected electronic device following completion of said issued verbal command by more than a certain time threshold.

32. The system according to claim 27 further including a plurality of light emitters, each of said light emitters being associated with and proximal to at least one of said plurality of gaze sensors.

33. The system according to claim 32 wherein each of said gaze sensors is configured to detect light emitted by at least one of said light emitters and reflected off a pair of eyes of said user when said user provides said device-targeting gaze.

34. The system according to claim 33 wherein each of said gaze sensors is operative to detect a characteristic reflection off a retinal portion of said user'"'"'s eyes only when said user gazes substantially in a direction of a particular gaze sensor.

35. The system according to claim 33 wherein each of said gaze sensors is operative to detect a characteristic reflection off a corneal portion of said user'"'"'s eyes only when said user gazes substantially in the direction of a particular gaze sensor.

36. The system according to claim 35 wherein said characteristic reflection is such that a pair of corneal reflection is detected and each of said corneal reflections is determined to be substantially centered within a dark pupil region of one of said user'"'"'s eyes.