Method and apparatus for communication between humans and devices

• US 8,096,660 B2
• Filed: 07/26/2010
• Issued: 01/17/2012
• Est. Priority Date: 03/21/2003
• Status: Expired due to Fees

• Current Assignee: Queen's University At Kingston
• Original Assignee: Queen's University At Kingston
• Inventors: Vertegaal, Roel, Shell, Jeffrey S.
• Primary Examiner(s): MANUEL, GEORGE C
• Application Number:
  US12/843,399
• Publication Number:
  US 20110043617A1
• Time in Patent Office: 540 Days
• Field of Search: 351/209
• US Class Current: 351/209
• CPC Class Codes: G05B 23/02 Electric testing or monitoring
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units

Abstract

This invention relates to methods and apparatus for improving communications between humans and devices. The invention provides a method of modulating operation of a device, comprising: providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user'"'"'s attentive state is eye contact of the user with the device that is sensed by the attentive user interface.

Claims

1. Apparatus for communication between a user and a device, comprising:

a hardware sensor in or on the device that senses attention of the user specifically toward the device; and

a processor that processes a signal from the hardware sensor and outputs to the device a measure or index of the user'"'"'s attention toward the device; and

wherein operation of the device is modulated on the basis of the measure or index of the user'"'"'s attention toward the device;

wherein the operation that is modulated is initiated by the device and provides a notification and/or information and/or communication to the user based on the user'"'"'s attention toward the device.

2. The apparatus of claim 1, wherein the hardware sensor senses attention of the user by sensing one or more of eye contact, eye movement, eye position, eye gaze direction, voice, body presence, body orientation, head and/or face orientation, user activity, and brain activity/arousal.

3. The apparatus of claim 1, wherein the hardware sensor senses attention of the user by sensing one or more of eye contact, eye gaze direction, body orientation, head and/or face orientation, and user activity.

4. The apparatus of claim 2, wherein the hardware sensor senses eye gaze direction of the user.

5. The apparatus of claim 2, wherein the hardware sensor senses eye contact of the user.

6. The apparatus of claim 5, wherein the hardware sensor:

obtains successive full-frame video fields of alternating bright and dark video images of the pupils; and

subtracts the images between frames to locate the pupils.

7. The apparatus of claim 6, wherein the hardware sensor:

detects a glint in the eyes; and

confirms eye contact when the glint is aligned with the pupils.

8. The apparatus of claim 1, wherein, when the user'"'"'s attention is not directed toward the device, modulating comprises the device notifying the user progressively, from a less interruptive notification to a more interruptive notification.

9. The apparatus of claim 8, wherein the notification is of at least one type selected from the group consisting of auditory, visual, and tactile.

10. The apparatus of claim 1, wherein the hardware sensor is attached to or embedded in the device.

11. The apparatus of claim 1, further comprising the device.

12. The apparatus of claim 1, wherein the device is selected from an electronic device, a machine, and furniture.

13. The apparatus of claim 1, wherein the device is a personal computer.

14. The apparatus of claim 1, wherein the device is a cellular telephone.

15. The apparatus of claim 1, wherein the device is a telephone.

16. The apparatus of claim 1, wherein the device is a personal digital assistant (PDA).

17. The apparatus of claim 1, wherein the device is an appliance.

18. The apparatus of claim 1, wherein operation of the device is modulated by forwarding a notification from the device to another device or a network of devices.

19. The apparatus of claim 1, wherein operation of the device is modulated by modulating a notification being sent to the user by the device.

20. The apparatus of claim 1, wherein operation of the device is modulated by forwarding the measure or index of the user'"'"'s attention to another device or a network of devices.

21. The apparatus of claim 1, wherein the operation that is modulated is selected from notification, communication, information transfer, and a combination thereof.

22. Apparatus for communication between a user and a network of devices, comprising:

a hardware sensor in or on each device of the network of devices, each hardware sensor sensing attention of a user specifically toward its respective device; and

a processor that processes signals from the hardware sensors and outputs a measure or index of the user'"'"'s attention toward the devices;

wherein operation of one or more of the devices is modulated on the basis of the measure or index of the user'"'"'s attention toward the devices;

wherein the operation that is modulated is initiated by at least one of the devices and provides a notification and/or information and/or communication to the user based on the user'"'"'s attention toward the devices.

23. The apparatus of claim 22, wherein the operation that is modulated is selected from notification, communication, information transfer, and a combination thereof.

24. The apparatus of claim 22, wherein modulating comprises routing the notification, communication, information transfer, or combination thereof, to a device to which the user'"'"'s attention is directed.

25. The apparatus of claim 22, wherein modulating further comprises modulating notification of the user progressively, from a less interruptive notification to a more interruptive notification.

26. The apparatus of claim 22, wherein sensing the user'"'"'s attention comprises sensing eye contact of the user with each device.

27. The apparatus of claim 22, wherein each device is independently a personal computer, cellular telephone, a telephone, a personal digital assistant (PDA), or an appliance.

28. The apparatus of claim 22, further comprising one or more of the devices.

29. Apparatus for communication between users and devices, comprising:

a first hardware sensor in or on a first device of a network of devices, the first hardware sensor sensing a first user'"'"'s attention specifically toward the first device;

a second hardware sensor in or on a second device of a network of devices, the second hardware sensor sensing a second user'"'"'s attention specifically toward the second device;

a first proxy associated with the first device, the first proxy communicating to the first user the second user'"'"'s attention toward the second device;

a second proxy associated with the second device, the second proxy communicating to the second user the first user'"'"'s attention toward the first device;

a processor that processes signals from the first and second hardware sensors and relays to the network a measure or index of the first and second users'"'"' attention toward the respective first and second devices;

wherein communication between the first and second devices is modulated on the basis of the measure or index of the first and second users'"'"' attention toward their respective devices and provides a notification to the first and second users.

30. The apparatus of claim 29, wherein communication between the first and second devices is enabled when respective proxies indicate that respective attention of the first and second users are toward their respective devices.

31. The apparatus of claim 29, wherein the network comprises more than two devices.

32. The apparatus of claim 29, wherein each device is independently a personal computer, cellular telephone, a telephone, a personal digital assistant (PDA), or an appliance.

33. The apparatus of claim 29, further comprising one or more of the devices.