Method and apparatus for communication between humans and devices

• US 20040183749A1
• Filed: 03/21/2003
• Published: 09/23/2004
• Est. Priority Date: 03/21/2003
• Status: Active Grant

• Current Assignee: Jeffrey S. Shell, Roel Vertegaal
• Original Assignee: Queen's University At Kingston
• Inventors: Vertegaal, Roel
• Granted Patent: US 7,762,665 B2
• Time in Patent Office: Days
• US Class Current: 345/7
• CPC Class Codes: G05B 23/02 Electric testing or monitoring
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units

Abstract

This invention relates to methods and apparatus for improving communications between humans and devices. The invention provides a method of modulating operation of a device, comprising: providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user'"'"'s attentive state is eye contact of the user with the device that is sensed by the attentive user interface.

Claims

1. A method of modulating operation of a device, comprising:

providing an attentive user interface for obtaining information about an attentive state of a user; and

modulating operation of a device on the basis of said obtained information, wherein said operation that is modulated is initiated by said device.

2. The method of claim 1, wherein said information about said user'"'"'s attentive state is eye contact of said user with said device that is sensed by said attentive user interface.

3. The method of claim 1, wherein said information about said user'"'"'s attentive state is eye contact of said user with a subject that is sensed by said attentive user interface.

4. The method of claim 3, wherein said subject is human.

5. The method of claim 3, wherein said subject is another device.

6. The method of claim 4, wherein said information about said user'"'"'s attentive state is eye contact of said user with said human that is sensed by said attentive user interface.

7. The method of claim 2, wherein said sensing of eye contact comprises:

obtaining successive full-frame video fields of alternating bright and dark video images of said user'"'"'s pupils; and

subtracting said images between frames to locate said pupils;

wherein locating said pupils confirms eye contact of said user.

8. The method of claim 7, wherein said sensing of eye contact further comprises:

detecting a glint in the user'"'"'s eyes; and

confirming eye contact of said user when said glint is aligned with said pupils.

9. The method of claim 6, wherein said sensing of eye contact comprises:

obtaining successive full-frame video fields of alternating bright and dark video images of said human'"'"'s pupils; and

subtracting said images between frames to locate said pupils;

wherein locating said pupils confirms eye contact of said human.

10. The method of claim 9, wherein said sensing of eye contact further comprises:

detecting a glint in the user'"'"'s eyes; and

confirming eye contact of said user when said glint is aligned with said pupils.

11. The method of claim 1, wherein, when said user'"'"'s attention is not directed toward said device, said modulating step comprises notifying said user progressively, from a less interruptive notification to a more interruptive notification.

12. The method of claim 11, wherein said notification is of at least one type selected from the group consisting of audio, visual, and tactile.

13. The method of claim 1, wherein, when said user'"'"'s attention is directed toward said other device, said modulating step comprises routing a notification to said other device.

14. The method of claim 1, wherein said information about an attentive state of said user is based on one or more indices selected from the group consisting of eye contact, eye movement, eye position, eye gaze direction, voice, body presence, body orientation, head and/or face orientation, user activity, and brain activity/arousal.

15. The method of claim 1, wherein said information about said user'"'"'s attentive state is eye gaze direction that is sensed by said attentive user interface.

16. The method of claim 1, wherein said attentive user interface is attached to or embedded in said device.

17. The method of claim 1, wherein said attentive user interface is attached to or embedded in a member of the group consisting of clothing, eyewear, jewelry, and furniture.

18. The method of claim 1, wherein said device is a personal computer.

19. The method of claim 1, wherein said device is a cellular telephone.

20. The method of claim 1, wherein said device is a telephone.

21. The method of claim 1, wherein said device is a personal digital assistant (PDA).

22. The method of claim 1, wherein said device is an appliance.

23. The method of claim 1, wherein said modulating step comprises forwarding said obtained information to another device or a network of devices.

24. The method of claim 1, wherein said modulating step comprises modulating a notification being sent to said user.

25. The method of claim 2, wherein said modulating step comprises forwarding said obtained information to another device or a network of devices.

26. The method of claim 4, wherein eye contact of said human subject is detected by said attentive user interface.

27. The method of claim 26, wherein said attentive user interface is associated with eyeglasses worn by said user.

28. A method of modulating operation of a network of devices, comprising:

providing each device of a network of devices with an attentive user interface for obtaining information about an attentive state of a user with respect to each device; and

modulating operation of said devices on the basis of said obtained information, wherein said operation that is modulated is initiated by at least one of said devices.

29. The method of claim 28, wherein said operation that is modulated is selected from the group consisting of notification, communication, information transfer, and a combination thereof.

30. The method of claim 29, wherein said modulating step comprises routing said notification, communication, information transfer, or combination thereof, to a device with which said user is engaged.

31. The method of claim 29, wherein, said modulating operation further comprises modulating notification of said user progressively, from a less interruptive notification to a more interruptive notification.

32. The method of claim 28, wherein said information about said user'"'"'s attentive state is eye contact of said user with each said device, said eye contact being sensed by said attentive user interface.

33. A method of modulating communication over a network of at least two devices, comprising:

providing a first device of a network of devices with an attentive user interface for obtaining information about a first user'"'"'s attentive state toward said first device;

providing a second device of a network of devices with an attentive user interface for obtaining information about a second user'"'"'s attentive state toward said second device;

providing said first device of said network with a proxy for communicating to said first user said information about said second user'"'"'s attentive state toward said second device;

providing said second device of said network with a proxy for communicating to said second user said information about said first user'"'"'s attentive state toward said first device;

relaying to said network said information about said first and second users'"'"' attentive states toward said respective first and second devices;

wherein communication between said first and second devices is modulated on the basis of the attentive states of said first and second users toward their respective devices.

34. The method of claim 33, wherein communication between said first and second devices is enabled when respective proxies indicate that respective attentive states of said first and second users are toward their respective devices.

35. The method of claim 33, wherein said device is a telephone.

36. The method of claim 33, wherein said proxy is a representation of a user'"'"'s eyes.

37. The method of claims 33, wherein said network comprises more than two devices.

38. A method of modulating operation of a cellular telephone, comprising:

providing an attentive user interface for obtaining information about an attentive state of a user; and

modulating operation of a cellular telephone on the basis of said obtained information, wherein said operation that is modulated is initiated by said cellular telephone.

39. The method of claim 38, wherein said information about said user'"'"'s attentive state is eye contact of said user with said cellular telephone that is sensed by said attentive user interface.

40. A method of modulating operation of a graphical user interface, comprising:

providing a graphical user interface for displaying one or more images to a user;

determining said user'"'"'s eye gaze direction to obtain information about which image is being viewed by said user; and

using said information to enlarge, on said graphical user interface, said image being viewed by said user, and to shrink, on said graphical user interface, one or more images not being viewed by said user, wherein said enlarging of an image does not obscure said one or more images not being viewed.

41. Apparatus for detecting eye contact of a subject looking at a user, comprising an eye contact sensor worn by said user that indicates eye contact of a subject looking at the user.

42. The apparatus of claim 41, wherein said apparatus comprises eyeglasses.

43. An eye contact sensor, comprising:

an image sensor for obtaining successive full-frame video fields of alternating bright and dark video images of a user'"'"'s pupils; and

means for subtracting said images between frames to locate said pupils;

wherein said located pupils indicate eye contact of said user.

44. The eye contact sensor of claim 43, further comprising:

means for detecting alignment of a glint in said user'"'"'s eyes with said user'"'"'s pupils;

wherein alignment of said glint with said pupils indicates eye contact of said user.