Method and apparatus for communication between humans and devices

• US 10,296,084 B2
• Filed: 02/10/2017
• Issued: 05/21/2019
• Est. Priority Date: 03/21/2003
• Status: Active Grant

• Current Assignee: Jeffrey S. Shell, Roel Vertegaal
• Original Assignee: Queen's University At Kingston
• Inventors: Vertegaal, Roel, Shell, Jeffrey S.
• Primary Examiner(s): Patel, Sanjiv D.
• Application Number:
  US15/429,733
• Publication Number:
  US 20170371407A1
• Time in Patent Office: 830 Days
• CPC Class Codes: G05B 23/02 Electric testing or monitoring
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units

Abstract

This invention relates to methods and apparatus for improving communications between humans and devices. The invention provides a method of modulating operation of a device, comprising: providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user'"'"'s attentive state is eye contact of the user with the device that is sensed by the attentive user interface.

Claims

1. A notification method for a cellular phone, comprising:

embedding at least one camera and at least one processor in the cellular phone for sensing a user'"'"'s visual attention toward a display screen of the cellular phone;

using the at least one camera to capture at least one image of at least one of the user'"'"'s pupils;

using the at least one processor to process the at least one image from the camera to determine a glint near the at least one pupil;

using the at least one processor to determine an alignment of the glint relative to the at least one pupil and to determine whether or not the user'"'"'s visual attention is directed toward the at least one camera based on said alignment, without calibration;

using the user'"'"'s visual attention directed toward the at least one camera as an indication that the user'"'"'s visual attention is directed toward the display screen; and

managing when or how to present a notification or message initiated by the cellular phone based on whether the user'"'"'s visual attention is directed toward the display screen of the cellular phone.

2. The method of claim 1, wherein using the at least one processor to process the at least one image does not require calibration of the user'"'"'s eye coordinates to x, y display screen coordinates.

3. The method of claim 1, wherein using the at least one processor to process the at least one image comprises:

locating the at least one pupil, a center of the pupil, and the glint in the image;

calculating the distance between the glint and the center of the pupil;

wherein the distance is used to determine whether the pupil center is within a threshold distance of the glint;

wherein when the distance is within the threshold distance is indicative of visual attention of the user toward the display screen of the cellular phone.

4. The method of claim 1, wherein managing when or how to present a notification or message comprises displaying a notification when the user'"'"'s visual attention toward the display screen of the cellular phone is detected.

5. The method of claim 1, wherein managing when or how to present a notification or message comprises displaying content of a message when the user'"'"'s visual attention toward the display screen of the cellular phone is detected.

6. The method of claim 1, wherein managing when or how to present a notification or message comprises preventing displaying a notification when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

7. The method of claim 1, wherein managing when or how to present a notification or message comprises preventing displaying content of a message when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

8. The method of claim 1, wherein managing when or how to present a notification or message comprises lowering volume of an auditory notification or preventing an auditory notification when the user'"'"'s visual attention toward the display screen of the cellular phone is detected.

9. The method of claim 1, wherein managing when or how to present a notification or message comprises raising volume of an auditory notification when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

10. The method of claim 1, wherein managing when or how to present a notification or message comprises selecting an auditory notification in place of a visual notification when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

11. A cellular phone, comprising:

at least one camera embedded in the cellular phone that captures at least one image of at least one of a user'"'"'s pupils;

a display screen; and

at least one processor;

wherein the at least one processor processes the at least one image and determines a glint near the at least one pupil;

wherein the at least one processor determines an alignment of the glint relative to the at least one pupil and determines whether or not the user'"'"'s visual attention is directed toward the at least one camera based on said alignment, without calibration;

wherein the user'"'"'s visual attention directed toward the at least one camera indicates that the user'"'"'s visual attention is directed toward the display screen; and

wherein the at least one processor manages when or how to present a notification or message initiated by the cellular phone based on whether the user'"'"'s visual attention is directed toward the display screen of the cellular phone.

12. The cellular phone of claim 11, wherein the at least one camera is a visible light camera.

13. The cellular phone of claim 11, comprising:

an infrared illuminator embedded in the cellular phone that illuminates a field of view of the at least one camera;

wherein the at least one camera is an infrared camera.

14. The cellular phone of claim 11, wherein the glint in the at least one image includes a reflection of an infrared illuminator on a cornea of an eye of the user;

wherein the reflection is used by the at least one processor to determine a location of the glint relative to the at least one pupil without calibration of the user'"'"'s eye coordinates to x, y display screen coordinates.

15. The cellular phone of claim 14, wherein the at least one processor processes the at least one image by:

locating the pupil, a center of the pupil, and the reflection on the cornea in the image;

calculating a distance between a center of the reflection on the cornea and the center of the pupil;

wherein the distance is used to determine whether the pupil center is within a threshold distance of the reflection on the cornea;

wherein when the distance is within the threshold distance is indicative of the visual attention of the user toward the display screen of the cellular phone.

16. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by displaying a notification when the user'"'"'s visual attention toward the display screen of the cellular phone is detected.

17. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by displaying content of a message when the user'"'"'s visual attention toward the display screen of the cellular phone is detected.

18. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by preventing displaying a notification when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

19. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by preventing displaying content of a message when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

20. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by lowering volume of an auditory notification or preventing an auditory notification when the user'"'"'s visual attention toward the display screen of the cellular phone is detected.

21. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by raising volume of an auditory notification when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.

22. The cellular phone of claim 11, wherein the at least one processor manages when or how to present a notification or message by selecting an auditory notification in place of a visual notification when the user'"'"'s visual attention toward the display screen of the cellular phone is not detected.