Method and apparatus for communication between humans and devices

• US 8,292,433 B2
• Filed: 09/19/2005
• Issued: 10/23/2012
• Est. Priority Date: 03/21/2003
• Status: Expired due to Fees

• Current Assignee: Queen's University At Kingston
• Original Assignee: Queen's University At Kingston
• Inventors: Vertegaal, Roel
• Primary Examiner(s): GREECE, JAMES R
• Application Number:
  US11/228,468
• Publication Number:
  US 20060093998A1
• Time in Patent Office: 2,591 Days
• Field of Search: 351209-210, 351/246, 382/103
• US Class Current: 351/209
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

This invention relates to methods and apparatus for improving communications between humans and devices. The invention provides a method of modulating operation of a device, comprising providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user'"'"'s attentive state is eye contact of the user with the device that is sensed by the attentive user interface.

Claims

1. A method for determining a subject'"'"'s response and/or interest toward an object, surface, or visual material, comprising:

using a hardware sensor coupled to the object, surface, or visual material, the hardware sensor sensing at least one index of the subject'"'"'s attention specifically toward the hardware sensor together with the object, surface, or visual material; and

using a processor to output a result indicating the subject'"'"'s response and/or interest in the object, surface, or visual material based on the sensed at least one index of the subject'"'"'s attention;

wherein sensing does not elicit a change in the at least one index of the subject'"'"'s attention;

wherein the at least one index is eye contact or head and/or face orientation;

wherein sensing includes;

(i) obtaining alternating odd and even video frames of at least one of the subjects'"'"' pupils including determining whether a frame was illuminated by on-axis or off-axis illumination and whether a pixel is classified as non-zero or zero;

(ii) subtracting the alternating odd and even video frames to produce a subtracted frame;

(iii) detecting at least one pupil and a glint in the subtracted frame; and

(iv) determining alignment of the at least one pupil and the glint;

wherein alignment of the at least one pupil and the glint confirms eye contact of the subject and is related to the subject'"'"'s response and/or interest toward the object, surface, or visual material.

2. A method of monitoring the subject'"'"'s response and/or interest toward the object, surface, or visual material, comprising performing the method of claim 1 two or more times.

3. The method of claim 1, further comprising using the result for marketing information about the object, surface, or visual material.

4. The method of claim 1, wherein the object, surface, or visual material comprises an advertisement, and the method further comprises using the result for assessing the effectiveness of the advertisement.

5. The method of claim 1, further comprising:

providing a plurality of hardware sensors, each hardware sensor being coupled to a region of the object, surface, or visual material, wherein hardware sensors simultaneously in the subject'"'"'s field of view are synchronized, a first hardware sensor sensing the at least one index of the subject'"'"'s attention specifically toward the first hardware sensor together with its region of the object, surface, or visual material;

relating, using a processor, the at least one sensed index to the subject'"'"'s response and/or interest toward the first hardware sensor'"'"'s region of the object, surface, or visual material; and

outputting a result corresponding to the subject'"'"'s response and/or interest toward the first hardware sensor'"'"'s region of the object, surface, or visual material.

6. The method of claim 5, further comprising interpolating results corresponding to the at least one index of the subject'"'"'s attention sensed by two or more hardware sensors, wherein interpolating indicates that the region of the object, surface, or visual material that is of interest to the subject is between the regions of the two or more hardware sensors.

7. The method of claim 1, wherein the object, surface, or visual material is selected from a sign, a computer screen, a television screen, a whiteboard, a chalkboard, art, a window, an interactive surface, a billboard, an advertisement, and a static surface.

8. The method of claim 1, wherein presence of pupil candidates in a frame indicates on-axis illumination.

9. The method of claim 1, further comprising synchronizing the subtraction and the on-axis and off-axis illumination with a video camera clock.

10. The method of claim 1, wherein subtraction comprises subtracting an even frame from an odd frame such that the result is not negative.

11. The method of claim 1, wherein detecting a pupil comprises:

determining a threshold intensity of the image, and

eliminating pixels with intensity below the threshold intensity;

wherein a pupil is detected from remaining pixels.

12. The method of claim 1, further comprising detecting glint in a region of interest based on pupil size.

13. The method of claim 1, wherein determining alignment of the pupil and the glint comprises calculating a distance between pupil center and glint center, and comparing the distance to a threshold, wherein, when the distance is below the threshold, eye contact is detected.

14. An apparatus for determining a subject'"'"'s response and/or interest toward an object, surface, or visual material, comprising:

a hardware sensor coupled to the object, surface, or visual material, the hardware sensor sensing at least one index of the subject'"'"'s attention specifically toward the hardware sensor together with the object, surface, or visual material; and

a processor that outputs a result indicating the subject'"'"'s response and/or interest in the object, surface, or visual material based on the sensed at least one index of the subject'"'"'s attention;

wherein sensing does not elicit a change in the at least one index of the subject'"'"'s attention;

wherein the at least one index of the subject'"'"'s attention is eye contact or head and/or face orientation;

wherein the hardware sensor comprises;

(i) a digital imaging source for obtaining alternating odd and even video frames of at least one of the subject'"'"'s pupils;

(ii) a subtractor that subtracts the alternating odd and even video frames to produce a subtracted frame;

(iii) a detector that detects the at least one pupil and a glint in the subtracted frame; and

(iv) a processor that determines alignment of the at least one pupil and the glint and whether a frame was illuminated by on-axis or off-axis illumination including determining whether a pixel is classified as non-zero or zero;

wherein an odd or even video frame was illuminated by on-axis or off-axis illumination;

wherein alignment of the at least one pupil and the glint confirms eye contact of the subject and is related to the subject'"'"'s response and/or interest toward the object, surface, or visual material.

15. The apparatus of claim 14, further comprising:

a plurality of hardware sensors, each hardware sensor being coupled to a region of the object, surface, or visual material, wherein hardware sensors simultaneously in the subject'"'"'s field of view are synchronized, a first hardware sensor for sensing at least one index of the subject'"'"'s attention toward the first hardware sensor together with its region of the object, surface, or visual material;

a processor that relates the at least one sensed index corresponding to the first hardware sensor to the response and/or interest of the subject toward the first hardware sensor'"'"'s region of the object, surface, or visual material and outputs a result corresponding to the subject'"'"'s response and/or interest toward the first hardware sensor'"'"'s region of the object, surface, or visual material.

16. The apparatus of claim 15, wherein the processor interpolates results corresponding to the at least one index of the subject'"'"'s attention sensed by two or more hardware sensors, wherein interpolating indicates that the region of the object, surface, or visual material that is of interest to the subject is between the regions associated with the two or more hardware sensors.

17. The apparatus of claim 14, wherein the object, surface, or visual material is selected from a sign, a computer screen, a television screen, a whiteboard, a chalkboard, art, a window, an interactive surface, a billboard, an advertisement, and a static surface.

18. The apparatus of claim 14, wherein the subtraction and the on-axis and off-axis illumination are synchronized with a digital imaging source clock.

19. The apparatus of claim 14, wherein the subtractor performs subtraction of even frames from odd frames such that the result is not negative.

20. The apparatus of claim 14, wherein the detector determines a threshold intensity of the image, eliminates pixels with intensity below the threshold intensity;

and detects a pupil from remaining pixels.

21. The apparatus of claim 14, wherein the detector determines a region of interest to detect glint based on the size of the pupil, determines a threshold intensity within the region of interest, eliminates pixels with intensity below the threshold intensity;

and detects glint from remaining pixels.

22. The apparatus of claim 14, wherein the processor determines alignment of the pupil and the glint by calculating a distance between pupil center and glint center, and comparing the distance to a threshold, wherein, when the distance is below the threshold, eye contact is detected.