Method and apparatus for communication between humans and devices

• US 20060093998A1
• Filed: 09/19/2005
• Published: 05/04/2006
• Est. Priority Date: 03/21/2003
• Status: Active Grant

• Current Assignee: Queen's University At Kingston
• Original Assignee: Queen's University At Kingston
• Inventors: Vertegaal, Roel
• Granted Patent: US 8,292,433 B2
• Time in Patent Office: Days
• US Class Current: 434/236
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

This invention relates to methods and apparatus for improving communications between humans and devices. The invention provides a method of modulating operation of a device, comprising providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user'"'"'s attentive state is eye contact of the user with the device that is sensed by the attentive user interface.

Claims

1-46. -46. (canceled)

47. A method of obtaining information about a subject'"'"'s response and/or interest toward an object, surface, or visual material, comprising:

providing an attentive user interface in association with the object, surface, or visual material, the attentive user interface (i) sensing at least one index of the subject'"'"'s attentive state toward the object, surface, or visual material, and (ii) outputting information corresponding to the sensed index; and

relating the information to the subject'"'"'s response and/or interest toward the object, surface, or visual material.

48. The method of claim 47, further comprising monitoring the information over time to track the subject'"'"'s response and/or interest toward the object, surface, or visual material.

49. The method of claim 47, further comprising using the information for marketing information about the object, surface, or visual material.

50. The method of claim 47, wherein the object, surface, or visual material comprises an advertisement, and the method further comprises using the information for assessing the effectiveness of the advertisement.

51. The method of claim 47, wherein at least one index is selected from eye contact, eye movement, eye position, eye gaze direction, and head and/or face orientation.

52. The method of claim 47, wherein at least one index is selected from voice, body presence, body orientation, activity, and brain activity and/or arousal.

53. The method of claim 47, wherein at least one index is eye contact.

54. The method of claim 47, further comprising:

providing a plurality of attentive user interfaces, each attentive user interface being associated with a region of the object, surface, or visual material, wherein attentive user interfaces simultaneously in the subject'"'"'s field of view are synchronized, a first attentive user interface (i) sensing one or more indices of the subject'"'"'s attentive state toward the object, surface, or visual material, and (ii) outputting information corresponding to the sensed index; and

relating the information to the subject'"'"'s response and/or interest toward the first attentive user interface'"'"'s associated region of the object, surface, or visual material.

55. The method of claim 54, further comprising interpolating information corresponding to indices of the subject'"'"'s attentive state sensed by two or more attentive user interfaces, wherein interpolating indicates that the region the object, surface, or visual material of interest to the subject is between the regions associated with the two or more attentive user interfaces.

56. The method of claim 47, wherein the object, surface, or visual material is selected from a sign, a computer screen, a television screen, a whiteboard, a chalkboard, art, a window, an interactive surface, a billboard, an advertisement, and a static surface.

57. The method of claim 53, further comprising:

(i) obtaining alternating odd and even video frames of the subjects'"'"' pupils;

(ii) subjecting the odd and even video frames to rolling subtraction to produce a subtracted frame;

(iii) detecting the pupils and a glint in the subtracted frame; and

(iv) determining alignment of the pupil and the glint;

wherein alignment of the pupil and the glint confirms eye contact of the user.

58. The method of claim 57, wherein obtaining an odd or even video frame comprises determining whether a frame was illuminated by on-axis or off-axis illumination.

59. The method of claim 57, wherein presence of pupil candidates in a frame indicates on-axis illumination.

60. The method of claim 57, further comprising synchronizing the rolling subtraction and the on-axis and off-axis illumination with a video camera clock.

61. The method of claim 58, wherein determining whether a frame was illuminated by on-axis or off-axis illumination comprises determining whether a pixel is classified as non-zero or zero, respectively.

62. The method of claim 57, wherein rolling subtraction comprises subtracting an even frame from an odd frame such that the result is not negative.

63. The method of claim 57, wherein detecting pupils comprises:

determining a threshold intensity of the image, and

eliminating pixels with intensity below the threshold intensity;

wherein pupils are detected from remaining pixels.

64. The method of claim 57, further comprising detecting glint in a region of interest based on pupil size.

65. The method of claim 57, wherein determining alignment of the pupil and the glint comprises calculating a distance between pupil center and glint center, and comparing the distance to a threshold, wherein, when the distance is below the threshold, eye contact is detected.

66. An apparatus for obtaining information about a subject'"'"'s response and/or interest towards an object, surface, or visual material, comprising:

an attentive user interface associated with the object, surface, or visual material, the attentive user interface comprising (i) means for sensing at least one index of the subject'"'"'s attentive state toward the object, surface, or visual material, and (ii) means for outputting information corresponding to the sensed index; and

means for relating the information to the subject'"'"'s response and/or interest towards the object, surface, or visual material.

67. The method of claim 66, wherein at least one index is selected from eye contact, eye movement, eye position, eye gaze direction, and head and/or face orientation.

68. The apparatus of claim 66, wherein the at least one index is eye contact.

69. The apparatus of claim 66, further comprising:

a plurality of attentive user interfaces, each attentive user interface being associated with a region of the object, surface, or visual material, wherein attentive user interfaces simultaneously in the subject'"'"'s field of view are synchronized; and

means for relating the information corresponding to a first attentive user interface to the response and/or interest of the subject towards the first attentive user interface'"'"'s associated region of the object, surface, or visual material.

70. The apparatus of claim 69, further comprising means for interpolating information corresponding to indices of the subject'"'"'s attentive state sensed by two or more attentive user interfaces, wherein interpolating indicates that the region of the object, surface, or visual material of interest to the subject is between the regions associated with the two or more attentive user interfaces.

71. The apparatus of claim 66, wherein the object, surface, or visual material is selected from a sign, a computer screen, a television screen, a whiteboard, a chalkboard, art, a window, an interactive surface, a billboard, an advertisement, and a static surface.

72. The apparatus of claim 68, wherein the attentive user interface comprises:

(i) a digital imaging source for obtaining alternating odd and even video frames of the subjects'"'"' pupils;

(ii) a subtractor for rolling subtraction of the odd and even video frames to produce a subtracted frame;

(iii) a detector for detecting the pupils and a glint in the subtracted frame; and

(iv) a processor for determining alignment of the pupil and the glint;

wherein alignment of the pupil and the glint confirms eye contact of the user.

73. The apparatus of claim 72, wherein the rolling subtraction and the on-axis and off-axis illumination are synchronized with a digital imaging source clock.

74. The apparatus of claim 72, wherein the subtractor performs rolling subtraction of even frames from odd frames such that the result is not negative.

75. The apparatus of claim 72, wherein the detector determines a threshold intensity of the image, eliminates pixels with intensity below the threshold intensity;

and detects pupils from remaining pixels.

76. The apparatus of claim 72, wherein the detector determines a region of interest to detect glint based on the size of the pupil, determines a threshold intensity within the region of interest, eliminates pixels with intensity below the threshold intensity;

and detects glint from remaining pixels.

77. The apparatus of claim 72, wherein the processor determines alignment of the pupil and the glint by calculating a distance between pupil center and glint center, and comparing the distance to a threshold, wherein, when the distance is below the threshold, eye contact is detected.