Method and system for organizing electronic messages using eye-gaze technology

• US 8,156,186 B2
• Filed: 11/11/2005
• Issued: 04/10/2012
• Est. Priority Date: 11/11/2005
• Status: Active Grant

• Current Assignee: Scenera Technologies LLC (Scenera Research LLC)
• Original Assignee: Scenera Technologies LLC (Scenera Research LLC)
• Inventors: Singh, Mona P.
• Primary Examiner(s): Backer, Firmin
• Assistant Examiner(s): Emdadi, Keyvan
• Application Number:
  US11/164,145
• Publication Number:
  US 20070112916A1
• Time in Patent Office: 2,342 Days
• Field of Search: 709/206
• US Class Current: 709/206
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 41/5093 wherein the managed service relates to messaging or chat services
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

A method for organizing messages received and transmitted between at least two computer systems over a network and a system configured to implement the method are described. In one aspect, the method includes tracking where a user is gazing as the user is composing an outgoing message on a first computer system. During a composition time period in which the user is composing the outgoing message, a relevance rating for each of a plurality of gaze areas is determined. Each gaze area is associated with one of a plurality of messages displayed by the first computer system, and the relevance rating of a gaze area is based on a number of times the user'"'"'s gaze dwells on the displayed message associated with the gaze area after the user'"'"'s gaze is directed away from the displayed message. The method further includes identifying at least one displayed message that is related to the outgoing message based on the relevance ratings of each gaze area, and associating an identifier for each related message with the outgoing message.

Claims

1. A method for organizing messages received and transmitted between at least two computer systems over a network, the method comprising:

tracking where a user of a first computer system is gazing as the user is composing an outgoing message;

determining during a composition time period in which the user is composing the outgoing message a relevance rating for each of a plurality of gaze areas, wherein each gaze area is associated with one of a plurality of messages displayed by the first computer system, the plurality of displayed messages each having been previously composed by the user or by another user exchanging messages with the user, and wherein the relevance rating of a gaze area is based on a number of times the user'"'"'s gaze dwells on the displayed message associated with the gaze area after the user'"'"'s gaze is directed away from the displayed message;

identifying, based on the relevance rating of each of the plurality of gaze areas, at least one displayed message that is related to the outgoing message; and

associating with the outgoing message an identifier for each displayed message that is related to the outgoing message,

wherein at least one of the preceding actions is performed on at least one electronic hardware component.

2. The method according to claim 1 wherein determining the relevance rating of a gaze area includes:

counting a number of times the user visits the gaze area by directing the user'"'"'s gaze to the gaze area; and

calculating a revisit ratio of the gaze area, the revisit ratio comprising the number of visits to the gaze area divided by a total number of visits to all gaze areas.

3. The method according to claim 2 wherein the relevance rating for a gaze area is further based on a dwell time associated with the gaze area, and wherein determining the relevance rating for the gaze area further includes:

calculating a dwell time ratio of the gaze area, the dwell time ratio comprising an aggregate of an amount of time the user'"'"'s gaze dwells on the gaze area divided by the composition time period; and

calculating a weighted sum of the revisit ratio and the dwell time ratio.

4. The method of claim 2 wherein the relevance rating for a gaze area is further based on a dwell time associated with the gaze area, and wherein determining the relevance rating for the gaze area further includes:

calculating a dwell time-to-length ratio of the gaze area, the dwell time-to-length ratio comprising an aggregate of an amount of time the user'"'"'s gaze dwells on the gaze area divided by a total number of characters in the displayed message associated with the gaze area; and

calculating a weighted sum of the revisit ratio and the dwell time-to-length ratio.

5. The method of claim 2 wherein the relevance rating for a gaze area is further based on a set of emoticons present in the displayed message associated with the gaze area.

6. The method of claim 5 wherein determining the relevance rating for the gaze area further includes:

determining in the outgoing message a set of emoticons used;

comparing the set of emoticons in the outgoing message with the set of emoticons in the message associated with the gaze area;

based on the comparison, determining an intersection size of the message associated with the gaze area, wherein the intersection size is based on a number of common emoticons in the outgoing message and in the message associated with the gaze area;

calculating a similarity ratio of the gaze area, the similarity ratio comprising the intersection size of the gaze area divided by an aggregate of intersection sizes of all gaze areas; and

calculating a weighted sum of the revisit ratio and the similarity ratio.

7. The method of claim 2 wherein the relevance rating for a gaze area is further based on an amount of time the user'"'"'s gaze dwells on at least one item included in the displayed message associated with the gaze area and included in the outgoing message, wherein the at least one item comprises either a word, an emoticon, or a phrase.

8. The method of claim 7 wherein determining the relevance rating for the gaze area further includes:

determining an amount of time the user'"'"'s gaze dwells on each of the at least one items in the displayed message associated with the gaze area;

aggregating the amount of time for each of the at least one items;

calculating an item-gaze ratio of the gaze area, the item-gaze ratio comprising the aggregated item time of the gaze area divided by a sum of aggregated item times of all gaze areas; and

calculating a weighted sum of the revisit ratio and the item-gaze ratio.

9. The method of claim 1 wherein associating with the outgoing message an identifier for each displayed message related to the outgoing message includes associating information related to the relevance rating of each related displayed message as metadata of the outgoing message.

10. The method of claim 1 further comprising transmitting the outgoing message with each associated identifier to a second computer system.

11. The method of claim 1 further comprising:

receiving by the first computer system an incoming message from a second computer system, wherein the incoming message includes one or more identifiers of one or more messages;

identifying which, if any, of the displayed messages are related to the incoming message based on the one or more identifiers; and

presenting the identified related messages on a user interface of the first computer system.

12. The method of claim 11 wherein each of the one or more identifiers included in the incoming message is paired with a relevance rating and wherein presenting the identified related messages further includes presenting the respective relevance rating of each identified related message on the user interface of the first computer system.

13. A system comprising:

a messaging module for transmitting and receiving messages to and from other computer systems of a network;

a display device coupled to the messaging module for displaying a plurality of messages that have been previously received and transmitted and for displaying an outgoing message;

a user input device coupled to the messaging module for allowing the user to compose the outgoing message; and

an eye-tracking system coupled to the messaging module and to the display device for tracking where the user is gazing as the user composes the outgoing message,

wherein the messaging module determines, during a composition time period in which the user is composing the outgoing message, a relevance rating for each of a plurality of gaze areas, wherein each gaze area is associated with one of the plurality of displayed messages, the plurality of displayed messages each having been previously composed by the user or by another user exchanging messages with the user, and wherein the relevance rating of a gaze area is based on a number of times the user'"'"'s gaze dwells on the displayed message associated with the gaze area after the user'"'"'s gaze is directed away from the displayed message, identifies, based on the relevance rating of each of the plurality of gaze areas, at least one displayed message that is related to the outgoing message, and associates with the outgoing message an identifier for each identified related displayed message.

14. The system of claim 13 wherein the messaging module is configured to count a number of times the user visits the gaze area by directing the user'"'"'s gaze to the gaze area and calculate a revisit ratio of the gaze area, the revisit ratio comprising the number of visits to the gaze area divided by a total number of visits to all gaze areas.

15. The system of claim 14 wherein the relevance rating for a gaze area is further based on a dwell time associated with the gaze area, and wherein the messaging module is further configured to calculate a dwell time ratio of the gaze area, the dwell time ratio comprising an aggregate of an amount of time the user'"'"'s gaze dwells on the gaze area divided by the composition time period and to calculate a weighted sum of the revisit ratio and the dwell time ratio.

16. The system of claim 14 wherein the relevance rating for a gaze area is further based on a dwell time associated with the gaze area, and wherein the messaging module is further configured to calculate a dwell time-to-length ratio of the gaze area, the dwell time-to-length ratio comprising an aggregate of an amount of time the user'"'"'s gaze dwells on the gaze area divided by a total number of characters in the displayed message associated with the gaze area and to calculate a weighted sum of the revisit ratio and the dwell time-to-length ratio.

17. The system of claim 14 wherein the relevance rating for a gaze area is further based on a set of emoticons present in the displayed message associated with the gaze area.

18. The system of claim 17 wherein the messaging module is further configured to determine in the outgoing message a set of emoticons used, to compare the set of emoticons in the outgoing message with the set of emoticons in the message associated with the gaze area, to determine, based on the comparison, an intersection size of the message associated with the gaze area, wherein the intersection size is based on a number of common emoticons in the outgoing message and in the message associated with the gaze area, to calculate a similarity ratio of the gaze area, the similarity ratio comprising the intersection size of the gaze area divided by an aggregate of intersection sizes of all gaze areas, and to calculate a weighted sum of the revisit ratio and the similarity ratio.

19. The system of claim 14 wherein the relevance rating for a gaze area is further based on an amount of time the user'"'"'s gaze dwells on at least one item included in the displayed message associated with the gaze area and included in the outgoing message, wherein the at least one item comprises either a word, an emoticon, or a phrase.

20. The system of claim 19 wherein the messaging module is further configured to determine an amount of time the user'"'"'s gaze dwells on each of the at least one items in the displayed message associated with the gaze area, to aggregate the amount of time for each of the at least one items, to calculate an item-gaze ratio of the gaze area, the item-gaze ratio comprising the aggregated item time of the gaze area divided by a sum of aggregated item times of all gaze areas, and to calculate a weighted sum of the revisit ratio and the item-gaze ratio.

21. The system of claim 13 wherein the messaging module is configured to associate the identifier and information related to the relevance rating of each identified related displayed message as metadata of the outgoing message.

22. The system of claim 13 wherein the messaging module is configured to transmit the outgoing message with each associated identifier to a second computer system.

23. The system of claim 13 wherein the messaging module is configured to receive an incoming message from a second computer system, wherein the incoming message includes one or more identifiers of one or more messages, to identify which, if any, of the displayed messages are related to the incoming message based on the one or more identifiers, and to present the identified related messages on the display device of the first computer system.

24. The system of claim 23 wherein each of the one or more identifiers included in the incoming message is paired with a relevance rating and wherein the messaging module is configured to present the respective relevancy rating of each identified related message along with the identified related messages on the display device.

25. A non-transitory computer readable medium containing a computer program for organizing messages received and transmitted between at least two computer systems over a network, the computer program comprising executable instructions for:

tracking where a user of a first computer system is gazing as the user is composing an outgoing message;

determining during a composition time period in which the user is composing the outgoing message a relevance rating for each of a plurality of gaze areas, wherein each gaze area is associated with one of a plurality of messages displayed by the first computer system, the plurality of displayed messages each having been previously composed by the user or by another user exchanging messages with the user, and wherein the relevance rating of a gaze area is based on a number of times the user'"'"'s gaze dwells on the displayed message associated with the gaze area after the user'"'"'s gaze is directed away from the displayed message;

identifying, based on the relevance rating of each of the plurality of gaze areas, at least one displayed message that is related to the outgoing message; and

associating with the outgoing message an identifier for each identified displayed message that is related to the outgoing message.

26. The computer readable medium according to claim 25 wherein the executable instructions for determining the relevance rating of a gaze area includes executable instructions for counting a number of times the user visits the gaze area by directing the user'"'"'s gaze to the gaze area, and calculating a revisit ratio of the gaze area, the revisit ratio comprising the number of visits to the gaze area divided by a total number of visits to all gaze areas.

27. The computer readable medium according to claim 26 wherein the relevance rating for a gaze area is further based on a dwell time associated with the gaze area, and wherein the executable instructions for determining the relevance rating for the gaze area further include instructions for:

calculating a dwell time ratio of the gaze area, the dwell time ratio comprising an aggregate of an amount of time the user'"'"'s gaze dwells on the gaze area divided by the composition time period; and

calculating a weighted sum of the revisit ratio and the dwell time ratio.

28. The computer readable medium of claim 26 wherein the relevance rating for a gaze area is further based on a dwell time associated with the gaze area, and wherein the executable instructions for determining the relevance rating for the gaze area further include instructions for:

calculating a dwell time-to-length ratio of the gaze area, the dwell time-to-length ratio comprising an aggregate of an amount of time the user'"'"'s gaze dwells on the gaze area divided by a total number of characters in the displayed message associated with the gaze area; and

calculating a weighted sum of the revisit ratio and the dwell time-to-length ratio.

29. The computer readable medium of claim 26 wherein the relevance rating for a gaze area is further based on a set of emoticons present in the displayed message associated with the gaze area, and wherein the executable instructions for determining the relevance rating for the gaze area further include instructions for:

determining in the outgoing message a set of emoticons used;

comparing the set of emoticons in the outgoing message with the set of emoticons in the message associated with the gaze area;

based on the comparison, determining an intersection size of the message associated with the gaze area, wherein the intersection size is based on a number of common emoticons in the outgoing message and in the message associated with the gaze area;

calculating a similarity ratio of the gaze area, the similarity ratio comprising the intersection size of the gaze area divided by an aggregate of intersection sizes of all gaze areas; and

calculating a weighted sum of the revisit ratio and the similarity ratio.

30. The computer readable medium of claim 26 wherein the relevance rating for a gaze area is further based on an amount of time the user'"'"'s gaze dwells on at least one item included in the displayed message associated with the gaze area and included in the outgoing message, wherein the at least one item comprises either a word, an emoticon, or a phrase, and wherein the executable instructions for determining the relevance rating for the gaze area further include instructions for:

determining an amount of time the user'"'"'s gaze dwells on each of the at least one items in the displayed message associated with the gaze area;

aggregating the amount of time for each of the at least one items;

calculating an item-gaze ratio of the gaze area, the item-gaze ratio comprising the aggregated item time of the gaze area divided by a sum of aggregated item times of all gaze areas; and

calculating a weighted sum of the revisit ratio and the item-gaze ratio.

31. The computer readable medium of claim 25 wherein the executable instructions for associating with the outgoing message the identifier for each identified displayed message includes instructions for associating information related to the relevance rating of each identified displayed message as metadata of the outgoing message.

32. The computer readable medium of claim 25 further comprising executable instructions for transmitting the outgoing message with each associated identifier to a second computer system.

33. The computer readable medium of claim 25 further comprising executable instructions for:

receiving by the first computer system an incoming message from a second computer system, wherein the incoming message includes one or more identifiers of one or more messages;

identifying which, if any, of the displayed messages are related to the incoming message based on the one or more identifiers; and

presenting the identified related messages on a user interface of the first computer system.

34. The computer readable medium of claim 33 wherein each of the one or more identifiers included in the incoming message is paired with a relevance rating and wherein the executable instructions for presenting the identified related messages further includes instructions for presenting the respective relevance rating of each identified related message on the user interface of the first computer system.

35. A system comprising:

means for transmitting and receiving messages to and from other computer systems of a network;

means for displaying a plurality of messages that have been previously received and transmitted and for displaying an outgoing message;

means for allowing the user to compose the outgoing message; and

means for tracking where the user is gazing as the user composes the outgoing message,

wherein the means for transmitting and receiving messages further determines, during a composition time period in which the user is composing the outgoing message, a relevance rating for each of a plurality of gaze areas, wherein each gaze area is associated with one of the plurality of displayed messages, the plurality of displayed messages each having been previously composed by the user or by another user exchanging messages with the user, and wherein the relevance rating of a gaze area is based on a number of times the user'"'"'s gaze dwells on the displayed message associated with the gaze area after the user'"'"'s gaze is directed away from the displayed message, identifies, based on the relevance rating of each of the plurality of gaze areas, at least one displayed message that is related to the outgoing message, and associates with the outgoing message an identifier for each identified displayed message that is related to the outgoing message.