Method and system for learning an attention model for an image

• US 7,562,056 B2
• Filed: 10/12/2004
• Issued: 07/14/2009
• Est. Priority Date: 10/12/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Xie, Xing
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Tran; Mai T
• Application Number:
  US10/963,930
• Publication Number:
  US 20060112031A1
• Time in Patent Office: 1,736 Days
• Field of Search: 706/20
• US Class Current: 706/20
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/248 by interactive preprocessing or interactive shape modelling, e.g. feature points assigned by a user
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]

Abstract

A system for learning an attention model for an image based on user navigation actions while viewing the image is provided. An attention model learning system generates an initial attention model based on static features derived from the image. The learning system logs the actions of users as they view the image. The learning system identifies from the actions of the users those areas of the image that may be of user interest: After the learning system identifies areas that may be of user interest, the learning system attempts to identify attention objects based on those identified areas. After identifying the user attention objects, the learning system updates the attention model for the image to reflect the additional information provided by the user attention objects.

Claims

1. A method performed by a computer for learning an attention model for an image, the method comprising:

receiving indications of actions of a user while viewing the image and indications of action focus areas of the image currently being viewed to which the actions relate; and

generating by the computer the attention model based on the actions of the user, the attention model identifying attention objects of the image, an attention object corresponding to an area of the image in which the user has expressed interest based on the actions of the user while viewing the image, the attention objects being identified by

dividing the image into images zones such that the image zones encompass the entire image, each image zone having an interest score indicating interest of the user in the image zone;

for each indication of an action,

for each image zone within the action focus area of the action, updating the interest score for the image zone based on the indicated action;

identifying as initial attention objects those image zones with interest scores indicating the user expressed most interest in those image zones;

expanding the initial attention objects by adding adjacent image zones with an interest score indicating a threshold interest in the adjacent image zones; and

designating an expanded attention object as an attention object for the image.

2. The method of claim 1 including generating an initial attention model based on static features of the image and wherein the generating of the attention model based on the actions of the user includes updating the attention model.

3. The method of claim 1 wherein the indications compose a browse log of actions of the user collected while the user viewed the image.

4. The method of claim 1 wherein the attention model includes attention objects and the generating of the attention model includes:

generating a user interest map of areas of the image of user interest as determined from the actions of the user;

creating attention objects of the image based on the user interest map; and

updating attention objects of the attention model based on the created attention objects.

5. The method of claim 4 wherein the user interest map is generated by identifying image zones of the image in which the actions indicate user interest.

6. The method of claim 5 wherein an image zone has attributes that include a time first viewed, a number of views, a display ratio, and a time of action.

7. The method of claim 5 wherein an image zone has an attribute that represents a number of interesting actions relating to the image zone.

8. The method of claim 5 wherein an image zone has an attribute that represents a display ratio.

9. The method of claim 5 wherein an image zone has an attribute that represents time of action.

10. The method of claim 4 wherein the creating of attention objects includes:

identifying primary attention objects based on viewing duration;

identifying secondary attention objects derived from the primary attention objects that represent a larger attention object; and

modifying the identified primary attention objects based on the identified secondary attention objects.

11. The method of claim 4 wherein the updating of attention objects of the attention model includes:

when a new attention object is similar to an existing attention object, modifying attributes of the existing attention object based on attributes of the new attention object; and

when a new attention object is not similar to an existing attention object, adding the new attention object to the attention model.

12. The method of claim 4 wherein:

the generating of a user interest map includes identifying image zones of the image in which the actions indicate user interest;

the creating of attention objects includes identifying primary attention objects based on viewing duration, identifying secondary attention objects derived from the primary attention objects that represent a larger attention object, and modifying the identified primary attention objects based on the identified secondary attention objects; and

the updating of attention objects of the attention model includes when a new attention object is similar to an existing attention object, modifying attributes of the existing attention object based on attributes of the new attention object, and when a new attention object is not similar to an existing attention object, adding the new attention object to the attention model.

13. A computer-implemented method for identifying user interest in an image, the method comprising:

receiving indications of actions of a user while viewing the image;

dividing the image into image zones;

identifying actions of the user that indicate user interest in the image;

for identified actions,

determining by a computer an image zone of the image associated with the action; and

indicating by the computer that the determined image zone is of user interest by updating an interest score for the image zone; and

identifying attention objects of the image by

identifying by the computer as initial attention objects those image zones with interest scores indicating the user expressed most interest in those image zones;

expanding by the computer the initial attention objects by adding adjacent image zones with an interest score indicating a threshold interest in the adjacent image zones; and

designating an expanded attention object as an attention object for the image.

14. The method of claim 13 wherein when an identified action is associated with multiple image zones, indicating that the multiple image zones are of user interest.

15. The method of claim 13 wherein the indicating that the determined image zone is of user interest includes setting an interest score.

16. The method of claim 15 wherein the interest score is a weighted viewing time.

17. The method of claim 16 wherein the viewing time is weighted based on distance of image zone from focus point of the identified action.

18. The method of claim 13 wherein indicating that the determined image zone is of user interest includes setting a display ratio for the image zone.

19. The method of claim 13 wherein indicating that the determined image zone is of user interest includes setting a number of different viewings of the image zone.

20. The method of claim 13 wherein indicating that the determined image zone is of user interest includes setting a first time of viewing the image zone.

21. A computer-implemented method for identifying attention objects of an image, the method comprising:

receiving indications of actions of a user while viewing the image;

identifying actions of the user that indicate user interest; and

defining areas associated with the identified actions as attention objects of the image by;

generating by a computer a user interest map having image zones and an interest score for each image zone, the interest score for an image zone indicating user interest in the image zone as indicated by the identified actions;

for each image zone of the user interest map with an interest score indicating a primary threshold level of user interest, designating that image zone as an attention object; and

for each attention object that has been designated, when an image zone that is adjacent to that attention object has an interest score indicating a secondary threshold level of user interest, expanding that attention object to include that adjacent image zone.

22. The method of claim 21 wherein user interest is based on time between actions.

23. The method of claim 21 wherein the defining includes selecting areas as attention objects based on time the user spent viewing the areas.

24. The method of claim 23 including setting an attention value of an attention object based on time the user spent viewing the area.

25. The method of claim 21 wherein the defining includes growing an area based on time the user spent viewing adjacent areas.

26. The method of claim 21 wherein the identifying of attention objects is based on actions of multiple users.

27. A computer-readable storage medium containing instructions for controlling a computer system to learn an attention model for an image, by a method comprising:

generating an attention model based on static features of the image, the attention model identifying attention objects of the image that may be of interest to a user;

displaying the image to a plurality of users;

for each of the plurality of users to whom the image is displayed,

receiving indications of actions of that user while viewing the image; and

updating the attention model based on the actions of the users by;

generating a user interest map having image zones and an interest score for each image zone, the interest score for an image zone indicating user interest in the image zone as indicated by the received indications of user actions;

for each image zone of the user interest map with an interest score indicating a primary threshold level of user interest,

when the image zone is within an attention object of the attention model, designating that attention object as a primary attention object; and

when the image zone is not within an attention object of the attention model, designating that image zone as a primary attention object; and

for each primary attention object that has been designated, when an image zone that is adjacent to that primary attention object has an interest score indicating a secondary threshold level of user interest, expanding that attention object to include that adjacent image zone; and

providing the updated attention model for use in displaying the image.

28. The computer-readable storage medium of claim 27 wherein the indications compose a browse log of actions of the user collected while the user viewed the image.

29. The computer-readable storage medium of claim 27 wherein an image zone has attributes that include a time first viewed, a number of views, a display ratio, and a time viewed.

30. The computer-readable storage medium of claim 27 wherein an image zone has an attribute that represents a number of interesting actions relating to the image zone.

31. The computer-readable storage medium of claim 27 wherein an image zone has an attribute that represents a display ratio.

32. The computer-readable storage medium of claim 27 wherein an image zone has an attribute that represents time viewed.

33. The computer-readable storage medium of claim 27 wherein the updating of attention objects of the attention model includes:

when a new attention object is similar to an existing attention object, modifying attributes of the existing attention object based on attributes of the new attention object; and

when a new attention object is not similar to an existing attention object, adding the new attention object to the attention model.