Computation of intrinsic perceptual saliency in visual environments, and applications

• US 20020154833A1
• Filed: 07/23/2001
• Published: 10/24/2002
• Est. Priority Date: 03/08/2001
• Status: Abandoned Application

• Current Assignee: California Institute of Technology
• Original Assignee: California Institute of Technology
• Inventors: Itti, Laurent, Koch, Christof
• Application Number:
  US09/912,225
• Publication Number:
  US 20020154833A1
• Time in Patent Office: Days
• US Class Current: 382/325
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

Detection of image salience in a visual display of an image. The image is analyzed at multiple spatial scales and over multiple feature channels to determine the likely salience of different portions of the image. One application for the system is in an advertising context. The detection may be improved by second order statistics, e.g. mean and the standard deviations of different image portions relative to other portions. Different edges may be considered as being extended edges by looking at the edges over multiple spatial scales. One set of feature channels can be optimized for use in moving images, and can detect motion or flicker. The images can be obtained over multiple spectral ranges the user can be instructed about how to maximize the saliency. This can be applied to automatically evaluate and optimize sales or advertisement displays.

Claims

1. A method, comprising:

analyzing an image to determine salient parts of an image representation without analyzing the actual content of the image, and

using said salient parts to determine an effectiveness of said image in displaying its content.

2. A method as in claim 1, wherein said analyzing comprises analyzing pixels of said image using mean pixel values.

3. A method as in claim 1, wherein said analyzing comprises analyzing pixels on said image using higher order statistical variations.

4. A method as in claim 1, wherein said image representation includes a single image at a single time.

5. A method as in claim 1, wherein said image representation includes a sequence of images over time.

6. A method as in claim 1, wherein said using comprises evaluating an effectiveness of said image in an advertising context.

7. A method as in claim 1, wherein said using comprises evaluating a display showing one or more items for sale.

8. A method, comprising:

obtaining an electronic file indicative of image content;

forming at least a plurality of feature maps, each feature map representing information about a saliency measure in some area of the image content, said forming comprising detecting differences between a current portion of the image and a surrounding portion of the image using first order, second order or higher order statistics.

9. A method as in claim 8, wherein said second order statistics includes standard deviation.

10. A method as in claim 9, further comprising calculating information indicating a sum of pixels, and second information indicative of a sum of square of pixels at a plurality of different spatial resolution levels.

11. A method as in claim 10, wherein said different spatial resolution levels include different resolution levels within a pyramid scheme.

12. A method as in claim 8 further comprising using both information about mean values and information about standard deviation values.

13. A method as in claim 8, wherein said feature maps include information on intensity.

14. A method as in claim 8, wherein said feature maps include information on color.

15. A method as in claim 8, wherein said feature maps include information about a plurality of different spectral components.

16. A method as in claim 15, further comprising using redundancies between the different spectral components to evaluate said images.

17. A method as in claim 8, wherein said image content includes information about a sequence of moving images.

18. A method as in claim 9, further comprising calculating an image pyramid, where for each of a plurality of different resolutions, said image pyramid stores a sum of all corresponding pixels for current level and lower levels, and a sum of squares of all corresponding pixels for current level and lower levels.

20. A method as in claim 19, wherein said comparing comprises comparing a plurality of different resolution versions of the image to detect said extended contours.

21. A method as in claim 19, wherein said comparing comprises carrying out a nonlinear detection of salient contours.

22. A method as in claim 19, wherein said comparing comprises comparing mean value differences between each part of an image and a surrounding part of an image.

23. A method as in claim 19, wherein said comparing comprises comparing higher order statistical information about each part of an image and a surrounding part of said image.

24. A method as in claim 20, further comprising calculating a plurality of reduced resolution versions of the image at multiple spatial scales, and analyzing said versions of the image.

25. A method as in claim 20, further comprising using edge elements at specified spatial scales to reinforce other edge elements at other spatial scales.

26. A method as in claim 19, wherein said comparing comprises forming a filter mask, saturating said filter mask according to a specified value to form a nonlinearly filtered value, and using said nonlinearly filtered value to detect said contours.

27. A method as in claim 26, further comprising filtering values indicative of said image using a difference of Gaussian filter.

28. A method as in claim 19, wherein said comparing comprises comparing a plurality of different orientation versions of said image to detect said extended contours.

29. A method as in claim 28, wherein said comparing comprises determining a field of influence of contours based on location, and preferred orientation among the contours.

30. A method as in claim 19, wherein said comparing comprises finding interaction among contours across multiple spatial scales.

31. A method as in claim 19, wherein said comparing comprises finding an interaction among contours over a global detection of the image and over a local detection of the image.

19. A method, comprising:

comparing one portion of an image to another portion of an image to detect salient portions of the image, said comparing comprises determining extended contours in the image which are not complete edges, and rating said contours as part of a saliency detection.

32. A method comprising:

analyzing a sequence of temporally changing images, using an automated computer program; and

automatically finding salient portions in said images, based on said analyzing.

33. A method as in claim 32, wherein said automatically finding comprises extracting motion in said images, and using said motion as a feature channel to detect said salient portions.

34. A method as in claim 32, wherein said extracting motion comprises applying three-dimensional spatio-temporal filters to a sequence of images, and using said filters to detect motion having specified characteristics.

35. A method as in claim 32, wherein said applying comprises applying a plurality of spatio-temporal three-dimensional filters, and wherein each of said three-dimensional filters detects specified motion at a specified speed in a specified direction, and each of said filters detects said different speeds and different directions.

36. A method as in claim 35, wherein said filters detect motion across luminance.

37. A method as in claim 35, wherein said filters detect motion across chrominance.

38. A method as in claim 32, further comprising computing an absolute value of a temporal derivative of image intensity, and detecting a change in said image intensity over time greater than a predetermined amount, to detect flicker in the image or a portion thereof.

39. A method as in claim 38, wherein said computing comprises detecting an absolute value of temporal derivatives of color channels that are greater than a predetermined threshold.

40. A method as in claim 38, wherein said computing comprises detecting an absolute value of temporal derivatives of luminance channels that are greater than a predetermined threshold.

41. A method as in claim 32, further comprising using said automatically finding to evaluate an advertisement.

48. A method as in claim 35, further comprising forming a composite map from outputs of said plurality of filters.

49. A method as in claim 35, wherein said filters operate nonlinearly.

50. A method as in claim 39, wherein said operate nonlinearly comprises defining a maximum value and a minimum value.

51. A method as in claim 32, wherein said automatically finding comprises detecting flicker in portions of the image.

52. A method as in claim 51, wherein said detecting flicker comprises detecting flicker only in a portion of the image, but not in the entire image.

53. A method as in claim 32, further comprising using said automatically finding to optimize a display of visual information.

42. A method, comprising:

analyzing an image to determine salient parts of the image representation by obtaining information about the image in at least two different spectral ranges; and

correlating said information about the image to determine salient portions of the image, without looking for specific content of the image.

43. A method as in claim 42, wherein said analyzing comprises using said salient portions to determine an effectiveness of said image in displaying a product.

44. A method as in claim 43, wherein said analyzing comprises analyzing pixels of said image using mean pixel values.

45. A method as in claim 43, wherein said analyzing comprises analyzing pixels on said image using second higher order statistical variations.

46. A method as in claim 43, wherein said image representation is a single image at a single time.

47. A method as in claim 43, wherein said image representation is a sequence of images in time representing a moving scene.

54. A method as in claim 42, further comprising using said analying to optimize a display of visual information.

55. A method, comprising:

analyzing an image representing a display of visual information to determine salient parts of the image representation; and

automatically increasing a salience of a specified part of the image.

56. A method as in claim 55 wherein said automatically increasing comprises systematically changing a value of a parameter and determining the effect of said parameter on said salience.

57. A method as in claim 55, wherein said automatically increasing comprises determining rules for salience increase, and using said rules to increase a salience of the specified part of the image.

58. A method as in claim 55, wherein said display of visual information is an advertisement.

59. A method as in claim 55, wherein said automatically increasing comprises changing a shape of the specified part.