Systems and methods for generating a comprehensive user attention model

• US 20040088726A1
• Filed: 11/01/2002
• Published: 05/06/2004
• Est. Priority Date: 11/01/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Lu, Lie, Ma, Yu-Fei
• Granted Patent: US 7,274,741 B2
• Time in Patent Office: Days
• US Class Current: 725/46
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04N 21/8453 by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/854 Content authoring

Abstract

Systems and methods to generate an attention model for computational analysis of video data are described. In one aspect, feature components from a video data sequence are extracted. Attention data is generated by applying multiple attention models to the extracted feature components. The generated attention data is integrated into a comprehensive user attention model for the computational analysis of the video data sequence.

Claims

1. A method for generating a comprehensive user attention model, the method comprising:

extracting feature components from a video data sequence;

generating attention data based on application of multiple attention models to the feature components; and

integrating the attention data to create the comprehensive user attention model.

2. A method as recited in claim 1, wherein the feature components comprise image sequence, audio, and textual components.

3. A method as recited in claim 1, wherein the multiple attention models comprise a combination of visual, audio, and/or linguistic attention models.

4. A method as recited in claim 1, wherein the multiple attention models comprise a combination of static and dynamic attention models.

5. A method as recited in claim 1, wherein the multiple attention models comprise motion, static, face, and/or camera attention models.

6. A method as recited in claim 1, wherein the multiple attention models comprise saliency, speech, and/or music attention models.

7. A method as recited in claim 1, wherein the multiple attention models comprise closed caption, and/or automated speech recognition attention models.

8. A method as recited in claim 1, wherein integrating the attention data further comprises integrating the attention data via linear combination.

9. A method as recited in claim 1, wherein the comprehensive user attention model is represented as:

10. A method as recited in claim 9, wherein {overscore (M_v)}, {overscore (M_a)}, and {overscore (M_l)} are defined as follows:

11. A method as recited in claim 10, wherein the multiple criteria comprise:

if S_cm>

=1, the magnifier is turned on;

if S_cm=0, the magnifier is turned off; and

wherein a large S_cm value indicates a more powerful magnifier than a low S_cm value.

12. A method as recited in claim 1, wherein the multiple attention models comprise a camera attention model and one or more other visual attention models, and wherein generating the attention data further comprises multiplying a sum of the one or more other visual attention models by quantized factors to determine emphasis of the camera attention model with respect to the other visual attention model(s), the quantized factors being camera attention factors.

13. A method as recited in claim 12, wherein quantized factor values range from zero (0) to two (2).

14. A computer-readable medium comprising computer-executable instructions executable by a processor to generate a comprehensive user attention model, the computer-executable instructions comprising instructions for:

extracting feature components from a video data sequence;

generating attention data based on application of multiple attention models to the feature components; and

integrating the attention data to create the comprehensive user attention model.

15. A computer-readable medium comprising computer-executable instructions executable by a processor to generate an attention model, the computer-executable instructions comprising instructions for:

extracting feature components from a video data sequence;

generating attention data based on application of at least visual and audio attention models to the feature components; and

linearly combining the attention data to generate a generic user attention model that integrates results of the multiple visual, audio, and linguistic attention models.

16. A computer-readable medium as recited in claim 15, wherein generating attention data is further based on application of a linguistic attention model to one or more portions of the feature components.

17. A computer-readable medium as recited in claim 15, wherein the feature components comprise image sequence, audio, and textual components.

18. A computer-readable medium as recited in claim 15, wherein the audio attention models comprise saliency, speech, and/or music attention models.

19. A computer-readable medium as recited in claim 15, wherein the linguistic attention models comprise closed caption, and/or automated speech recognition attention models.

20. A computer-readable medium as recited in claim 15, wherein the generic user attention model is represented as:

21. A computer-readable medium as recited in claim 20, wherein {overscore (M_v)}, {overscore (M_a)}, and {overscore (M_l)} are defined as follows:

22. A computer-readable medium as recited in claim 21, wherein the multiple criteria comprise:

if S_cm>

=1, the magnifier is turned on;

if S_cm=0, the magnifier is turned off; and

wherein a large S_cm value indicates a more powerful magnifier than a low S_cm value.

23. A computer-readable medium as recited in claim 15, wherein the visual attention models comprise motion, static, face, and/or camera attention models.

24. A computer-readable medium as recited in claim 23, wherein the camera attention model is based at least in part on the following criteria:

during camera zooming operations, frame importance increases temporally and is a function of zooming speed such that a first frame generated during a fast zooming operation is of higher relative importance that a second frame generated during a slower zooming operation; and

during camera panning operations, frame importance is an inverse of panning speed and a function of panning direction.

25. A computer-readable medium as recited in claim 24, wherein frames generated during a horizontal camera panning operation are calculated to be of lesser relative importance as compared to frames generated during a vertical panning operation.

26. A computer-readable medium as recited in claim 24, wherein calculated importance of a frame generated during panning or zooming operations is reduced from a higher importance to a lower importance as a function of ending the panning or zooming operation and passage of a certain period of time.

27. A computing device for creating a comprehensive user attention model, the computing device comprising:

a processer;

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

generating visual, audio, and linguistic attention data based on application of multiple attention models to a plurality of video data sequence feature components, the feature components comprising image sequence, audio, and text-related features;

integrating the visual, audio, and linguistic attention data to create the comprehensive user attention model; and

wherein the comprehensive user attention model is a computational representation of elements of the video data sequence that attract user attention.

28. A computing device as recited in claim 27, wherein the computer-program instructions for generating further comprise instructions for creating the visual attention data with motion, static, face, and/or camera attention models.

29. A computing device as recited in claim 27, wherein the computer-program instructions for generating further comprise instructions for creating the audio attention data with saliency, speech, and/or music attention models.

30. A computing device as recited in claim 27, wherein the computer-program instructions for generating further comprise instructions for creating the linguistic attention data with closed caption, and/or automated speech recognition attention models.

31. A computing device as recited in claim 27, wherein the computational representation is defined as:

32. A computing device as recited in claim 31, wherein {overscore (M_v)}, {overscore (M_a)}, and {overscore (M_l)} are defined as follows:

33. A computing device as recited in claim 32, wherein the multiple criteria comprise:

if S_cm>

=1, the magnifier is open;

if S_cm=0, the magnifier is closed; and

wherein a large S_cm value indicates a more powerful magnifier than a low S_cm value.

34. A computing device as recited in claim 27, wherein the multiple attention models comprise a camera attention model, and wherein the computer-program instructions for generating the visual attention data generate camera attention data based at least in part on the following criteria:

during camera zooming operations, frame importance increases temporally and is a function of zooming speed such that a first frame generated during a fast zooming operation is of higher relative importance that a second frame generated during a slower zooming operation; and

during camera panning operations, frame importance is an inverse of panning speed and a function of panning direction.

35. A computing device as recited in claim 34, wherein frames generated during a horizontal camera panning operation are calculated to be of lesser relative importance as compared to frames generated during a vertical panning operation.

36. A computing device as recited in claim 34, wherein calculated importance of a frame generated during panning or zooming operations is reduced from a higher importance to a lower importance as a function of ending the panning or zooming operation and passage of a certain period of time.

37. A computing device for generating an attention model, the computing device comprising:

means for extracting image sequence, audio, and text-related feature components from a video data sequence;

means for generating attention data by applying visual, audio, and linguistic attention models to the feature components; and

means for integrating the attention data via into a comprehensive user attention model.

38. A computing device as recited in claim 37, wherein the comprehensive user attention model is a computational representation of elements of the video data sequence that are substantially likely to attract user attention.

39. A computing device as recited in claim 37, wherein the visual attention models comprise motion, static, face, and/or camera attention models.

40. A computing device as recited in claim 37, wherein the audio attention models comprise saliency, speech, and/or music attention models.

41. A computing device as recited in claim 37, wherein the linguistic attention models comprise closed caption, and/or automated speech recognition attention models.