Systems and methods for generating a comprehensive user attention model

• US 7,274,741 B2
• Filed: 11/01/2002
• Issued: 09/25/2007
• Est. Priority Date: 11/01/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Lu, Lie, Ma, Yu-Fei
• Primary Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US10/286,053
• Publication Number:
  US 20040088726A1
• Time in Patent Office: 1,789 Days
• Field of Search: 375/240.16, 375/240.08, 375/240.09, 345/629, 345/419, 382/190, 382/156, 382/203, 707/1
• US Class Current: 375/240.08
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04N 21/8453 by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/854 Content authoring

Abstract

Systems and methods to generate an attention model for computational analysis of video data are described. In one aspect, feature components from a video data sequence are extracted. Attention data is generated by applying multiple attention models to the extracted feature components. The generated attention data is integrated into a comprehensive user attention model for the computational analysis of the video data sequence.

Claims

1. A computer-implemented method for generating a comprehensive user attention model, the method comprising

extracting feature components from a video data sequence:

generating attention data based on application of multiple attention models to the feature components;

integrating the attention data to create the comprehensive user attention model; and

wherein the comprehensive user attention model is represented as;

A=w_v·

M_v+w_a·

M_a+w_l·

M_l,

w_v, w_a, w_l representing weights for linear combination, and M_v, M_a, and M_l indicating normalized visual, audio, and linguistic attention models.

2. The method of claim 1, wherein the feature components comprise image sequence, audio, and textual components.

3. The method of claim 1, wherein the multiple attention models comprise a combination of visual, audio, and/or linguistic attention models.

4. The method of claim 1, wherein the multiple attention models comprise a combination of static and dynamic attention models.

5. The method of claim 1, wherein the multiple attention models comprise motion, static, face, and/or camera attention models.

6. The method of claim 1, wherein the multiple attention models comprise saliency, speech, and/or music attention models.

7. The method of claim 1, wherein the multiple attention models comprise closed caption, and/or automated speech recognition attention models.

8. The method of claim 1, wherein integrating the attention data ifirther comprises integrating the attention data via linear combination.

9. The method of claim 1, wherein M_l, M_a, and M_l are defined as follows:

$\begin{array}{c}{M}_v=\left(\underset{i=1}{\overset{p}{\sum <br><br>}}<br><br>w_i·<br><br>\stackrel{\_}{M_i}\right)\times <br><br>{\left({\stackrel{\_}{M}}_{c<br><br><br><br>m}\right)}^{S_{c<br><br><br><br>m}};<br><br>\\ M_a=\left(\underset{j=1}{\overset{q}{\sum <br><br>}}<br><br>w_j·<br><br>\stackrel{\_}{M_j}\right)\times <br><br>{\left({\stackrel{\_}{M}}_{\mathrm{as}}\right)}^{S_{\mathrm{as}}};<br><br>\\ M_l=\underset{k=1}{\overset{r}{\sum <br><br>}}<br><br>w_k·<br><br>\stackrel{\_}{M_k}<br><br>;<br><br>\mathrm{and}\end{array}$

wherein w_i, w_j, w_k are weights in visual, audio, and linguistic attention models respectively, wherein M_cm comprises a normalized camera attention used as a visual attention model magnifier, and wherein S_cm comprises a magnifier switch that is based on multiple criteria.

10. The method of claim 9, wherein the multiple criteria comprise:

if S_cm>

=1, the magnifier is turned on;

if S_cm=0, the magnifier is turned off; and

wherein a large S_cm value indicates a more powerful magnifier than a low S_cm value.

11. A computer-implemented method for generating a comprehensive user attention model, the method comprising

extracting feature components from a video data sequence;

generating attention data based on application of multiple attention models to the feature components;

integrating the attention data to create the comprehensive user attention model; and

wherein the multiple attention models comprise a camera attention model and one or more other visual attention models, and wherein generating the attention data further comprises multiplying a sum of the one or more other visual attention models by quantized factors to determine emphasis of the camera attention model with respect to the other visual attention model(s), the quantized factors being camera attention factors.

12. The method of claim claim 11, wherein quantized factor values range from zero (0) to two (2).

13. A tangible computer-readable medium storing computer-executable instructions executable by a processor to generate an attention model, the computer-executable instructions comprising instructions for:

extracting feature components from a video data sequence;

generating attention data based on application of at least visual and audio attention models to the feature components;

linearly combining the attention data to generate a generic user attention model that integrates results of the multiple visual, audio, and linguistic attention models; and

wherein the generic user attention model is represented as;

A=w_v·

M_v+w_a·

M_a+w_l·

M_l,

w_v, w_a, w_l representing weights for linear combination, and wherein M_v, M_a, and M_l represent normalized visual, audio, and linguistic attention models.

14. The computer-readable medium of claim 13, wherein generating attention data is further based on application of a linguistic attention model to one or more portions of the feature components.

15. The computer-readable medium of claim 13, wherein the feature components comprise image sequence, audio, and textual components.

16. The computer-readable medium of claim 13, wherein the audio attention models comprise saliency, speech, and/or music attention models.

17. The computer-readable medium of claim 13, wherein the linguistic attention models comprise closed caption, and/or automated speech recognition attention models.

18. The computer-readable medium of claim 13, wherein M_v, M_a, and M_l are defined as follows:

$\begin{array}{c}{M}_v=\left(\underset{i=1}{\overset{p}{\sum <br><br>}}<br><br>w_i·<br><br>\stackrel{\_}{M_i}\right)\times <br><br>{\left({\stackrel{\_}{M}}_{c<br><br><br><br>m}\right)}^{S_{c<br><br><br><br>m}},\\ M_a=\left(\underset{j=1}{\overset{q}{\sum <br><br>}}<br><br>w_j·<br><br>\stackrel{\_}{M_j}\right)\times <br><br>{\left({\stackrel{\_}{M}}_{\mathrm{as}}\right)}^{S_{\mathrm{as}}},\\ M_l=\underset{k=1}{\overset{r}{\sum <br><br>}}<br><br>w_k·<br><br>\stackrel{\_}{M_k}<br><br>,\end{array}$

w_i, w_j, w_k being weighted values of visual, audio, and linguistic attention models respectively, M_cm representing a normalized camera attention used as a visual attention model magnifier, and S_cm identifying a magnifier switch tat is based on multiple criteria.

19. The computer-readable medium of claim 18, wherein the multiple criteria comprise:

if S_cm>

=1, the magnifier is turned on;

if S_cm=0, the magnifier is turned off; and

wherein a large S_cm value indicates a more powerful magnifier than a low S_cm value.

20. The computer-readable medium of claim 13, wherein the visual attention models comprise motion, static, face, and/or camera attention models.

21. The computer-readable medium of claim 20, wherein the camera attention model is based at least in part on the following criteria:

during camera zooming operations, frame importance increases temporally and is a function of zooming speed such that a first frame generated during a fast zooming operation is of higher relative importance that a second frame generated during a slower zooming operation; and

during camera panning operations, frame importance is an inverse of panning speed and a function of panning direction.

22. The computer-readable medium of claim 21, wherein frames generated during a horizontal camera panning operation are calculated to be of lesser relative importance as compared to frames generated during a vertical panning operation.

23. The computer-readable medium of claim 21, wherein calculated importance of a frame generated during panning or zooming operations is reduced from a higher importance to a lower importance as a function of ending the panning or zooming operation and passage of a certain period of time.

24. A computing device for creating a comprehensive user attention model, the computing device comprising:

a processer;

a memory coupled to the processor the memory comprising computer-program instructions executable by the processor for;

generating visual, audio, and linguistic attention data based on application of multiple attention models to a plurality of video data sequence feature components, the feature components comprising image sequence, audio, and text-related features;

integrating the visual, audio, and linguistic attention data to create the comprehensive user attention model;

wherein the comprehensive user attention model is a computational representation of elements of the video data sequence tat attract user attention; and

wherein the computational representation is defined as;

A=w_v·

M_v+w_a·

M_a+w_l·

M_l,

w_v, W_a, w_l representing weights for linear combination, and wherein M_v, M_a, and M_l represent normalized visual, audio, and linguistic attention models.

25. The computing device of claim 24, wherein the computer-program instructions for generating further comprise instructions for creating the visual attention data with motion, static, face, and/or camera attention models.

26. The computing device of claim 24, wherein the computer-program instructions for generating further comprise instructions for creating the audio attention data with saliency, speech, and/or music attention models.

27. The computing device of claim 24, wherein the computer-program instructions for generating further comprise instructions for creating the linguistic attention data with closed caption, and/or automated speech recognition attention models.

28. The computing device of claim 24, wherein M_v, M_a, and M_l are defined as follows:

$\begin{array}{c}{M}_v=\left(\underset{i=1}{\overset{p}{\sum <br><br>}}<br><br><br><br>w_i·<br><br>\stackrel{\_}{M_i}\right)\times <br><br>{\left({\stackrel{\_}{M}}_{c<br><br><br><br>m}\right)}^{S_{c<br><br><br><br>m},}\\ M_a=\left(\underset{j=1}{\overset{q}{\sum <br><br>}}<br><br><br><br>w_j·<br><br>\stackrel{\_}{M_j}\right)\times <br><br>{\left({\stackrel{\_}{M}}_{a<br><br><br><br>s}\right)}^{S_{a<br><br><br><br>s},}\\ M_l=\underset{k=1}{\overset{r}{\sum <br><br>}}<br><br><br><br>w_k·<br><br>\stackrel{\_}{M_k},\end{array}$

w_i, w_j, w_k being weighted values of visual, audio, and linguistic attention models respectively, M_cm representing a normalized camera attention used as a visual attention model magnifier, and S_cm identifying a magnifier switch that is based on multiple criteria.

29. The computing device of claim 28, wherein the multiple criteria comprise:

if S_cm>

=1, the magnifier is open;

if S_cm=0, the magnifier is closed; and

wherein a large S_cm value indicates a more powerful magnifier than a low S_cm value.

30. The computing device of claim 24, wherein the multiple attention models comprise a camera attention model, and wherein the computer-program instructions for generating the visual attention data generate camera attention data based at least in part on the following criteria:

during camera zooming operations, frame importance increases temporally and is a function of zooming speed such that a first frame generated during a fast zooming operation is of higher relative importance that a second frame generated during a slower zooming operation; and

during camera panning operations, frame importance is an inverse of panning speed and a function of panning direction.

31. The computing device of claim 30, wherein frames generated during a horizontal camera panning operation are calculated to be of lesser relative importance as compared to frames generated during a vertical panning operation.

32. The computing device of claim 30, wherein calculated importance of a frame generated during panning or zooming operations is reduced from a higher importance to a lower importance as a function of ending the panning or zooming operation and passage of a certain period of time.