Systems and methods for generating a motion attention model

• US 7,116,716 B2
• Filed: 11/01/2002
• Issued: 10/03/2006
• Est. Priority Date: 11/01/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Ma, Yu-Fei
• Primary Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US10/285,933
• Publication Number:
  US 20040086046A1
• Time in Patent Office: 1,432 Days
• Field of Search: 375/240.16, 375/240.29, 345/474, 345/629, 382/170, 382/236, 382/107, 340/982
• US Class Current: 375/240.16
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Systems and methods to generate a motion attention model of a video data sequence are described. In one aspect, a motion saliency map B is generated to precisely indicate motion attention areas for each frame in the video data sequence. The motion saliency maps are each based on intensity I, spatial coherence Cs, and temporal coherence Ct values. These values are extracted from each block or pixel in motion fields that are extracted from the video data sequence. Brightness values of detected motion attention areas in each frame are accumulated to generate, with respect to time, the motion attention model.

Claims

1. A method for generating a motion attention model of a video data sequence, the method comprising:

generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C_s, and temporal coherence C_t values from each location of a block MB_ij in motion fields extracted from the video data sequence;

accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and

wherein I is a normalized magnitude of a motion vector that is calculated according to;

$I<br><br>\left(i,j\right)=\sqrt{{\mathrm{dx}}_{i,j}^2+{\mathrm{dy}}_{i,j}^2}/\mathrm{Max}<br><br><br><br>\mathrm{Mag},$

wherein (dx_i,j, dy_i,j) denotes two components of the motion vector in motion field, and MaxMag is the maximum magnitude of motion vectors.

5. The method of claim 1, wherein the video data sequence is in an MPEG data format.

6. The method of claim 1 wherein the motion field is a motion vector field, or an optical flow field.

7. The method of claim 1, wherein the motion attention model is calculated according to:

$M_{\mathrm{motion}}=\left(\underset{r\in <br><br>\mathrm{\Lambda <br><br>}}{\sum <br><br>}<br><br><br><br>\underset{q\in <br><br>{\mathrm{\Omega <br><br>}}_r}{\sum <br><br>}<br><br>B_q\right)/N_{\mathrm{MB}},$

B_q being brightness of a block in the motion saliency map, Λ

being the set of detected motion attention areas, Ω

_r denoting a set of blocks in each detected motion attention area, N_MB being a number of blocks in a motion field; and

wherein an M_motion value for each frame in the video data sequence represents a continuous motion attention curve with respect to time.

2. A method for generating a motion attention model of a video data sequence, the method comprising:

generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C_s, and temporal coherence C_t values from each location of a block MB_ij in motion fields extracted from the video data sequence;

accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and

wherein C_s is calculated with respect to spatial widow w as follows;

$\mathrm{Cs}<br><br>\left(i,j\right)=-\underset{t=1}{\overset{n}{\sum <br><br>}}<br><br><br><br>p_s<br><br>\left(t\right)<br><br>\mathrm{Log}<br><br>\left(p_s<br><br>\left(t\right)\right),<br><br>p_s<br><br>\left(t\right)={\mathrm{SH}}_{i,j}^w<br><br>\left(t\right)/\underset{k=1}{\overset{n}{\sum <br><br>}}<br><br><br><br>{\mathrm{SH}}_{i,j}^w<br><br>\left(k\right);<br><br>\mathrm{and}$

wherein SH^w_i,j(t) is a spatial phase histogram whose probability distribution function is p_s(t), and n is a number of histogram bins.

3. A method for generating a motion attention model of a video data sequence, the method comprising:

generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C_s, and temporal coherence C_t values from each location of a block MB_ij in motion fields extracted from the video data sequence;

accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and

wherein C_t is calculated with respect to a sliding window of size L frames along time t axis as;

$\mathrm{Ct}<br><br>\left(i,j\right)=-\underset{t=1}{\overset{n}{\sum <br><br>}}<br><br><br><br>p_t<br><br>\left(t\right)<br><br>\mathrm{Log}<br><br>\left(p_t<br><br>\left(t\right)\right),<br><br>p_t<br><br>\left(t\right)={\mathrm{TH}}_{i,j}^L<br><br>\left(t\right)/\underset{k=1}{\overset{n}{\sum <br><br>}}<br><br><br><br>{\mathrm{TH}}_{i,j}^L<br><br>\left(k\right);<br><br>\mathrm{and}$

wherein TH^L_i,j(t) is a temporal phase histogram whose probability distribution function is p_t(t), and n is a number of histogram bins.

4. A method for generating a motion attention model of a video data sequence, the method comprising:

generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C_s, and temporal coherence C_t values from each location of a block MB_ij in motion fields extracted from the video data sequence;

accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and

wherein the method further comprises instructions for generating the motion saliency map B according to

B=I×

C_t×

(1I×

C_s).

8. A computer-readable medium for generating a motion attention model of a video data sequence, the computer-readable medium comprising computer-program instructions executable by a processor for:

generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C_s, and temporal coherence C_t values from each location of a block MB_ij in motion fields extracted from the video data sequence; and

accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence.

9. A computer-readable medium for generating a motion attention model of a video data sequence, the computer-readable medium comprising computer-program instructions executable by a processor for:

extracting a motion field between a current frame and a next frame of the video data sequence;

determining, at each location of a block MB_ij, intensity I, spatial coherence C_s, and temporal coherence C_t values from the motion field;

integrating intensity I, spatial coherence C_s, and temporal coherence C_t to generate a motion saliency map B, the motion saliency map precisely indicating motion attention areas in the motion field; and

accumulating brightness of detected motion attention areas to indicate a motion attention degree for the current frame.

10. The computer-readable medium of claim 9, wherein the video data sequence is in an MPEG data format.

11. The computer-readable medium of claim 10, wherein the motion vector field is a motion vector field in MPEG stream, an optical flow field, or the result of block-based motion estimation.

12. The computer-readable medium of claim 9, wherein the computer-program instructions for determining intensity I further comprise instructions for generating intensity I as a normalized magnitude of the motion vector according to:

$I<br><br>\left(i,j\right)=\sqrt{{\mathrm{dx}}_{i,j}^2+{\mathrm{dy}}_{i,j}^2}/\mathrm{Max}<br><br><br><br>\mathrm{Mag},$

wherein (dx_i,j, dy_i,j) denotes two components of the motion vector, and MaxMag is the maximum magnitude of motion vector.

13. The computer-readable medium of claim 9, wherein the computer-program instructions for determining spatial coherence C_s further comprise instructions for:

generating a phase histogram in a spatial window with the size of w×

w (pixels) at each location of the block;

measuring phase distribution is measured by entropy as follows;

$\mathrm{Ct}<br><br>\left(i,j\right)=-\underset{t=1}{\overset{n}{\sum <br><br>}}<br><br><br><br>p_t<br><br>\left(t\right)<br><br>\mathrm{Log}<br><br>\left(p_t<br><br>\left(t\right)\right),<br><br>p_t<br><br>\left(t\right)={\mathrm{TH}}_{i,j}^L<br><br>\left(t\right)/\underset{k=1}{\overset{n}{\sum <br><br>}}<br><br><br><br>{\mathrm{TH}}_{i,j}^L<br><br>\left(k\right);<br><br>\mathrm{and}$

wherein SH^w_i,j(t) is a spatial phase histogram whose probability distribution function is p_s(t), and ii is a number of histogram bins.

14. The computer-readable medium of claim 9, wherein the computer-program instructions for determining further comprise instructions for generating temporal coherence C_t further comprise instructions for:

generating temporal coherence C_t with respect to a sliding window of size L frames along time t axis, as;

$\begin{array}{c}\mathrm{Ct}<br><br>\left(i,j\right)=-\underset{t=1}{\overset{n}{\sum <br><br>}}<br><br>p_t<br><br>\left(t\right)<br><br>\mathrm{Log}<br><br>\left(p_t<br><br>\left(t\right)\right),\\ p_t<br><br>\left(t\right)={\mathrm{TH}}_{i,j}^L<br><br>\left(t\right)/\underset{k=1}{\overset{n}{\sum <br><br>}}<br><br>{\mathrm{TH}}_{i,j}^L<br><br>\left(k\right);<br><br>\mathrm{and}\end{array}$

wherein TH^L_i,j(t) is a temporal phase histogram whose probability distribution function is p_t(t), and n is a number of histogram bins.

15. The computer-readable medium of claim 9, wherein the computer-program instructions for integrating further comprises instructions for generating the motion saliency map B according to B=I×

C_t×

(1−

I×

C_s).

16. The computer-readable medium of claim 9, wherein the computer-program instructions for accumulating brightness further comprise instructions for detecting salient motion regions by employing histogram balance, median filtering, binarization, region growing, and/or region selection to the detected motion attention areas.

17. The computer-readable medium of claim 9, further comprising computer-program instructions for executing the extracting, determining, integrating, and accumulating for each frame in the video data sequence to generate a motion attention model for all frames in the video data sequence with respect to time.

18. The computer-readable medium of claim 17, wherein the motion attention model for all frames is calculated according to:

$M_{\mathrm{motion}}=\left(\underset{r\in <br><br>\mathrm{\Lambda <br><br>}}{\overset{}{\sum <br><br>}}<br><br>\underset{q\in <br><br>{\mathrm{\Omega <br><br>}}_r}{\overset{}{\sum <br><br>}}<br><br>B_q\right)/N_{\mathrm{MB}},$

B_q being brightness of a block in the motion saliency map, Λ

being the set of detected motion attention areas, Ω

_r denoting a set of blocks in each detected motion attention area, N_MB being a number of blocks in a motion field; and

wherein an M_motion value for each frame in the video data sequence represents a continuous motion attention curve with respect to time.

19. A computing device for generating a motion attention model of a video data sequence that includes multiple frames, the computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for,

for each frame in the video data sequence;

(a) extracting a motion field between a current frame and a next frame of the video data sequence;

(b) determining, for each block MB_ij represented by the motion field, intensity I, spatial coherence C_s, and temporal coherence C_t values to generate a motion saliency map B, the motion saliency map precisely indicating motion attention areas for each frame in the video data sequence; and

accumulating brightness of detected motion attention areas to generate the motion attention model with respect to time.

20. The computing device of claim 19, wherein the video data sequence is in an MPEG data format.

21. The computing device of claim 19, wherein the motion field is a motion vector field, or an optical flow field.

22. The computing device of claim 19, wherein the computer-program instructions for determining intensity I farther comprise instructions for generating intensity I as a normalized magnitude of the motion vector according to:

$I<br><br>\left(i,j\right)=\sqrt{{\mathrm{dx}}_{i,j}^2+{\mathrm{dy}}_{i,j}^2}/\mathrm{MaxMag},$

wherein (dx_i,j, dy_i,j) denotes two components of the motion vector, and MaxMag is the maximum magnitude of motion vector.

23. The computing device of claim 19, wherein the computer-program instructions for determining spatial coherence C_s further comprise instructions for:

generating a phase histogram in a spatial window with the size of w×

w (pixels) at each location of the block;

measuring phase distribution is measured by entropy as follows;

$\begin{array}{c}\mathrm{Cs}<br><br>\left(i,j\right)=-\underset{t=1}{\overset{n}{\sum <br><br>}}<br><br>p_s<br><br>\left(t\right)<br><br>\mathrm{Log}<br><br>\left(p_s<br><br>\left(t\right)\right),\\ p_s<br><br>\left(t\right)={\mathrm{SH}}_{i,j}^w<br><br>\left(t\right)/\underset{k=1}{\overset{n}{\sum <br><br>}}<br><br>{\mathrm{SH}}_{i,j}^w<br><br>\left(k\right);<br><br>\mathrm{and}\end{array}$

wherein SH^w_i,j(t) is a spatial phase histogram whose probability distribution function is p_s(t), and n is a number of histogram bins.

24. The computing device of claim 19, wherein the computer-program instructions for determining further comprise instructions for generating temporal coherence C_t further comprise instructions for:

generating temporal coherence C_t with respect to a sliding window of size L frames along time t axis, as;

$\begin{array}{c}\mathrm{Ct}<br><br>\left(i,j\right)=-\underset{t=1}{\overset{n}{\sum <br><br>}}<br><br>p_t<br><br>\left(t\right)<br><br>\mathrm{Log}<br><br>\left(p_t<br><br>\left(t\right)\right),\\ p_t<br><br>\left(t\right)={\mathrm{TH}}_{i,j}^L<br><br>\left(t\right)/\underset{k=1}{\overset{n}{\sum <br><br>}}<br><br>{\mathrm{TH}}_{i,j}^L<br><br>\left(k\right);<br><br>\mathrm{and}\end{array}$

wherein TH^L_i,j(t) is a temporal phase histogram whose probability distribution function is p₁(t), and n is a number of histogram bins.

25. The computing device of claim 19, wherein the computer-program instructions for integrating further comprises instructions for generating the motion saliency map B according to B=I×

C_t×

(1−

I×

C_s).

26. The computing device of claim 19, wherein the computer-program instructions for accumulating brightness further comprise instructions for detecting salient motion regions by employing histogram balance, median filtering, binarization, region growing, and/or region selection to the detected motion attention areas.

27. The computing device of claim 19, wherein the motion attention model is calculated according to:

$M_{\mathrm{motion}}=\left(\underset{r\in <br><br>\mathrm{\Lambda <br><br>}}{\overset{}{\sum <br><br>}}<br><br>\underset{q\in <br><br>{\mathrm{\Omega <br><br>}}_r}{\overset{}{\sum <br><br>}}<br><br>B_q\right)/N_{\mathrm{MB}},$

B_q being brightness of a block in the motion saliency map, Λ

being the set of detected motion attention areas, Ω

_r denoting a set of blocks in each detected motion attention area, N_MB being a number of blocks in a motion field; and

wherein an M_motion value for each frame in the video data sequence represents a continuous motion attention curve with respect to time.