Method for detecting visual saliencies of video image based on spatial and temporal features

• US 9,466,006 B2
• Filed: 01/21/2015
• Issued: 10/11/2016
• Est. Priority Date: 02/24/2014
• Status: Active Grant

• Current Assignee: Beijing University of Technology (Beijing Education Commission)
• Original Assignee: Beijing University of Technology (Beijing Education Commission)
• Inventors: Duan, Lijuan
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/601,254
• Publication Number:
  US 20160210528A1
• Time in Patent Office: 629 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

The present invention relates to a method for detecting visual saliencies of a video image based on spatial and temporal features, including: dividing an input image into image blocks and vectorizing the image blocks; decreasing dimensions of each image block through principal component analysis; calculating a dissimilarity between each image block and each of the other image blocks; calculating a visual saliency of each image block by combining a distance between image blocks, to obtain a spatial feature saliency map; imposing a central bias on the spatial feature saliency map; calculating a motion vector of each image block, extracting a temporal visual saliency of the current image by combining motion vectors of previous two frames, to obtain a temporal feature saliency map; integrating the spatial feature saliency map and the temporal feature saliency map to obtain a spatiotemporal feature saliency map, and smoothing the spatiotemporal feature saliency map to obtain a resulted image finally reflecting a saliency of each region on the current image. In the present invention, a saliency map integrating the temporal features and the spatial features, so that saliencies in different regions in a video may be predicted more accurately.

Claims

1. A method for detecting visual saliencies of a video image based on spatial and temporal features, characterized in that, the method comprises the following steps:

1) selecting a current image from a video, dividing the current image into L non-overlapping square image blocks, each of the image blocks containing K² pixels;

2) vectorizing each pixel in each of the image blocks into a column vector, each value in the column vectors being a R value, a G value and a B value of a RGB value of the pixel such that the column vector has a length of 3K² values;

3) jointing column vectors of all image blocks in a row direction, to form a value matrix of the current image which has 3K² rows and L columns;

4) performing a dimension decreasing operation on the value matrix of the current image by utilizing a principal component analysis algorithm;

5) calculating a spatial visual saliency of each image block in the current image with decreased dimensions, adding the spatial visual saliency to a position corresponding a position of the image block in the current image, to constitute a two-dimension spatial feature saliency map of the current image;

6) calculating a temporal saliency of each image block in the current image with decreased dimensions according to a block matching method, adding the temporal visual saliency to a position corresponding a position of the image block in the current image, to constitute a two-dimension temporal feature saliency map of the current image; and

7) integrating the two-dimension spatial feature saliency map and the two-dimension temporal feature saliency map, to obtain a spatiotemporal feature saliency map.

2. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 1, characterized in that,

the step 7) further comprises:

performing a smoothing operation on the spatiotemporal feature saliency map through a two-dimension Gaussian smoothing operator, to obtain a resulted image finally reflecting a saliency of each region on the current image.

3. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 1, characterized in that, a plurality of frames of images of the video is acquired at equal time intervals, and the current image is one of the frames of images.

4. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 3, characterized in that,

in the step 6), calculating a temporal saliency of each image block in the current image with decreased dimensions according to a block matching method comprises:

firstly, calculating a motion vector of the image block in the video, combining motion vectors of image blocks corresponding to the image block in previous frames of images in the video, to obtain a temporal visual saliency of the image block.

5. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 4, characterized in that,

calculating a motion vector of the image block in the video, comprising:

in a previous frame image of the current image, searching out an image block which has a least matching error with respect to the image block, and taking a horizontal displacement and a vertical displacement of the image block between the two image blocks as the motion vector of the image block of the current image.

6. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 5, characterized in that,

obtaining a temporal visual saliency of the image block comprises:

combining a horizontal displacement and a vertical displacement of the motion vector of the image block of the current image into a component V(t); and

an average of components of corresponding image blocks in previous frames of images is subtracted from the component V(t), to obtain the temporal visual saliency of the image block in the current image.

7. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 6, characterized in that, the previous frames of images are previous 3 frames of images.

8. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 1, characterized in that, in the step 5),

constituting a two-dimension spatial feature saliency map of the current image comprises:

performing a central bias operation on the two-dimension spatial feature saliency map according to average attention weights of human eyes.

9. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 1, characterized in that, in the step 1),

if the current image is a square figure, the current image is divided into L non-overlapping square image blocks;

and

if the current image is a non-square figure, the current image is stretched into a square figure.

10. The method for detecting visual saliencies of a video image based on spatial and temporal features of claim 1, characterized in that, in the step 5),

calculating a spatial visual saliency of each image block in the current image with decreased dimensions, comprises:

calculating a dissimilarity between the image block and each of the other image blocks in the current image, and determining the spatial visual saliency of the image block according to an Euclidean distance between the image block and each of the other image blocks.

11. A method for detecting visual saliencies of a video image based on spatial and temporal features, characterized in that, the method comprises the following steps:

1) acquiring a plurality of frames of images of the video with a predetermined time interval, selecting a current image from the frames of images, dividing the current image into L non-overlapping square image blocks, each of the image blocks containing K² pixels;

2) vectorizing each pixel in each of the image blocks into a column vector, each value in the column vectors being one of a R value, a G value and a B value of a RGB value of the pixel such that the column vector has a length of 3K² values;

3) jointing column vectors of all image blocks in a row direction, to form a value matrix of the current image which has 3K² rows and L columns;

4) performing a dimension decreasing operation on the value matrix of the current image by utilizing a principal component analysis algorithm;

5) calculating a spatial visual saliency of each image block in the current image with decreased dimensions, adding the spatial visual saliency to a position corresponding a position of the image block in the current image, to constitute a two-dimension spatial feature saliency map of the current image;

6) calculating a temporal saliency of each image block in the current image with decreased dimensions according to a block matching method, adding the temporal visual saliency to a position corresponding a position of the image block in the current image, to constitute a two-dimension temporal feature saliency map of the current image; and

7) integrating the two-dimension spatial feature saliency map and the two-dimension temporal feature saliency map, to obtain a spatiotemporal feature saliency map.