Device and method for analyzing the correlation between an image and another image or between an image and a video

• US 9,189,700 B2
• Filed: 11/11/2011
• Issued: 11/17/2015
• Est. Priority Date: 02/21/2011
• Status: Active Grant

• Current Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Original Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Inventors: Lee, Jaehyung
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US14/000,266
• Publication Number:
  US 20140044361A1
• Time in Patent Office: 1,467 Days
• Field of Search: 382/210, 382/103, 382/155, 382/159, 382190-192
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

The present invention relates to a device and method for analyzing the correlation between an image and another image or between an image and a video. The device for analyzing the correlation between images and the method for using same include: a feature data generating unit for determining a feature point of an image and generating feature data which includes feature point orientation information on each determined feature point; and a relation analyzing unit for analyzing the correlation between an image and another image using feature data generated from the feature data generating unit. The relation analyzing unit includes: a unit for determining corresponding feature points, which determines a pair of corresponding feature points between compared images using feature data generated from the feature data generating unit; and a reliability estimating unit for estimating the reliability of the analysis of the relation between images on the basis of feature point orientation information on a feature point, in pairs of feature points determined by the unit for determining corresponding feature points. According to the present invention, provided are a device and method for quickly and efficiently analyzing a correlation, such as whether or not there is a similarity between an image and another image or between an image and a video, wherein said video includes an image, or a frame of said video corresponds to an image.

Claims

1. An apparatus for analyzing a correlation between images, comprising:

a feature data generation unit for determining feature points of each image, and generating feature data including pieces of feature point orientation information for the respective feature points; and

a relation analysis unit for analyzing a correlation between an image and another image using the feature data generated by the feature data generation unit,

wherein at least one of the feature data generation unit and the relation analysis unit is implemented by a hardware computing device;

wherein the relation analysis unit comprises

a corresponding feature point determination unit for determining pairs of corresponding feature points between the images to be compared using the feature data generated by the feature data generation unit, and

a reliability estimation unit for estimating reliability of analysis of a correlation between the images, based on the feature point orientation information of the feature points included in the feature point pairs determined by the corresponding feature point determination unit;

wherein the reliability estimation unit estimates the reliability of the analysis of the relation between the images, based on a probability density function for estimated values of differences between orientations of the feature points included in the respective feature point pairs; and

wherein the probability density function is used to calculate a mean and a variation of the estimated values, a probability (p) of pairs of feature points being observed is calculated using a standard normal distribution function based on the mean and the variation, and the calculated probability (p) or a function based on the probability (p) is compared with a threshold, thus enabling the reliability of the analysis of the correlation between the images to be estimated.

2. The apparatus of claim 1, wherein the probability (p) of the pairs of feature points being observed is calculated by

3. An apparatus for analyzing a correlation between images, comprising:

a feature data generation unit for determining feature points of each image, and generating feature data including pieces of feature point orientation information for the respective feature points; and

a relation analysis unit for analyzing a correlation between an image and another image using the feature data generated by the feature data generation unit,

wherein at least one of the feature data generation unit and the relation analysis unit is implemented by a hardware computing device;

wherein the relation analysis unit comprises

a corresponding feature point determination unit for determining pairs of corresponding feature points between the images to be compared using the feature data generated by the feature data generation unit, and

a reliability estimation unit for estimating reliability of analysis of a correlation between the images, based on the feature point orientation information of the feature points included in the feature point pairs determined by the corresponding feature point determination unit;

wherein the reliability estimation unit estimates the reliability of the analysis of the relation between the images, based on a probability density function for estimated values of differences between orientations of the feature points included in the respective feature point pairs;

wherein the feature data generation unit comprises;

a feature point determination unit for determining feature points from each image and extracting feature point information of the determined feature points;

a feature point orientation estimation unit for estimating pieces of orientation information for the respective feature points determined by the feature point determination unit; and

a feature data configuration unit for generating binary feature vectors based on the feature point information and the orientation information, for the respective feature points determined by the feature point determination unit, and generating feature data of the image including the generated binary feature vectors; and

wherein the feature data configuration unit generates, for the feature points determined by the feature point determination unit, surrounding image areas including the respective feature points, aligns the generated surrounding image areas in an identical orientation based on the orientation information, divides each of aligned surrounding image areas into sub-regions, and generates binary feature vectors based on averages of brightness values of the divided sub-regions.

4. The apparatus of claim 3,

wherein each binary feature vector is generated by at least one selected from among difference vectors and double-difference vectors obtained from the averages of brightness values of the sub-regions,

wherein selection of at least one from among the difference vectors and the double-difference vectors obtained from averages of brightness values of the sub-regions is performed in correspondence with respective bits of the binary feature vector, and

wherein a linear combination or a nonlinear combination is calculated for the difference vectors and the double-difference vectors selected in correspondence with the respective bits, and resulting values of the calculation are compared with a threshold, thus enabling values of corresponding bits of the binary feature vector to be determined.

5. The apparatus of claim 4, wherein alignment is performed based on a criterion preset for the respective bits of the binary feature vector.

6. A method of analyzing a correlation between images, comprising:

a first step of determining feature points of each image, and generating feature data including pieces of feature point orientation information for the respective feature points; and

a second step of analyzing a correlation between an image and another image using the feature data generated at the first step,

wherein the second step comprises;

2-1) determining pairs of corresponding feature points between the images to be compared using the generated feature data; and

2-2) estimating reliability of analysis of a correlation between the images, based on the feature point orientation information of the feature points,

wherein the step of 2-2 is configured to estimate the reliability of the analysis of the relation between the images, based on a probability density function for estimated values of differences between orientations of the feature points included in the respective feature point pairs, and

wherein the step of 2-2 further includes calculating a mean and a variation of the estimated values using the probability density function, calculating a probability (p) of pairs of feature points being observed using a standard normal distribution function based on the mean and the variation, and comparing the calculated probability (p) or a function based on the probability (p) with a threshold, so as to estimate the reliability of the analysis of the correlation between the images.

7. An apparatus for analyzing a correlation between an image and a video, comprising:

an image feature data generation unit for determining feature points of an image and generating image feature data including pieces of feature point orientation information for the respective feature points;

a video feature data generation unit for determining, for a video, feature points for one or more of frames constituting the video, and generating pieces of frame-based image feature data including pieces of feature point orientation information for the respective frame-based feature points, thus generating video feature data; and

a relation analysis unit for comparing the image feature data with the video feature data and then analyzing a correlation between the image and the video,

wherein at least one of the image feature data generation unit, the video feature data generation unit and the relation analysis unit is implemented by a hardware computing device;

wherein the relation analysis unit comprises;

a candidate selection unit for determining a matching video by comparing the image feature data with the pieces of frame-based image feature data of the video feature data, and determining one or more of frames constituting the determined video, or for determining one or more matching frames by comparing the image feature data with the pieces of frame-based image feature data of the video feature data;

a corresponding feature point determination unit for determining pairs of corresponding feature points between the image and the one or more frames determined by the candidate selection unit, based on the image feature data of the image and pieces of image feature data of the determined frames; and

a reliability estimation unit for estimating reliability of analysis of the correlation between the image and the video based on pieces of feature point orientation information of feature points included in the feature point pairs determined by the corresponding feature point determination unit,

wherein the reliability estimation unit estimates the reliability of the analysis of the relation between the image and the video, based on a probability density function for estimated values of differences between orientations of the feature points included in the respective feature point pairs, and

wherein;

the image feature data generation unit comprises;

an image feature point determination unit for determining feature points from the image and extracting feature point information of the determined feature points;

an image feature point orientation estimation unit for estimating pieces of orientation information for the respective feature points determined by the feature point determination unit; and

an image feature data configuration unit for generating binary feature vectors based on the feature point information and the orientation information, for the respective feature points determined by the feature point determination unit, and generating feature data of the image including the generated binary feature vectors, and the video feature data generation unit comprises;

a frame selection unit for extracting, for each video, frames constituting the video at regular time interval, calculating a difference between each extracted frame and one or more previously extracted frames, and selecting the extracted frame when the difference is equal to or greater than a threshold;

a frame-based feature point determination unit for determining feature points for the frame selected by the frame selection unit and extracting feature point information of the determined feature points;

a frame-based feature point orientation estimation unit for estimating pieces of orientation information for the respective feature points determined by the frame-based feature point determination unit; and

a frame-based feature data configuration unit for generating binary feature vectors based on the feature point information and the orientation information, for the respective feature points determined by the frame-based feature point determination unit, and configuring frame-based feature data including the generated binary feature vectors.

8. A method of analyzing a correlation between an image and a video, comprising:

a first step of determining feature points of an image and generating image feature data including pieces of feature point orientation information for the respective feature points;

a second step of determining, for a video, feature points for one or more of frames constituting the video, and generating pieces of frame-based image feature data including pieces of feature point orientation information for the respective frame-based feature points, thus generating video feature data; and

a third step of comparing the image feature data with the video feature data and then analyzing a correlation between the image and the video,

wherein the third step comprises the steps of;

3-1) determining a matching video by comparing the image feature data with the pieces of frame-based image feature data of the video feature data, and determining one or more of frames constituting the determined video, or for determining one or more matching frames by comparing the image feature data with the pieces of frame-based image feature data of the video feature data;

3-2) determining pairs of corresponding feature points between the image and the one or more frames determined at 3-1), based on the image feature data and pieces of image feature data of the determined frames; and

3-3) estimating reliability of analysis of the correlation between the image and the video based on pieces of feature point orientation information of feature points included in the feature point pairs determined at 3-2),

wherein the step of 3-3) is configured to estimate the reliability of the analysis of the relation between the image and the video, based on a probability density function for estimated values of differences between orientations of the feature points included in the respective feature point pairs, and

wherein;

the first step further comprises;

determining the feature points from the image and extracting feature point information of the determined feature points;

estimating pieces of orientation information for the respective feature points determined by the feature point determination unit; and

generating binary feature vectors based on the feature point information and the orientation information, for the respective feature points determined by the feature point determination unit, and generating feature data of the image including the generated binary feature vectors, and the second step further comprises;

extracting, for each video, frames constituting the video at regular time interval, calculating a difference between each extracted frame and one or more previously extracted frames, and selecting the extracted frame when the difference is equal to or greater than a threshold;

determining feature points for the frame selected by the frame selection unit and extracting feature point information of the determined feature points;

estimating pieces of orientation information for the respective feature points determined by the frame-based feature point determination unit; and

generating binary feature vectors based on the feature point information and the orientation information, for the respective feature points determined by the frame-based feature point determination unit, and configuring frame-based feature data including the generated binary feature vectors.