Image processing method and image processing apparatus

• US 20080278633A1
• Filed: 05/09/2007
• Published: 11/13/2008
• Est. Priority Date: 05/09/2007
• Status: Abandoned Application

• Current Assignee: Monolith Company Limited
• Original Assignee: Monolith Company Limited
• Inventors: Yamashita, Shinichi, Borovikov, Igor, Tsoupko-Sitnikov, Mikhail, Endo, Masuharu
• Application Number:
  US11/798,035
• Publication Number:
  US 20080278633A1
• Time in Patent Office: Days
• US Class Current: 348/699
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: H04N 19/119 Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

A corresponding point information generator computes matching between a source image frame and a destination image frame in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel matching. A motion vector detector determines a motion vector for each pixel in the source image frame according to a result of matching. A reliability area isolating unit segments an image frame in which a motion vector is determined into blocks, so as to isolate, in each block, a reliable area characterized by relatively high precision of the motion vector as calculated and a non-reliable area characterized by relatively low precision of the motion vector. A motion vector improving unit calculates, when a motion vector of a reliable area is applied to a pixel in a non-reliable area adjacent to the reliable area, an error between a pixel value occurring at the destination as a result of application and a pixel value of a corresponding pixel in the destination image frame, and, when the error is equal to or smaller than a threshold, incorporates the pixel in the non-reliable area into the reliable area, and replaces the motion vector of that pixel by the motion vector of the reliable area.

Claims

1. An image processing method comprising:

computing matching between two image frames in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel correspondence; and

for a pixel characterized by relatively low reliability of correspondence, performing block matching between images so as to determine correspondence block by block.

2. An image processing method comprising:

initial matching in which correspondence point information is determined for each pixel in a source image frame and in a destination image frame in image data comprising consecutive image frames;

determining a motion vector according to a result of matching and determining for each pixel the reliability of the motion vector thus determined; and

for a pixel characterized by relatively low reliability of the motion vector as determined, performing block matching between blocks in the source image frame and in the destination image frame, and determining an updated motion vector by re-calculation, each block comprising a plurality of pixels.

3. An image processing apparatus comprising:

a matching processor which computes matching between a source image frame and a destination image frame in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel correspondence;

a motion vector detector which determines a motion vector for each pixel in the source image frame, according to a result of matching;

a reliability area isolating unit which segments an image frame in which a motion vector is determined into blocks, so as to partition into a reliable area characterized by relatively high precision of the motion vector as calculated and a non-reliable area characterized by relatively low precision of the motion vector; and

a motion vector improving unit which calculates, when a motion vector of a reliable area is applied to a pixel in a non-reliable area adjacent to the reliable area, an error between a pixel value occurring at the destination as a result of application and a pixel value of a corresponding pixel in the destination image frame, and, when the error is equal to or smaller than a threshold, incorporates the pixel in the non-reliable area into the reliable area, and replaces the motion vector of that pixel by the motion vector of the reliable area.

4. The image processing apparatus according to claim 3, further comprising:

a block matching unit which performs block matching on blocks with pixels, which are included in the non-reliable area of the source image frame and which are not incorporated into the reliable area by the motion vector improving unit, so as to exhaustively search for a block in the destination image frame characterized by the smallest matching error, wherein

the motion vector of the block subjected to matching is replaced by the motion vector identified as a result of block matching.

5. An image processing apparatus comprising:

a motion vector detector which determines motion vectors in a forward direction and in a reverse direction between a source image frame and a destination image frame in image data comprising consecutive image frames; and

an occlusion detector which compares the motion vector in the forward direction with the motion vector in the reverse direction, and, when there is a pixel characterized by a difference between the two, determines that the pixel is included in either i) an area in which an object including the pixel inside is hidden by another object within the same frame, or ii) an occlusion area in which the object including the pixel inside hides another object within the same frame behind.

6. The image processing apparatus according to claim 5, wherein

the occlusion detector processes a pixel for which the size of the motion vector is zero either in the forward direction or the reverse direction such that the detector determines an area including the pixel to be a covered area hidden by an occluder, when the source image frame includes a point corresponding to the pixel but the destination image frame does not include a point corresponding to the pixel, and the detector determines the area including the pixel to be an uncovered area that presents itself from behind an occluder, when the source image frame does not include a point corresponding to the pixel but the destination image frame includes a point corresponding to the pixel.

7. The image processing apparatus according to claim 6, wherein

a sum of sets of the covered area and the uncovered area is used as a mask to be applied to an image frame.

8. The image processing apparatus according to claim 7, further comprising:

an edge detector which detects an edge between an object and a background in an image frame, so as to create an edge image; and

and edge extractor which removes an edge between stationary objects in the source image frame or the destination image frame by calculating a sum of sets of the mask and the edge image, so as to extract only an edge portion between a moving object and a stationary object.

9. A computer program product comprising:

initial matching in which correspondence point information is determined for each pixel in a source image frame and in a destination image frame in image data comprising consecutive image frames;

determining a motion vector according to a result of matching and determining for each pixel the reliability of the motion vector thus determined; and

for a pixel characterized by relatively low reliability of the motion vector thus determined, performing block matching between blocks in the source image frame and in the destination image frame, and determining an updated motion vector by re-calculation, each block comprising a plurality of pixels.

10. An image processing method comprising:

computing matching between two image frames in image data comprising consecutive image frames so as to determine corresponding point information indicating correspondence between the image frames; and

determining a motion vector for each pixel according to a result of matching;

detecting, on the basis of the motion vector thus calculated, an area in which an object is hidden in a frame by another frame within the same frame and an occlusion area in which an object hides another object within the same frame; and

isolating between a stationary portion and a moving portion in an image frame, on the basis of the motion vector and the occlusion area.