Image processing apparatus and image processing method, and storage medium

• US 6,784,927 B1
• Filed: 12/17/1998
• Issued: 08/31/2004
• Est. Priority Date: 12/22/1997
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Itokawa, Osamu
• Primary Examiner(s): Garber, Wendy R.
• Assistant Examiner(s): MISLEH, JUSTIN P
• Application Number:
  US09/213,953
• Time in Patent Office: 2,084 Days
• Field of Search: 348/208.99, 348/208.1, 348/208.3, 348/208.6, 348/208.14, 348/169, 348/170, 348/699, 348/171, 348/172, 348/208.4, 382/103
• US Class Current: 348/208.1
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

An image processing apparatus for correcting the position of a specific object included in a moving image formed by a plurality of successive images to a predetermined position on the frame, sets a reference point on the object included in the first image of the plurality of images, which serves as a reference image, detects the positions of points, which correspond to the reference point, from a plurality of second images, which are different from the first image, of the plurality of images, detects the shift amounts between the corresponding points and the reference point in units of images on the basis of the positions of the corresponding points detected from the plurality of second images, and the position of the reference point, and corrects the positions of the object included in the plurality of second images in correspondence with the shift amounts detected in units of images.

Claims

1. An image processing apparatus for correcting a position of a specific object included in a moving image formed by a plurality of successive images to a predetermined position on a frame, comprising:

reference point setting means for setting a reference point on the object included in a first image which serves as a reference image;

position detection means for detecting a position of a corresponding point, which corresponds to the reference point in the first image, in each of a plurality of second images which are different from the first image;

shift amount detection means for detecting a shift amount of the object between the first image and each of the second images on the basis of a distance between the position of the corresponding point detected from each of the plurality of second images, and a position of the reference point; and

corrected image generation means for generating a corrected image by correcting position of the object included in each of the plurality of second images in accordance with the shift amounts detected by said shift amount detection means in units of the second images,

wherein said corrected image generation means includes;

image shift means for shifting the plurality of second images in correspondence with the shift amounts detected by said shift amount detection means in units of images; and

image synthesis means for selecting images corresponding to missing portions, which have formed with respect to original recording regions of the plurality of second images upon shifting the second images by said images shift means, from the plurality of second images, and synthesizing the selected images as the images of the missing portions;

wherein said image synthesis means includes pattern matching means for dividing the second image as an object from which the images of the missing portion is to be selected, into a plurality of blocks, and selecting the image of the missing portion on the basis of matching results between the respective blocks and an image having the missing portion and for using a shift amount of a background portion obtained by excluding the object from each of the plurality of second image as a reference for matching.

2. The apparatus according to claim 1, further comprising display means for displaying the corrected image, which is corrected by said corrected image generation means.

3. The apparatus according to claim 1, wherein said corrected image generation means further includes:

maximum shift amount detection means for detecting maximum shift amounts in upper, lower, right, and left directions from the shift amounts produced with respect to original recording regions of the plurality of second images upon shifting the second images by said image shift means; and

predetermined image insertion means for generating the corrected image of the moving image by inserting a predetermined image corresponding to the maximum shift amounts in the upper, lower, right, and left directions detected by said maximum shift amount detection means in the first image and the plurality of second images, which have been shifted by said image shift means.

4. The apparatus according to claim 1, wherein the plurality of images are frame images.

5. An image processing method for correcting a position of a specific object included in a moving image formed by a plurality of successive images to a predetermined position on a frame, comprising:

a reference point setting step of setting a reference point on the object included in a first image which serves as a reference image;

a position detection step of detecting a positions of a corresponding point, which correspond to the reference point, in the first image, in each of a plurality of second images, which are different from the first image;

a shift amount detection step of detecting a shift amount of the object between the first image and each of the second images on the basis of a distance between the position of the corresponding point detected from each of the plurality of second images, and a position of the reference point; and

a corrected image generation step of generating a corrected image of the moving image by correcting position of the object included in each of the plurality of second images in accordance with the shift amounts detected in units of the second images,

wherein said corrected image generation step includes;

an image shift step for shifting the plurality of second images in correspondence with the shift amounts detected by said shift amount detection step in units of images; and

an image synthesis step for selecting images corresponding to missing portions, which have formed with respect to original recording regions of the plurality of second images upon shifting the second images by said images shift step, from the plurality of second images, and synthesizing the selected images as the images of the missing portions;

wherein said image synthesis step includes a pattern matching step for dividing the second image as an object from which the images of the missing portion is to be selected, into a plurality of blocks, and selecting the image of the missing portion on the basis of matching results between the respective blocks and an image having the missing portion and for using a shift amount of a background portion obtained by excluding the object from each of the plurality of second image as a reference for matching.

6. The method according to claim 5, wherein the corrected image generation step further includes:

a maximum shift amount detection step of detecting maximum shift amounts in upper, lower, right, and left directions from the shift amounts produced with respect to original recording regions of the plurality of second images upon shifting the second images in the images shift step; and

a predetermined images insertion step of generating the corrected image of the moving image by inserting a predetermined image corresponding to the maximum shift amounts in the upper, lower, right, and left directions detected in the maximum shift amount detection step in the first image and the plurality of second images, which have been shifted in the image shift step.

7. A computer-readable storage medium, which stores an image processing program for correcting a position of a specific object included in a moving image formed by a plurality of successive images to a predetermined position on a frame, said image processing program having:

a code of a reference point setting step of setting a reference point on the object included in a first image which serves as a reference image;

a code of a position detection step of detecting a position of a corresponding point, which corresponds to the reference point in the first image in each of a plurality of second images, which are different from the first image;

a code of a shift amount detection step of detecting a shift amount of the object between the first image and each of the second images on the basis of a distance between the positions of the corresponding points detected from each of the plurality of second images, and a position of the reference point; and

a code of a corrected image generation step of generating a corrected image by correcting position of the object included in each of the plurality of second images in accordance with the shift amount detected in units of the second images,

wherein the code of corrected image generation step includes;

a code of an image shift step of shifting the plurality of second images in correspondence with the shift amounts detected in said shift amount detection step in units of images; and

a code of an image synthesis step of selecting images corresponding to missing portions, which have formed with respect to original recording regions of the plurality of second images upon shifting the second images in said image shift step, from the plurality of second images, and synthesizing the selected images as the images of the missing portions,

wherein the code of the image synthesis step includes a code of a pattern matching step for dividing the second image as an object from which the images of the missing portion is to be selected, into a plurality of blocks, and selecting the image of the missing portion on the basis of matching results between the respective blocks and an image having the missing portion and for using a shift amount of a background portion obtained by excluding the object from each of the plurality of second image as a reference for matching.

8. The medium according to claim 7, wherein the code of the corrected image generation step further includes:

a code of a maximum shift amount detection step of detecting maximum shift amounts in upper, lower, right, and left directions from the shift amounts produced with respect to original recording regions of the plurality of second images upon shifting the second images in the images shifting step; and

a code of a predetermined image insertion step of generating the corrected image of the moving image by inserting a predetermined image corresponding to the maximum shift amounts in the upper, lower, right, and left directions detected in the maximum shift amount detection step in the first image and the plurality of second images, which have been shifted in the image shift step.