Image processing technique for generating more appropriate motion vectors for a captured image that is distorted

• US 10,038,847 B2
• Filed: 04/12/2016
• Issued: 07/31/2018
• Est. Priority Date: 04/13/2015
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Matsuoka, Masaaki
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Trehan, Akshay
• Application Number:
  US15/096,458
• Publication Number:
  US 20160301870A1
• Time in Patent Office: 840 Days
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A generation unit generates N distortion corrected images (N≥2) by performing coordinate conversion on each of N partial areas of a first captured image, using a conversion parameter that depends on a position of the partial area, and generates N distortion corrected images by performing coordinate conversion on each of corresponding N partial areas of a second captured image, using a conversion parameter that depends on a position of the partial area. A detection unit detects N motion vectors by comparing each of the N distortion corrected images generated from the first captured image with the corresponding distortion corrected image generated from the second captured image. A conversion unit generates N converted motion vectors by performing inverse coordinate conversion on each of the N motion vectors using a conversion parameter that depends on a position of the corresponding partial area.

Claims

1. An image processing apparatus comprising:

one or more processors configured to execute one or more programs so as to function as units comprising;

(1) an acquisition unit configured to acquire a first captured image and a second captured image generated by photo-electrically converting a subject image formed by an optical system;

(2) a generation unit configured to generate N distortion corrected images, N being an integer greater than or equal to 2, by performing coordinate conversion on each of N partial areas of the first captured image, using a conversion parameter that depends on a position of the partial area, and to generate N distortion corrected images by performing coordinate conversion on each of corresponding N partial areas of the second captured image, using a conversion parameter that depends on a position of the partial area;

(3) a detection unit configured to detect N motion vectors by comparing each of the N distortion corrected images generated from the first captured image with the corresponding distortion corrected image generated from the second captured image; and

(4) a conversion unit configured to generate N converted motion vectors by performing inverse coordinate conversion on each of the N motion vectors, using a conversion parameter that depends on a position of the corresponding partial area.

2. The image processing apparatus according to claim 1, wherein the one or more processors further function as units comprising:

an output unit configured to output a first image corresponding to a first partial area of the first captured image; and

an identification unit configured to identify a second partial area corresponding to a movement destination of the first partial area in the second captured image based on the N converted motion vectors,

wherein the output unit outputs a second image corresponding to the second partial area of the second captured image.

3. The image processing apparatus according to claim 2, wherein the output unit generates a first distortion corrected image as the first image by performing coordinate conversion on the first partial area, using a conversion parameter that depends on a position of the first partial area, and generates a second distortion corrected image as the second image by performing coordinate conversion on the second partial area, using a conversion parameter that depends on a position of the second partial area.

4. The image processing apparatus according to claim 2, wherein the identification unit calculates the movement destination of the first partial area based on an affine transformation of the N converted motion vectors.

5. The image processing apparatus according to claim 2, wherein each of the N partial areas of the first captured image is smaller than the first partial area, and

wherein each of the N partial areas of the second captured image is smaller than the second partial area.

6. The image processing apparatus according to claim 1, wherein the first captured image and the second captured image are circular images corresponding to a hemispherical field of view, and

wherein the conversion parameter include an azimuth angle and a zenith angle.

7. An image capturing apparatus comprising:

(A) one or more processors configured to execute one or more programs so as to function as units comprising;

(1) an acquisition unit configured to acquire a first captured image and a second captured image generated by photo-electrically converting a subject image formed by an optical system;

(2) a generation unit configured to generate N distortion corrected images, being an integer greater than or equal 2, to by performing coordinate conversion on each of N partial areas of the first captured image, using a conversion parameter that depends on a position of the partial area, and to generate N distortion corrected images by performing coordinate conversion on each of corresponding N partial areas of the second captured image, using a conversion parameter that depends on a position of the partial area;

(3) a detection unit configured to detect N motion vectors by comparing each of the N distortion corrected images generated from the first captured image with the corresponding distortion corrected image generated from the second captured image; and

(4) a conversion unit configured to generate N converted motion vectors by performing inverse coordinate conversion on each of the N motion vectors, using a conversion parameter that depends on a position of the corresponding partial area; and

(B) an image capturing unit configured to photo-electrically convert a subject image formed by the optical system.

8. A control method of an image processing apparatus, comprising:

acquiring a first captured image and a second captured image generated by photo-electrically converting a subject image formed by an optical system;

generating N distortion corrected images, N being an integer greater than or equal to 2, by performing coordinate conversion on each of N partial areas of the first captured image, using a conversion parameter that depends on a position of the partial area;

generating N distortion corrected images by performing coordinate conversion on each of corresponding N partial areas of the second captured image, using a conversion parameter that depends on a position of the partial area;

detecting N motion vectors by comparing each of the N distortion corrected images generated from the first captured image with the corresponding distortion corrected image generated from the second captured image; and

generating N converted motion vectors by performing inverse coordinate conversion on each of the N motion vectors, using a conversion parameter that depends on a position of the corresponding partial area.

9. A non-transitory computer-readable storage medium which stores a program for causing a computer to execute a control method comprising:

acquiring a first captured image and a second captured image generated by photo-electrically converting a subject image formed by an optical system;

generating N distortion corrected images, N being an integer greater than or equal to 2, by performing coordinate conversion on each of N partial areas of the first captured image, using a conversion parameter that depends on a position of the partial area;

generating N distortion corrected images by performing coordinate conversion on each of corresponding N partial areas of the second captured image, using a conversion parameter that depends on a position of the partial area;

detecting N motion vectors by comparing each of the N distortion corrected images generated from the first captured image with the corresponding distortion corrected image generated from the second captured image; and

generating N converted motion vectors by performing inverse coordinate conversion on each of the N motion vectors, using a conversion parameter that depends on a position of the corresponding partial area.