Method and device of image processing and camera

• US 10,594,941 B2
• Filed: 01/31/2018
• Issued: 03/17/2020
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Yu, Lifu, Li, Zefei, Cao, Zisheng, Wang, Mingyu
• Primary Examiner(s): Haskins, Twyler L
• Assistant Examiner(s): Tejano, Dwight Alex C
• Application Number:
  US15/884,985
• Publication Number:
  US 20180160045A1
• Time in Patent Office: 776 Days
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

An image processing method includes obtaining an acquired image captured by an image acquisition device and obtaining vibration information associated with the acquired image and generated while the image acquisition device capturing the acquired image. The vibration information includes an angle of vibration. The method further includes performing a distortion correction to the acquired image to obtain a distortion-corrected image based upon the vibration information and a preset distortion correction parameter, and determining a target image from the distortion-corrected image based upon the vibration information.

Claims

1. An image processing method comprising:

obtaining an acquired image captured by an image acquisition device;

obtaining vibration information associated with the acquired image and generated while the image acquisition device capturing the acquired image, the vibration information comprising an angular change;

determining a mapping from an image coordinate system in which the acquired image is positioned to a target coordinate system based upon the angular change;

performing a distortion correction to the acquired image to obtain the distortion-corrected image based upon the mapping from the image coordinate system to the target coordinate system and the preset distortion correction parameter; and

determining a target image from the distortion-corrected image based upon the vibration information;

wherein determining the mapping from the image coordinate system to the target coordinate system based upon the angular change comprises;

establishing a mapping from the image coordinate system to a world coordinate system based upon the angular change;

determining a mapping from the target coordinate system to the world coordinate system based upon the angular change; and

determining the mapping from the image coordinate system to the target coordinate system based upon the world coordinate system;

wherein determining the mapping from the target coordinate system to the world coordinate system based upon the angular change comprises;

establishing a mapping from the world coordinate system to a preset tangent plane coordinate system based upon the angular change;

establishing a mapping from the target coordinate system to the preset tangent plane coordinate system based upon the angular change; and

determining the mapping from the target coordinate system to the world coordinate system based upon the preset tangent plane coordinate system.

2. The method of claim 1, wherein determining the target image from the distortion-corrected image based upon the vibration information comprises:

determining an initial region from the distortion-corrected image,

performing a vibration compensation to the initial region based upon the vibration information to determine a target region, and

obtaining the target image corresponding to the target region.

3. The method of claim 2, wherein determining the initial region from the distortion-corrected image comprises:

determining the initial region from the distortion-corrected image based upon a preset angle of view.

4. The method of claim 1, wherein determining the target image from the distortion-corrected image based upon the vibration information comprises:

determining an image from the distortion-corrected image as the target image based upon the vibration information and a preset angle of view, the image corresponding to an area defined by the angle of view.

5. The method of claim 1, wherein:

the angular change includes a change of the position of the image acquisition device in capturing the acquired image relative to the position of the image acquisition device in capturing a previously acquired image or relative to a reference position, and

the angular change comprises at least one of a pitch angle, a yaw angle, or a roll angle.

6. The method of claim 1, further comprising, before performing the distortion correction to the acquired image based upon the vibration information and the preset distortion correction parameter:

performing a synchronization to the acquired image and the vibration information.

7. The method of claim 1, wherein the acquired image includes a fisheye image.

8. The method of claim 1, wherein performing the distortion correction to the acquired image to obtain the distortion-corrected image based upon the mapping from the image coordinate system to the target coordinate system and the preset distortion correction parameter comprises:

mapping a plurality of points in the acquired image to corresponding locations in the target coordinate system based upon the determined mapping from the image coordinate system to the target coordinate system and the distortion correction parameter;

obtaining pixel values of the points in the acquired image; and

generating the distortion-corrected image based upon the pixel values and the locations of the points in the target coordinate system.

9. An image processing device comprising:

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to;

obtain an acquired image captured by an image acquisition device;

obtain vibration information associated with the acquired image and generated while the image acquisition device capturing the acquired image, the vibration information comprising an angular change;

establish a mapping from an image coordinate system in which the acquired image is positioned to a world coordinate system based upon the angular change;

determine a mapping from a target coordinate system to the world coordinate system based upon the angular change;

establish a mapping from the world coordinate system to a preset tangent plane coordinate system based upon the angular change;

establish a mapping from the target coordinate system to the preset tangent plane coordinate system based upon the angular change;

determine a mapping from the target coordinate system to the world coordinate system based upon the preset tangent plane coordinate system;

determine a mapping from the image coordinate system to the target coordinate system based upon the world coordinate system;

perform a distortion correction to the acquired image to obtain the distortion-corrected image based upon the mapping from the image coordinate system to the target coordinate system and the preset distortion correction parameter; and

determine a target image from the distortion-corrected image based upon the vibration information.

10. The device of claim 9, wherein the instructions further cause the processor to:

determine an initial region from the distortion-corrected image;

perform a vibration compensation to the initial region based upon the vibration information to determine a target region; and

obtain the target image corresponding to the target region.

11. The device of claim 10, wherein the instructions further cause the processor to:

determine the initial region from the distortion-corrected image based upon a preset angle of view.

12. The device of claim 9, wherein the instructions further cause the processor to:

determine an image from the distortion-corrected image as the target image based upon the vibration information and a preset angle of view, the image corresponding to an area defined by the angle of view.

13. The device of claim 9, wherein:

the angular change includes a change of the position of the image acquisition device in capturing the acquired image relative to the position of the image acquisition device in capturing a previously acquired image or relative to a reference position, and

the angular change comprises at least one of a pitch angle, a yaw angle, or a roll angle.

14. The device of claim 9, wherein the instructions further cause the processor to, before performing the distortion correction to the acquired image based upon the vibration information and the preset distortion correction parameter:

perform a synchronization to the acquired image and the vibration information.

15. The device of claim 9, wherein the instructions further cause the processor to:

map a plurality of points in the acquired image to corresponding locations in the target coordinate system based upon the determined mapping from the image coordinate system to the target coordinate system and the distortion correction parameter;

obtain pixel values of the points in the acquired image; and

generate the distortion-corrected image based upon the pixel values and the locations of the points in the target coordinate system.

16. An aerial vehicle comprising:

a body;

an image acquisition device mounted on the body; and

an image processing device of claim 9 in data communication with the image acquisition device.