Image filling methods

• US 8,009,899 B2
• Filed: 06/22/2007
• Issued: 08/30/2011
• Est. Priority Date: 02/15/2007
• Status: Expired due to Fees

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Liu, Chi-Tsung, Wu, Yi-Leh, Chen, Chia-Chen, Tang, Cheng-Yuan
• Primary Examiner(s): Desire; Gregory M
• Application Number:
  US11/812,942
• Publication Number:
  US 20080199083A1
• Time in Patent Office: 1,530 Days
• Field of Search: 345/419, 345/422, 345/629, 345/633, 382/103, 382/107, 382/154, 382/254
• US Class Current: 382/154
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19

Abstract

Image filling methods. A plurality of images corresponding to a target object or a scene are captured at various angles. An epipolar geometry relationship between a filling source image and a specific image within the images is calculated. The filling source image and the specific image are rectified according to the epipolar geometry relationship. At least one filling target area in the rectified specific image is patched according to the rectified filling source image.

Claims

1. A computer-implemented image filling method for use in a computer, wherein the computer is programmed to perform the steps of:

capturing a plurality of images corresponding to a target object or a scene at various angles;

calculating an epipolar geometry relationship between a filling source image and a specific image within the images;

rectifying the filling source image and the specific image according to the epipolar geometry relationship; and

patching at least one filling target area in the rectified specific image according to the rectified filling source image,

wherein the method of calculating the epipolar geometry relationship comprises the steps of;

determining a plurality of first characteristic points in the specific image;

determining a plurality of second characteristic points in the filling source image;

determining a plurality of corresponding points within the first characteristic points and the second characteristic points;

determining an optimum group of the corresponding points for the specific image and the filling source image; and

calculating a fundamental matrix for the epipolar geometry relationship according to the optimum group of the corresponding points.

2. The method of claim 1 further comprising:

determining a specific region in the specific image and the filling source image; and

determining the corresponding points fall in the specific region within the first characteristic points and the second characteristic points.

3. The method of claim 1 further comprising determining an optimum group of the corresponding points for the specific image and the filling source image according to LMedS (Least Median of Squares) or RANSAC (Random Sample Consensus).

4. The method of claim 1 wherein the method of patching at least one filling target area in the rectified specific image according to the rectified filling source image comprises:

obtaining a patch block from the filling source image; and

patching the filling target area in the specific image using the patch block.

5. The method of claim 4 further comprising obtaining a patch block from the filling source image according to an epipolar line corresponding to the filling target area in the specific image.

6. The method of claim 5 further comprising:

respectively calculating similarities of a plurality of candidate patch blocks on the epipolar line in the filling source image and the filling target area; and

selecting the candidate patch block with highest similarity to the patch block.

7. The method of claim 6 further comprising calculating the similarity of respective candidate patch blocks according to a block-matching method.

8. The method of claim 1 further comprising calculating a priority of the at least one filling target area to be patched, wherein the method of calculating the priority of the at least one filling target area to be patched comprises:

calculating an edge intensity of the at least one filling target area; and

determining the priority of the at least one filling target area according to the edge intensity.

9. The method of claim 8 further comprising:

obtaining a contour of the at least one filling target area; and

calculating the edge intensity of the at least one filling target area at the contour.

10. The method of claim 1 further comprising adjusting the colors of the at least one filling target area.

11. The method of claim 1 further comprising returning the rectified specific image with image patched according to the epipolar geometry relationship.

12. The method of claim 1 further comprising:

determining whether the image filling for the specific image is complete; and

if not, selecting a new filling source image within the images, calculating an epipolar geometry relationship between the new filling source image and the specific image, rectifying the new filling source image and the specific image according to the epipolar geometry relationship, and patching at least one filling target area in the rectified specific image according to the rectified new filling source image.

13. A machine-readable storage medium comprising a program, which, when executed, causes a device to perform an image filling method, the method comprising:

capturing a plurality of images corresponding to a target object or a scene at various angles;

calculating an epipolar geometry relationship between a filling source image and a specific image within the images;

rectifying the filling source image and the specific image according to the epipolar geometry relationship; and

patching at least one filling target area in the rectified specific image according to the rectified filling source image,

wherein the method of calculating the epipolar geometry relationship comprises;

determining a plurality of first characteristic points in the specific image;

determining a plurality of second characteristic points in the filling source image;

determining a plurality of corresponding points within the first characteristic points and the second characteristic points.

determining an optimum group of the corresponding points for the specific image and the filling source image; and

calculating a fundamental matrix for the epipolar geometry relationship according to the optimum group of the corresponding points.

14. The storage medium of claim 13 wherein the method further comprises:

determining a specific region in the specific image and the filling source image; and

determining the corresponding points in the specific region within the first characteristic points and the second characteristic points.

15. The storage medium of claim 13 wherein the method further comprises determining an optimum group of the corresponding points for the specific image and the filling source image according to LMedS (Least Median of Squares) or RANSAC (Random Sample Consensus).

16. The storage medium of claim 13 wherein the method of patching at least one filling target area in the rectified specific image according to the rectified filling source image comprises:

obtaining a patch block from the filling source image; and

patching the filling target area in the specific image using the patch block.

17. The storage medium of claim 16 wherein the method further comprises obtaining a patch block from the filling source image according to an epipolar line corresponding to the filling target area in the specific image.

18. The storage medium of claim 17 wherein the method further comprises:

respectively calculating similarities of a plurality of candidate patch blocks on the epipolar line in the filling source image and the filling target area; and

selecting the candidate patch block with highest similarity to the patch block.

19. The storage medium of claim 18 wherein the method further comprises calculating the similarity of respective candidate patch blocks according to a block-matching method.

20. The storage medium of claim 13 wherein the method further comprises calculating a priority of the at least one filling target area to be patched, wherein the method of calculating the priority of the at least one filling target area to be patched comprises:

calculating an edge intensity of the at least one filling target area; and

determining the priority of the at least one filling target area according to the edge intensity.

21. The storage medium of claim 20 wherein the method further comprises:

obtaining a contour of the at least one filling target area; and

calculating the edge intensity of the at least one filling target area at the contour.

22. The storage medium of claim 13 wherein the method further comprises adjusting the colors of the at least one filling target area.

23. The storage medium of claim 13 wherein the method further comprises returning the rectified specific image with image patched according to the epipolar geometry relationship.

24. The storage medium of claim 13 wherein the method further comprises:

determining whether the image filling for the specific image is complete; and

if not, selecting a new filling source image within the images, calculating an epipolar geometry relationship between the new filling source image and the specific image, rectifying the new filling source image and the specific image according to the epipolar geometry relationship, and patching at least one filling target area in the rectified specific image according to the rectified new filling source image.