Estimating and removing lens distortion from scenes

• US 7,912,317 B2
• Filed: 10/29/2005
• Issued: 03/22/2011
• Est. Priority Date: 10/29/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Souchard, Christophe
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US11/261,094
• Publication Number:
  US 20070098296A1
• Time in Patent Office: 1,970 Days
• Field of Search: 382/275, 382/100, 382/254, 382/255, 382260-266, 382/274, 345/427, 345/428, 345/581, 345/611, 345/619, 345/647, 345650-652, 345/655, 345/661, 348/207.99, 348/208.4, 348/208.13, 348/241, 348/252, 348/333.12
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

Some embodiments provide a method of editing images with distortion caused by a camera lens. The method identifies a set of geometries on a set of images taken by the camera lens. The method finds transformations for applying on the set of identified geometries to remove distortion on the geometries. The method applies the transformations to images taken by the camera lens to remove distortions. In some embodiments the identified geometries are curvilinear lines that correspond to straight lines in the scene being captured by the lens. Some embodiments provide a method of matching distortion for adding graphical objects to images. The method first removes the distortion on the images as described above. The method then adds the graphical objects to these images. Finally, the method applies inverse transformations on the images to restore them to their original form.

Claims

1. A method of editing images with distortion caused with a camera lens, the method comprising:

in a display area of a user interface, displaying an image with said distortion, said image comprising a scene captured with said camera lens;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s identification of a set of graphs that corresponds to a set of geometries that appear distorted in said image, wherein the set of geometries comprises a curvilinear line that corresponds to substantially a straight line in said scene that appears curvilinear in said image;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein said displaying, receiving the selection, receiving the identification, determining, and applying are performed on a computing device by an image editing application that comprises said user interface.

2. The method of claim 1, wherein the plurality of images are a sequence of video images taken with said camera lens.

3. The method of claim 1, wherein the plurality of images comprises said image displayed in the display area of said user interface.

4. The method of claim 1, wherein the set of graphs comprises at least two graphs and the set of geometries comprises two geometries that correspond to the two graphs.

5. The method of claim 1, wherein receiving the user'"'"'s identification of the set of graphs comprises (i) receiving user input through said distortion removal tool, and (ii) drawing, based on the user input, at least one curvilinear line on said image that is displayed in said display area.

6. The method of claim 1, wherein a camera that captures still images comprises said camera lens, wherein the plurality of images comprises at least one still image taken with said camera.

7. The method of claim 1, wherein the plurality of images are images of a video and said image editing application is a video editing application.

8. The method of claim 1 further comprising:

adding a graphical object to a particular image in the plurality of images; and

applying an inverse of a particular transformation to said graphical object in order to distort the graphical object in accordance with the set of geometries that appear distorted in said image.

9. The method of claim 1, wherein the distortion removal tool allows the user to specify the set of graphs by inputting a plurality of locations on said image displayed in said display area.

10. A method of editing images with distortion caused with a camera lens, the method comprising:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool comprising a spline tool of an image editing application;

receiving the user'"'"'s identification of a set of geometries that appear distorted in said image by drawing a set of graphs on the image that corresponds to the set of geometries based on user input received through said spline tool;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein said displaying, receiving the selection, receiving the identification, determining, and applying are performed on a computing device by the image editing application that comprises said user interface.

11. The method of claim 10, wherein the plurality of images are a sequence of video images taken with said camera lens.

12. The method of claim 10, wherein (i) said image displayed in the display area comprises a scene captured with said camera lens, and (ii) the set of geometries comprises a curvilinear line that corresponds to substantially a straight line in said scene but appears curvilinear in said image.

13. A method of editing images with distortion caused with a camera lens, the method comprising:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s input of a plurality of locations on said image;

drawing a set of graphs comprising a curvilinear line on the image displayed in said display area based on the input of the plurality of locations, said set of graphs corresponding to a set of geometries that appear distorted in said image;

based on said input, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein said displaying, receiving the selection, receiving the input, drawing, determining, and applying are performed on a computing device by an image editing application that comprises said user interface.

14. The method of claim 13, wherein the plurality of images are a sequence of video images taken with said camera lens.

15. The method of claim 13, wherein the plurality of images are images of a video and said image editing application is a video editing application.

16. A method of editing images with distortion caused with a camera lens, the method comprising:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s input of a plurality of locations that identifies a set of graphs corresponding to a set of geometries that appear distorted in said image, said input comprising adding, deleting, or moving one or more points;

based on said input, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein said displaying, receiving the selection, receiving the input, determining, and applying are performed on a computing device by an image editing application that comprises said user interface.

17. The method of claim 16, wherein the plurality of images are a sequence of video images taken with said camera lens.

18. The method of claim 16, wherein a camera that captures still images comprises said camera lens, wherein the plurality of images comprises at least one still image taken with said camera.

19. A method of editing images with distortion caused with a camera lens, the method comprising:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s identification of a set of graphs that corresponds to a set of geometries that appear distorted in said image, wherein the set of graphs comprises only one graph and the set of geometries comprises one geometry that corresponds to said one graph;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein said displaying, receiving the selection, receiving the identification, determining, and applying are performed on a computing device by an image editing application that comprises said user interface.

20. The method of claim 19, wherein the plurality of images are a sequence of video images taken with said camera lens.

21. The method of claim 19, further comprising drawing, based on the user'"'"'s identification, at least one curvilinear line on said image that is displayed in said display area.

22. A method of editing images with distortion caused with a camera lens, the method comprising:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s identification of a set of graphs that corresponds to a set of geometries that appear distorted in said image;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein applying the transformations comprises resampling the plurality of images based on the transformations in order to produce a resampled plurality of images without said distortion, wherein said displaying, receiving the selection, receiving the identification, determining, and applying are performed on a computing device by an image editing application that comprises said user interface.

23. The method of claim 22, wherein the plurality of images are a sequence of video images taken with said camera lens.

24. The method of claim 22, further comprising:

adding a graphical object to a particular image in the plurality of images; and

applying an inverse of a particular transformation to said graphical object in order to distort the graphical object in accordance with the set of geometries that appear distorted in said image.

25. A computer readable medium storing a computer program that when executed by at least one processor edits images with distortion caused with a camera lens, said computer program comprising sets of instructions for:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s identification of a set of graphs that corresponds to a set of geometries that appear distorted in said image, wherein a particular graph in the set of graphs identifies a curvilinear line in the image that should appear as a straight line;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens.

26. The computer readable medium of claim 25, wherein the plurality of images are a sequence of video images taken with said camera lens.

27. The computer readable medium of claim 25, wherein the plurality of images comprises said image displayed in the display area of said user interface.

28. The computer readable medium of claim 25, wherein the set of graphs comprises at least two graphs and the set of geometries comprises two geometries that correspond to the two graphs.

29. The computer readable medium of claim 25, wherein the set of instructions for receiving the user'"'"'s identification of the set of graphs comprises sets of instructions for (i) receiving user input through said distortion removal tool, and (ii) drawing, based on the user input, at least one curvilinear line on said image that is displayed in said display area.

30. The computer readable medium of claim 25, wherein a camera that captures still images comprises said camera lens, wherein the plurality of images comprises at least one still image taken with said camera.

31. The computer readable medium of claim 25, wherein the plurality of images are images of a video and said computer program is a video editing application.

32. The computer readable medium of claim 25, wherein the computer program further comprises sets of instructions for:

adding a graphical object to a particular image in the plurality of images; and

applying an inverse of a particular transformation to said graphical object in order to distort the graphical object in accordance with the set of geometries that appear distorted in said image.

33. The computer readable medium of claim 25, wherein the distortion removal tool allows the user to specify the set of graphs by inputting locations for a plurality of points on said image displayed in said display area.

34. A computer readable medium storing a computer program that when executed by at least one processor edits images with distortion caused with a camera lens, said computer program comprising sets of instructions for:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool comprising a spline tool of the computer program;

receiving the user'"'"'s identification of a set of geometries that appear distorted in said image by drawing a set of graphs on the image that corresponds to the set of geometries based on user input received through said spline tool;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens.

35. The computer readable medium of claim 34, wherein the plurality of images are a sequence of video images taken with said camera lens.

36. The computer readable medium of claim 34, wherein the set of geometries comprises a curvilinear line that corresponds to substantially a straight line in said scene but appears curvilinear in said image.

37. A computer readable medium storing a computer program that when executed by at least one processor edits images with distortion caused with a camera lens, said computer program comprising sets of instructions for:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s input of a plurality of locations on said image;

drawing a set of graphs comprising a curvilinear line on the image displayed in said display area based on the input of the locations for a plurality of points, said set of graphs corresponding to a set of geometries that appear distorted in said image;

based on said input, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens.

38. The computer readable medium of claim 37, wherein the plurality of images are a sequence of video images taken with said camera lens.

39. The computer readable medium of claim 37, wherein a camera that captures still images comprises said camera lens, wherein the plurality of images comprises at least one still image taken with said camera.

40. A computer readable medium storing a computer program that when executed by at least one processor edits images with distortion caused with a camera lens, said computer program comprising sets of instructions for:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s input of a plurality of locations that identifies a set of graphs corresponding to a set of geometries that appear distorted in said image, said input comprising adding, deleting, or moving one or more points;

based on said input, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens.

41. The computer readable medium of claim 40, wherein the plurality of images are a sequence of video images taken with said camera lens.

42. The computer readable medium of claim 40, wherein the computer program further comprises sets of instructions for:

adding a graphical object to a particular image in the plurality of images; and

applying an inverse of a particular transformation to said graphical object in order to distort the graphical object in accordance with the set of geometries that appear distorted in said image.

43. A computer readable medium storing a computer program that when executed by at least one processor edits images with distortion caused with a camera lens, said computer program comprising sets of instructions for:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s identification of a set of graphs that corresponds to a set of geometries that appear distorted in said image, wherein the set of graphs comprises only one graph and the set of geometries comprises one geometry that corresponds to said one graph;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens.

44. The computer readable medium of claim 43, wherein the plurality of images are a sequence of video images taken with said camera lens.

45. The computer readable medium of claim 43, wherein the computer program further comprises a set of instructions for drawing, based on the user'"'"'s identification, at least one curvilinear line on said image that is displayed in said display area.

46. A computer readable medium storing a computer program that when executed by at least one processor edits images with distortion caused with a camera lens, said computer program comprising sets of instructions for:

in a display area of a user interface, displaying an image with said distortion;

receiving, in said user interface, a user'"'"'s selection of a distortion removal tool;

through said distortion removal tool, receiving the user'"'"'s identification of a set of graphs that corresponds to a set of geometries that appear distorted in said image;

based on said identification, determining transformations for applying on a plurality of images to remove said distortion; and

applying the transformations to said plurality of images to remove said distortion caused with the camera lens, wherein applying the transformations comprises resampling the plurality of images based on the transformation in order to produce a resampled plurality of images without said distortion.

47. The computer readable medium of claim 46, wherein the plurality of images are a sequence of video images taken with said camera lens.

48. The computer readable medium of claim 46, wherein the computer program further comprises a set of instructions for drawing, based on the user'"'"'s identification, at least one curvilinear line on said image that is displayed in said display area.