Image generating apparatus, image generating method, and image generating program

• US 10,129,466 B2
• Filed: 06/28/2016
• Issued: 11/13/2018
• Est. Priority Date: 07/07/2015
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Sato, Masanori, Kodama, Kazuya
• Primary Examiner(s): Messmore, Jonathan R
• Application Number:
  US15/195,194
• Publication Number:
  US 20170013248A1
• Time in Patent Office: 868 Days
• CPC Class Codes: G02B 21/367 providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 23/815 for controlling the resolution by using a single image

Abstract

An image generating apparatus includes: an input unit that inputs data on a defocused image group; a storage unit that stores data on a plurality of filters generated based on a three-dimensional blur function; and a computing unit that generates a new image by applying the filter to the defocused image group. When a first element of a first filter out of the plurality of filters is symmetrical or anti-symmetrical with a second element included in the first filter or a second filter, which is different from the first filter, the storage unit does not store the data on the first element, and the computing unit generates the data on the first element of the first filter based on data on the second element which is stored in the storage unit when the first filter is used.

Claims

1. An image generating apparatus, comprising:

an input unit configured to input data on a defocused image group constituted by a plurality of images acquired by imaging an object for a plurality of times while changing a focal position in an optical axis direction of an imaging optical system;

a storage unit configured to store data on a plurality of filters generated on the basis of a three-dimensional blur function for approximating a blur characteristic of the imaging optical system; and

a computing unit configured to acquire data on a filter to be used from the storage unit and to generate a new image by applying the filter to the defocused image group, wherein

when a first element of a first filter out of the plurality of filters is symmetrical or anti-symmetrical with a second element included in the first filter or a second filter, which is different from the first filter, the storage unit does not store the data on the first element of the first filter, and the computing unit generates the data on the first element of the first filter on the basis of data on the second element which is stored in the storage unit when the first filter is used.

2. The image generating apparatus according to claim 1, further comprising a filter calculation unit that generates the plurality of filters on the basis of the three-dimensional blur function and stores the filters in the storage unit, wherein

the filter calculation unit does not calculate the first element of the first filter.

3. The image generating apparatus according to claim 2, wherein

the plurality of filters are a plurality of blur reconstruction filters that are applied to respective images corresponding to respective focal positions constituting the defocused image group, in order to reconstruct an image having a new blur from the defocused image group.

4. The image generating apparatus according to claim 3, wherein

the first filter is a filter to be applied to an image corresponding to a first focal position,

the second filter is a filter to be applied to an image corresponding to a second focal position that is located in a position symmetrical with the first focal position with respect to a focusing position, and

the second element is an element of the second filter and is an element located in a same position as the first element in a coordinate system of the filter.

5. The image generating apparatus according to claim 3, wherein

the three-dimensional blur function is a function that has point symmetry with respect to a focusing position, and is expressed by a set of plurality of rays that pass through the focusing position, and

the second element is an element of the first filter and is an element located in a position that is point-symmetric with the first element in a coordinate system of the filter.

6. The image generating apparatus according to claim 5, wherein

both a shape of the three-dimensional blur function and a shape of the new blur applied to an image reconstructed by the blur reconstruction filter have axial symmetry with respect to a same symmetrical axis in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and is an element located in a position that is axially symmetrical with the first element in the coordinate system of the filter.

7. The image generating apparatus according to claim 5, wherein

both a shape of the three-dimensional blur function and a shape of the new blur applied to an image reconstructed by the blur reconstruction filter have point-symmetry with respect to an origin in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the filter calculation unit does not calculate an imaginary part of elements of the plurality of filters.

8. The image generating apparatus according to claim 5, wherein

both a shape of the three-dimensional blur function and a shape of the new blur applied to an image reconstructed by the blur reconstruction filter have point-symmetry with respect to an origin in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the storage unit does not store the data on an imaginary part of elements of the plurality of filters.

9. The image generating apparatus according to claim 5, wherein

a shape of the three-dimensional blur function has axial symmetry with respect to two symmetric axes intersecting orthogonally in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and a shape of the new blur applied to an image reconstructed by the blur reconstruction filter also has axial symmetry with respect to each of the two symmetrical axes, and

the second element is an element of the first filter and is an element located in a position that is axially symmetrical with the first element in the coordinate system of the filter.

10. The image generating apparatus according to claim 5, wherein

both a shape of the three-dimensional blur function and a shape of the new blur applied to an image constructed by the blur reconstruction filter have rotational symmetry with respect to an origin in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and is an element located in a position where the first element is rotated around the origin in the coordinate system of the filter.

11. The image generating apparatus according to claim 2, wherein

the plurality of filters are a plurality of ray separating filters corresponding to a plurality of view points respectively, for extracting information on a ray passing through each view point from the defocused image group.

12. The image generating apparatus according to claim 11, wherein

the three-dimensional blur function is a function that has point symmetry with respect to a focusing position and is expressed by a set of a plurality of rays that pass through the focusing position, and

the filter calculation unit does not calculate an imaginary part of elements of the plurality of filters.

13. The image generating apparatus according to claim 12, wherein

the second element is an element of the first filter, and is an element located in a position that is point-symmetrical with the first element in a coordinate system of the filter.

14. The image generating apparatus according to claim 12, wherein

a shape of the three-dimensional blur function has rotational symmetry with respect to an origin in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and is an element located in a position that is axially symmetrical with the first element in the coordinate system of the filter.

15. The image generating apparatus according to claim 12, wherein

a shape of the three-dimensional blur function has rotational symmetry with respect to an origin in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and is an element located in a position where the first element is rotated around the origin in a coordinate system of the filter.

16. The image generating apparatus according to claim 12, wherein

a shape of the three-dimensional blur function has axial symmetry on a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and is an element located in a position axially symmetrical with the first element in a coordinate system of the filter.

17. The image generating apparatus according to claim 12, wherein

a shape of the three-dimensional blur function has point symmetry with respect to an origin in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and is an element located in a position that is point-symmetrical with the first element with respect to the origin in a coordinate system of the filter.

18. The image generating apparatus according to claim 12, wherein

a shape of the three-dimensional blur function has axial symmetry with respect to each of two symmetrical axes intersecting orthogonally in a two-dimensional coordinate system on a plane perpendicular to an optical axis direction, and

the second element is an element of the first filter, and an element located in a position that is axially symmetrical with the first element in a coordinate system of the filter.

19. An image generating method, comprising:

a step in which a computer inputs data on a defocused image group constituted by a plurality of images acquired by imaging an object for a plurality of times while changing a focal position in an optical axis direction of an imaging optical system; and

a computing step in which the computer acquires data on a filter to be used from a storage unit, which stores data on a plurality of filters generated on the basis of a three-dimensional blur function for approximating a blur characteristic of the imaging optical system, and generates a new image by applying the filter to the defocused image group, wherein

when a first element of a first filter out of the plurality of filters is symmetrical or anti-symmetrical with a second element that is included in the first filter or a second filter, which is different from the first filter, the storage unit does not store the data on the first element of the first filter, and in the computing step, the data on the first element of the first filter is generated on the basis of data on the second element, which is stored in the storage unit, when the first filter is used.

20. A non-transitory computer readable storage medium storing a program causing a computer to execute:

a step of inputting data on a defocused image group constituted by a plurality of images acquired by imaging an object for a plurality of times while changing a focal position in an optical axis direction of an imaging optical system; and

a computing step of acquiring data on a filter to be used from a storage unit storing data on a plurality of filters generated on the basis of a three-dimensional blur function for approximating a blur characteristic of the imaging optical system, and generating a new image by applying the filter to the defocused image group, wherein

when a first element of a first filter out of the plurality of filters is symmetrical or anti-symmetrical with a second element included in the first filter or a second filter, which is different from the first filter, the storage unit does not store the data on the first element of the first filter, and in the computing step, the data on the first element of the first filter is generated on the basis of data on the second element, which is stored in the storage unit, when the first filter is used.