Image processing apparatus and method, data processing apparatus and method, and program and recording medium

• US 8,538,139 B2
• Filed: 07/30/2010
• Issued: 09/17/2013
• Est. Priority Date: 07/31/2009
• Status: Expired due to Fees

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Kameyama, Hirokazu
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): PARK, SOO JIN
• Application Number:
  US12/847,489
• Publication Number:
  US 20110026849A1
• Time in Patent Office: 1,145 Days
• Field of Search: 382/159, 382/254
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 10/76 based on eigen-space representations, e.g. from pose or different illumination conditions; Shape manifolds
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/175 Static expression

Abstract

The present invention determines the adopting ratio (weight coefficient) between the high image quality processing using the tensor projection method and the high image quality processing using another method according to the degree of deviation of the input condition of the input image, and combines these processes as appropriate. This allows a satisfactory reconstruction image to be acquired even in a case of deviation from the input condition, and avoids deterioration of the high quality image due to deterioration of the reconstruction image by the projective operation.

Claims

1. An image processing apparatus, comprising:

an information acquisition device for acquiring an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other, and a projective kernel tensor generated from the learning image group and the eigen projective matrix;

a first sub-kernel tensor generation device for generating a first sub-kernel tensor satisfying a condition specified by a first setting from the acquired projective kernel tensor;

a second sub-kernel tensor generation device for generating a second sub-kernel tensor satisfying a condition specified by a second setting from the acquired projective kernel tensor;

a first sub-tensor projection device for projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in an intermediate eigenspace;

a second sub-tensor projection device for generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing device for generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquisition device for acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determination device for determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection device; and

a synthesis device for determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing device according to the weight coefficient, and combining the modified image and the high quality image.

2. The image processing apparatus according to claim 1, further comprising

a learning image coefficient vector storing device for storing the coefficient vector of the learning image in the intermediate eigenspace.

3. The image processing apparatus according to claim 1,

wherein the weight coefficient determination device includes a weight coefficient calculation device for calculating the weight coefficient used for the synthesis device according to the mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace.

4. The image processing apparatus according to claim 1,

wherein the weight coefficient determination device determines the weight coefficient utilizing a reliability coefficient determined from a mutual relationship between a representative coefficient vector in a distribution of coefficient vectors of the input image and the coefficient vector in each area with respect to a distribution of the coefficient vectors of the input image in the intermediate eigenspace where the number of divisions is regarded as a sample when the input image is divided into a plurality of areas.

5. The image processing apparatus according to claim 1,

wherein the weight coefficient determination device determines the weight coefficient in consideration of a reconstruction error represented according to a difference between the predetermined coefficient vector of the learning image in the intermediate eigenspace and a high quality image to be generated from the input image.

6. The image processing apparatus according to claim 1,

wherein the weight coefficient calculation device determines the weight coefficient such that, relatively the longer the shortest distance between a representative coefficient vector representing a learning image coefficient vector group including at least one coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the eigenspace, the greater an adopting ratio of a process by the high image quality processing device is specified.

7. The image processing apparatus according to claim 6,

wherein the weight coefficient determination device determines the weight coefficient with reference to a lookup table representing a relationship of the weight coefficient with the shortest distance between the pre-stored representative coefficient vector and the coefficient vector of the input image in the intermediate eigenspace, on the basis of the shortest distance between the representative coefficient vector and the coefficient vector of the input image in the intermediate eigenspace.

8. The image processing apparatus according to claim 6,

wherein the shortest distance to the coefficient vector of the input image in the eigenspace is represented as any one of the Euclidean distance, Mahalanobis distance and KL distance.

9. The image processing apparatus according to claim 6,

wherein the representative coefficient vector is any one of the average value, median value, maximum value and minimum value of the coefficient vectors of the learning image included in the learning image coefficient vector group.

10. The image processing apparatus according to claim 6,

wherein the representative coefficient vector is the coefficient vector of the leaning image representing the center of gravity or circumferential boundary of the learning image coefficient vector group.

11. The image processing apparatus according to claim 1,

wherein the weight coefficient determination device determines the weight coefficient, according to a degree of similarity in direction between a representative coefficient vector representing a learning image coefficient vector group including at least one coefficient vector of the learning image in the eigenspace and the coefficient vector of the input image in the eigenspace, such that relatively the lower the degree of similarity, the greater an adopting ratio of a process by the high image quality processing device is specified.

12. The image processing apparatus according to claim 11,

wherein the degree of similarity between the direction of the representative coefficient vector and the coefficient vector of the input image in the eigenspace is represented as any one of an angle, an inner product, and an outer product between the representative coefficient vector and the coefficient vector of the input image in the eigenspace.

13. The image processing apparatus according to claim 1,

wherein the weight coefficient determination device determines the weight coefficient, with respect to a distribution of a learning image coefficient vector group including at least one coefficient vector of the learning image in the eigenspace and a distribution of the coefficient vector of the input image in the eigenspace where the number of divisions is regarded as a sample when the input image is divided into a plurality of areas, such that, the wider the distribution spread of the coefficient vector of the input image in comparison with that of the learning image coefficient vector group, the greater an adopting ratio of a process by the high image quality processing device is specified.

14. The image processing apparatus according to claim 13,

wherein the distribution spread of the coefficient vector of the input image in the eigenspace where the number of divisions of the input image is regarded as the sample is represented as the number of samples included in the distribution of the coefficient vector of the input image.

15. The image processing apparatus according to claim 13,

wherein the distribution spread of the coefficient vector of the input image in the eigenspace where the number of divisions of the input image is regarded as a sample and the distribution spread of the learning image coefficient vector group are represented in any one of a variance and a standard deviation.

16. The image processing apparatus according to claim 1,

wherein the weight coefficient is calculated so that the distribution profile of the learning image coefficient vector group including at least one coefficient vector of the learning image in the eigenspace and the distribution profile of the coefficient vector of the input image in the eigenspace where the number of divisions is regarded as a sample when the input image is divided into a plurality of areas, such that the higher a degree of similarity between the distribution profile of the learning image coefficient vector group and the distribution profile of the coefficient vector of the input image in the eigenspace, the greater an adopting ratio of a process including the first and second projective operations.

17. The image processing apparatus according to claim 1,

wherein the synthesis device includes a filtering process device for applying a filtering process to a boundary between the modified image and the high quality image.

18. The image processing apparatus according to claim 1,

wherein the first quality image is a relatively low quality image of the pair of images,

the second quality image is a relatively high quality image of the pair of images, and

the modified quality image is an image with higher quality than that of the input image.

19. The image processing apparatus according to claim 1,

wherein the first setting designates a projective relationship projecting the first quality image to the intermediate eigenspace, and

the second setting designates a projective relationship projecting the second quality image to the intermediate eigenspace.

20. The image processing apparatus according to claim 1,

wherein the projective operation is any one of locality preserving projection (LPP), locally linear embedding (LLE) and linear tangent-space alignment (LTSA).

21. The image processing apparatus according to claim 1,

wherein the learning image group includes the pairs of images regarding a human face as a target, and

the intermediate eigenspace is a personal difference eigenspace.

22. The image processing apparatus according to claim 1, further comprising:

a first characteristic area identification device for identifying a first characteristic area in the inputted image;

a compression process device for compressing an image part of the first characteristic area with respect to the inputted image by a first degree of compression and compressing an image part other than the first characteristic area by a second degree of compression greater than the first degree of compression; and

an image quality modification device for modifying image quality by projecting at least the first characteristic area by the first and the second sub-tensor projection devices.

23. An image processing apparatus, comprising:

an information acquisition device for acquiring an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other, a first sub-kernel tensor satisfying a condition specified by a first setting generated using a projective kernel tensor generated from the learning image group and the eigen projective matrix, and a second sub-kernel tensor satisfying a condition specified by a second setting generated using the projective kernel tensor;

a first sub-tensor projection device for projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in an intermediate eigenspace;

a second sub-tensor projection device for generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing device for generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquisition device for acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determination device for determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection device; and

a synthesis device for determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing device according to the weight coefficient, and combining the modified image and the high quality image.

24. An image processing apparatus, comprising:

an eigen projective matrix generation device for generating an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other;

a projective kernel tensor generation device for generating a projective kernel tensor specifying a corresponding relationship between the first quality images and an intermediate eigenspace and a corresponding relationship between the second quality images and the intermediate eigenspace;

a first sub-kernel tensor acquisition device for generating a first sub-kernel tensor satisfying a condition specified by a first setting from the generated projective kernel tensor;

a second sub-kernel tensor acquisition device for generating a second sub-kernel tensor satisfying a condition specified by a second setting from the generated projective kernel tensor;

a first sub-tensor projection device for projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in the intermediate eigenspace;

a second sub-tensor projection device for generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing device for generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquisition device for acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determination device for determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection device; and

a synthesis device for determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing device according to the weight coefficient, and combining the modified image and the high quality image.

25. The image processing apparatus according to claim 24, further comprising

a learning image coefficient vector calculation device for calculating the coefficient vector of the learning image in the intermediate eigenspace by projecting the learning image using the first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, instead of the learning image coefficient vector acquisition device.

26. An image processing method, including:

an information acquiring step of acquiring an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other, and a projective kernel tensor generated from the learning image group and the eigen projective matrix;

a first sub-kernel tensor generating step of generating a first sub-kernel tensor satisfying a condition specified by a first setting from the acquired projective kernel tensor;

a second sub-kernel tensor generating step of generating a second sub-kernel tensor satisfying a condition specified by a second setting from the acquired projective kernel tensor;

a first sub-tensor projecting step of projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in an intermediate eigenspace;

a second sub-tensor projecting step of generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing step of generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquiring step of acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determining step of determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projecting step; and

a synthesis step of determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing step according to the weight coefficient, and combining the modified image and the high quality image.

27. An image processing method, including:

an information acquiring step of acquiring an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other, a first sub-kernel tensor satisfying a condition specified by a first setting generated using a projective kernel tensor generated from the learning image group and the eigen projective matrix, and a second sub-kernel tensor satisfying a condition specified by a second setting generated using the projective kernel tensor;

a first sub-tensor projecting step of projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in an intermediate eigenspace;

a second sub-tensor projecting step of generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing step of generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquiring step of acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determining step of determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projecting step; and

a synthesis step of determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing step according to the weight coefficient, and combining the modified image and the high quality image.

28. An image processing method, including:

an eigen projective matrix generating step of generating an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other;

a projective kernel tensor generating step of generating a projective kernel tensor specifying a corresponding relationship between the first quality images and an intermediate eigenspace and a corresponding relationship between the second quality images and the intermediate eigenspace;

a first sub-kernel tensor acquiring step of generating a first sub-kernel tensor satisfying a condition specified by a first setting from the generated projective kernel tensor;

a second sub-kernel tensor acquiring step of generating a second sub-kernel tensor satisfying a condition specified by a second setting from the generated projective kernel tensor;

a first sub-tensor projecting step of projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in the intermediate eigenspace;

a second sub-tensor projecting step of generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing step of generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquiring step of acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determining step of determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection step; and

a synthesis step of determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing step according to the weight coefficient, and combining the modified image and the high quality image.

29. A non-transitory computer-readable medium in which computer readable code of an image processing program is stored, wherein the image processing program causes a computer to function as:

an information acquisition device for acquiring an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other, and a projective kernel tensor generated from the learning image group and the eigen projective matrix;

a first sub-kernel tensor generation device for generating a first sub-kernel tensor satisfying a condition specified by a first setting from the acquired projective kernel tensor;

a second sub-kernel tensor generation device for generating a second sub-kernel tensor satisfying a condition specified by a second setting from the acquired projective kernel tensor;

a first sub-tensor projection device for projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in an intermediate eigenspace;

a second sub-tensor projection device for generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing device for generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquisition device for acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determination device for determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection device; and

a synthesis device for determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing device according to the weight coefficient, and combining the modified image and the high quality image.

30. A non-transitory computer-readable medium in which computer readable code of an image processing program is stored, wherein the image processing program causes a computer to function as:

an information acquisition device for acquiring an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other, a first sub-kernel tensor satisfying a condition specified by a first setting generated using a projective kernel tensor generated from the learning image group and the eigen projective matrix, and a second sub-kernel tensor satisfying a condition specified by a second setting generated using the projective kernel tensor;

a first sub-tensor projection device for projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in an intermediate eigenspace;

a second sub-tensor projection device for generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing device for generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquisition device for acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determination device for determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection device; and

a synthesis device for determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing device according to the weight coefficient, and combining the modified image and the high quality image.

31. A non-transitory computer-readable medium in which computer readable code of an image processing program is stored, wherein the image processing program causes a computer to function as:

an eigen projective matrix generation device for generating an eigen projective matrix generated by a projective operation from a learning image group including pairs of first quality images and second quality images different in image quality from each other;

a projective kernel tensor generation device for generating a projective kernel tensor specifying a corresponding relationship between the first quality images and an intermediate eigenspace and a corresponding relationship between the second quality images and the intermediate eigenspace;

a first sub-kernel tensor acquisition device for generating a first sub-kernel tensor satisfying a condition specified by a first setting from the generated projective kernel tensor;

a second sub-kernel tensor acquisition device for generating a second sub-kernel tensor satisfying a condition specified by a second setting from the generated projective kernel tensor;

a first sub-tensor projection device for projecting an input image as a processing target using a first projective operation utilizing the eigen projective matrix and the first sub-kernel tensor, and calculating a coefficient vector in the intermediate eigenspace;

a second sub-tensor projection device for generating a modified image having a different image quality from that of the input image by projecting the calculated coefficient vector using a second projective operation utilizing the second sub-kernel tensor and the eigen projective matrix;

a high image quality processing device for generating a high quality image having the same size as that of the modified image from the input image;

a learning image coefficient vector acquisition device for acquiring a coefficient vector of the learning image in the intermediate eigenspace;

a weight coefficient determination device for determining a weight coefficient according to a mutual relationship between the coefficient vector of the learning image in the intermediate eigenspace and the coefficient vector of the input image in the intermediate eigenspace calculated by the first sub-tensor projection device; and

a synthesis device for determining an adopting ratio between a process including the first and second projective operations and a process by the high image quality processing device according to the weight coefficient, and combining the modified image and the high quality image.