IMAGE PROCESSING APPARATUS AND METHOD

• US 20190122383A1
• Filed: 03/10/2017
• Published: 04/25/2019
• Est. Priority Date: 03/24/2016
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: TAKAO, Yoshiyoki
• Granted Patent: US 10,872,434 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 9/3129 scanning a light beam on the display screen scanning a light beam on a screen in displays other than projection devices G09G3/02; scanning systems in general G02B26/10; projectors using laser light sources in general H04N9/3161
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

The present disclosure relates to an image processing apparatus and a method that can improve the robustness of posture estimation during viewing of contents with respect to a projected image. Similarity between a captured image obtained by imaging a projected image by an imaging unit, the projected image having been projected by a projection unit, and a simulation image simulating a captured image supposed to be obtained by imaging a projected image by the imaging unit, the projected image having been projected by the projection unit, is evaluated, and a posture of the projection unit or the imaging unit or postures of both of the projection unit and the imaging unit on the basis of an evaluation result of the similarity are set. The present disclosure can be applied to, for example, an image processing apparatus, a projection device, an imaging device, a projection imaging device, a control device, a projection imaging system, or the like.

Claims

1. An image processing apparatus comprising:

a similarity evaluation unit configured to evaluate similarity between a captured image obtained by imaging a projected image by an imaging unit, the projected image having been projected by a projection unit, and a simulation image simulating a captured image supposed to be obtained by imaging a projected image by the imaging unit, the projected image having been projected by the projection unit; and

a posture setting unit configured to set a posture of the projection unit or the imaging unit or postures of both of the projection unit and the imaging unit as an estimation result on a basis of an evaluation result of the similarity by the similarity evaluation unit.

2. The image processing apparatus according to claim 1, wherein

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit as the estimation result, the postures having been evaluated to have an optimum similarity by the similarity evaluation unit.

3. The image processing apparatus according to claim 2, wherein

the similarity evaluation unit obtains a sum of differences in absolute values of luminance values of pixels between the simulation image and the captured image, as an evaluation value indicating the similarity, and

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit, the postures having a minimum evaluation value, as the estimation result.

4. The image processing apparatus according to claim 2, wherein

the similarity evaluation unit obtains a sum of squares of differences in luminance values of pixels between the simulation image and the captured image, as an evaluation value indicating the similarity, and

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit, the postures having a minimum evaluation value, as the estimation result.

5. The image processing apparatus according to claim 2, wherein

the similarity evaluation unit obtains normalized cross-correlation between the simulation image and the captured image, as an evaluation value indicating the similarity, and

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit, the postures having a maximum evaluation value, as the estimation result.

6. The image processing apparatus according to claim 2, wherein

the similarity evaluation unit obtains zero-mean normalized cross-correlation between the simulation image and the captured image, as an evaluation value indicating the similarity, and

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit, the postures having a maximum evaluation value, as the estimation result.

7. The image processing apparatus according to claim 1, wherein

the similarity evaluation unit sets a virtual posture of the projection unit or the imaging unit or virtual postures of both of the projection unit and the imaging unit, generates the simulation image obtained at the virtual posture, and evaluates the similarity between the captured image and the simulation image.

8. The image processing apparatus according to claim 7, wherein

the similarity evaluation unit generates the simulation image with a luminance value or a predetermined color component.

9. The image processing apparatus according to claim 1, wherein

the similarity evaluation unit evaluates the similarity between a reduced image of the captured image and the simulation image.

10. The image processing apparatus according to claim 1, wherein

the similarity evaluation unit evaluates the similarity between a partial image in a predetermined range that is an object to be processed, of the captured image, and the simulation image.

11. The image processing apparatus according to claim 1, wherein

the similarity evaluation unit repeats evaluation of the similarity a plurality of times.

12. The image processing apparatus according to claim 11, wherein

the similarity evaluation unit evaluates the similarity of each time, making a resolution of the captured image, or a range of a partial image that is an object to be processed in the captured image variable.

13. The image processing apparatus according to claim 1, further comprising:

a corresponding point distance evaluation unit configured to obtain corresponding points between the captured image and the simulation image, and evaluate a distance between the corresponding points between the captured image and the simulation image.

14. The image processing apparatus according to claim 13, wherein

both of the evaluation of the distance between the corresponding points by the corresponding point distance evaluation unit and the evaluation of the similarity by the similarity evaluation unit are performed, and

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit as the estimation result on a basis of both of an evaluation result of the distance between the corresponding points by the corresponding point distance evaluation unit and an evaluation result of the similarity by the similarity evaluation unit.

15. The image processing apparatus according to claim 13, wherein

either the evaluation of the distance between the corresponding points by the corresponding point distance evaluation unit or the evaluation of the similarity by the similarity evaluation unit is performed, and

the posture setting unit sets the posture of the projection unit or the imaging unit or the postures of both of the projection unit and the imaging unit as the estimation result on a basis of an evaluation result of the distance between the corresponding points by the corresponding point distance evaluation unit or an evaluation result of the similarity by the similarity evaluation unit.

16. The image processing apparatus according to claim 1, wherein

the similarity evaluation unit evaluates the similarity between the captured image and the simulation image between each of a plurality of the projection units and each of a plurality of the imaging units, and

the posture setting unit sets the postures of the plurality of projection units or the plurality of imaging units, or the postures of both of the plurality of projection units and the plurality of imaging units, as the estimation result, on a basis of evaluation results of the similarity between each of the projection units and each of the imaging units by the similarity evaluation unit.

17. The image processing apparatus according to claim 1, further comprising:

a setting unit configured to perform setting regarding geometric correction of the projected image on a basis of the posture set by the posture setting unit.

18. The image processing apparatus according to claim 1, further comprising:

the projection unit configured to project the projected image.

19. The image processing apparatus according to claim 1, further comprising:

the imaging unit configured to image the projected image to obtain the captured image.

20. An image processing method comprising:

evaluating similarity between a captured image obtained by imaging a projected image by an imaging unit, the projected image having been projected by a projection unit, and a simulation image simulating a captured image supposed to be obtained by imaging a projected image by the imaging unit, the projected image having been projected by the projection unit; and

setting a posture of the projection unit or the imaging unit or postures of both of the projection unit and the imaging unit on a basis of an evaluation result of the similarity.