Image processing method and apparatus for processing an image pair

• US 9,948,913 B2
• Filed: 11/09/2015
• Issued: 04/17/2018
• Est. Priority Date: 12/24/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Li, Weiming, Liu, Zhihua, Wang, Xiying, Cho, Yang Ho, Xue, Kang, Zhou, Mingcai, Hong, Tao, Ma, Gengyu, Wang, Haitao
• Primary Examiner(s): Hausmann, Michelle M
• Application Number:
  US14/935,757
• Publication Number:
  US 20160189380A1
• Time in Patent Office: 890 Days
• Field of Search: None
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An image processing method and apparatus are provided. The image processing method may include determining whether stereoscopic objects that are included in an image pair and that correspond to each other are aligned on the same horizontal line. The method includes determining whether the image pair includes target objects having different geometric features from those of the stereoscopic objects if the stereoscopic objects are not aligned on the same horizontal line. The method includes performing image processing differently for the stereoscopic objects and for the target objects if the image pair includes the target objects.

Claims

1. An image processing method comprising:

determining whether stereoscopic objects are aligned on a same horizontal line by determining if a difference in viewpoint in a vertical direction exists between a first image and a second image included in an image pair, the stereoscopic objects being included in the image pair and corresponding to each other;

determining whether the image pair includes target objects if the stereoscopic objects are not aligned on the same horizontal line, the target objects having different geometric features from geometric features of the stereoscopic objects; and

performing image processing for the stereoscopic objects and for the target objects if the image pair includes the target objects, the image processing for the stereoscopic objects being different than the image processing for the target objects, the performing image processing including,

separating the target objects from the image pair,

compensating for image holes generated by separating the target objects from the image pair, and

if the difference in viewpoint in the vertical direction exists, reducing the difference in viewpoints in the vertical direction between the stereoscopic objects by performing image processing on the image pair with the compensated image holes.

2. The image processing method of claim 1, wherein the first and second images are acquired by capturing a same scene at different viewpoints.

3. The image processing method of claim 1, wherein the separating separates the target objects from the image pair based on at least one of a morphological feature, a temporal feature and a geometric feature of each of the target objects, the geometric feature meeting a standard.

4. The image processing method of claim 3, wherein the target objects are caption-type target objects, and

wherein the separating comprises,

extracting caption areas from the image pair based on the morphological feature,

identifying pixels corresponding to the caption-type target objects in the caption areas based on pixel values included in the caption areas, and

separating the identified pixels from the image pair.

5. The image processing method of claim 4, wherein the separating further comprises:

matching the caption areas in the image pair and determining a depth of each of the caption areas; and

determining, based on the depth, whether the caption areas are to be removed.

6. The image processing method of claim 3, wherein the target objects are icon-type target objects, and

wherein the separating comprises,

matching icon areas in the image pair based on a template image, the template image being formed and associated with icons, and

separating the matched icon areas from the image pair.

7. The image processing method of claim 6, wherein the template image is determined based on last updated pixel values by updating pixel values of the template image based on a difference in pixel values between the image pair and a neighboring image pair temporally adjacent to the image pair, and

wherein the updating is repeated until a rate of change in the pixel values due to the updating is less than a threshold rate.

8. The image processing method of claim 3, wherein the separating comprises:

dividing the first image included in the image pair into a plurality of patches;

matching the plurality of patches to the second image, the second image being included in the image pair and corresponding to the first image; and

separating the target objects based on a result of the matching from the image pair.

9. The image processing method of claim 8, wherein the matching comprises calculating a matching score indicating a similarity between the plurality of patches and areas of the second image, the areas being on the same horizontal line as the plurality of patches, and

wherein the separating the target objects based on the result of the matching from the image pair comprises determining the target objects based on the matching score and separating the determined target objects from the image pair.

10. The image processing method of claim 1, wherein the compensating compensates for the image holes based on areas other than the image holes in the image pair.

11. The image processing method of claim 10, wherein the compensating compensates for the image holes based on one of:

a scheme of compensating for a pixel included in the image hole based on values of pixels closest to the pixel in the image hole among pixels included in areas other than the image hole;

a scheme of compensating for the image holes based on a texture feature of a pixel adjacent to the image hole;

a scheme of estimating content information of the image pair based on type information of an object adjacent to the image hole and compensating for the image holes based on the estimated content information; and

a scheme of compensating for the image holes based on the first image and the second image, the first image and the second image being included in the image pair.

12. The image processing method of claim 1, wherein the performing image processing comprises:

performing image processing on the separated target objects; and

adding the target objects on which the image processing is performed to the image pair on which the image processing is performed.

13. The image processing method of claim 1, wherein the determining whether the stereoscopic objects are aligned on the same horizontal line comprises:

determining whether the image pair corresponds to a boundary frame of a video segment, the image pair being included in the video segment;

estimating a geometric feature of the image pair and determining whether the stereoscopic objects are aligned on the same horizontal line if the image pair is corresponds to the boundary frame; and

determining, based on estimated information about the video segment, whether the stereoscopic objects are aligned on the same horizontal line if the image pair does not correspond to the boundary frame.

14. The image processing method of claim 13, wherein the determining whether the image pair corresponds to the boundary frame comprises:

comparing an image feature vector of the image pair to an image feature vector of a previous image pair preceding the image pair; and

determining whether the image pair corresponds to the boundary frame based on the comparing.

15. The image processing method of claim 1, wherein each of the target objects is an object added to the image pair during editing of the image pair.

16. A non-transitory computer-readable storage medium storing computer readable instructions, which when executed by a processor, cause the processor to perform the method of claim 1.

17. An image processing apparatus comprising:

a communicator configured to receive an image pair included in a three-dimensional (3D) video; and

at least one processor configured to execute computer-readable instructions to,

determine whether stereoscopic objects are aligned on a same horizontal line by determining if a difference in viewpoint in a vertical direction exists between a first image and a second image included in an image pair, the stereoscopic objects being included in the image pair and corresponding to each other,

determine whether the image pair includes target objects if the stereoscopic objects are not aligned on the same horizontal line, the target objects having different geometric features from those of the stereoscopic objects, and

perform image processing for the stereoscopic objects and for the target objects if the image pair includes the target objects, the image processing for the stereoscopic objects being different from the image processing for the target objects, the at least one processor being further configured to,

separate the target objects from the image pair,

compensate for image holes generated by separating the target objects from the image pair, and

if the difference in viewpoint in the vertical direction exists, reduce the difference in viewpoints in the vertical direction between the stereoscopic objects by performing image processing on the image pair with the compensated image holes.

18. An image processing apparatus comprising:

a communicator configured to receive an image pair included in a three-dimensional (3D) video, the image pair including stereoscopic objects and target objects, the stereoscopic objects corresponding to each other and being part of a captured scene, the target objects being added to the image pair after the captured scene; and

at least one processor configured to execute computer readable instructions to,

determine whether stereoscopic objects are aligned on a same horizontal line by determining if a difference in viewpoint in a vertical direction exists between a first image and a second image included in an image pair, and

perform image processing for the stereoscopic objects and for the target objects if the stereoscopic objects are not aligned on the same horizontal line, the image processing for the stereoscopic objects being different from the image processing for the target objects, the at least one processor being further configured to,

separate the target objects from the image pair,

compensate for image holes generated by separating the target objects from the image pair, and

if the difference in viewpoint in the vertical direction exists reduce the difference in viewpoints in the vertical direction between the stereoscopic objects by performing image processing on the image pair with the compensated image holes.