Image processing method and apparatus

• US 10,063,832 B2
• Filed: 08/29/2016
• Issued: 08/28/2018
• Est. Priority Date: 09/01/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Cho, Yang Ho, Hwang, Hyoseok
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US15/250,083
• Publication Number:
  US 20170064280A1
• Time in Patent Office: 729 Days
• Field of Search: 382154, 382164, 382165, 382274, 382275, 358 326, 358 327
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 2013/0077 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

An image processing method includes generating a target color image of a second viewpoint by warping a color image of a first viewpoint to the second viewpoint using a depth image corresponding to the color image of the first viewpoint; determining a conversion relationship of temporally neighboring color images among a plurality of color images of the second viewpoint, the plurality of color images including the target color image; and restoring a first hole of the target color image based on the conversion relationship.

Claims

1. An image processing method comprising:

generating a target color image of a second viewpoint by warping a color image of a first viewpoint to the second viewpoint using a depth image corresponding to the color image of the first viewpoint;

determining a conversion relationship of temporally neighboring color images among a plurality of color images of the second viewpoint, the plurality of color images including the target color image; and

restoring a first hole of the target color image based on the conversion relationship.

2. The method of claim 1, further comprising:

restoring a remaining hole of the target color image using the target color image.

3. The method of claim 1, wherein the conversion relationship indicates a homography of the temporally neighboring color images.

4. The method of claim 1, wherein the determining comprises:

extracting a background feature of the plurality of color images; and

calculating the conversion relationship based on the extracted background feature.

5. The method of claim 4, wherein the extracting comprises:

extracting a plurality of features of the plurality of color images;

ranking the plurality of features based on a depth using depth values corresponding to the plurality of features; and

determining a first number of features among the ranked features to be the extracted background feature.

6. The method of claim 5, wherein the extracting of the plurality of features includes extracting a structure in the plurality of color images as at least one feature of the plurality of features.

7. The method of claim 4, wherein the calculating of the conversion relationship based on the background feature comprises:

detecting a first background feature of a first color image based on the extracted background feature;

detecting a second background feature of a second color image based on the extracted background feature;

matching the first background feature and the second background feature; and

calculating the conversion relationship based on a result of the matching,

wherein the first color image and the second color image are the temporally neighboring color images.

8. The method of claim 1, wherein the restoring comprises:

converting each of the plurality of color images to a target plane of the target color image based on the conversion relationship; and

restoring the first hole based on a result of the converting.

9. The method of claim 8, wherein the restoring of the first hole based on the result of the converting comprises:

allocating a depth value of a background area located adjacent to a second hole of a target depth image corresponding to the target color image to the second hole, the second hole corresponding to the first hole;

detecting depth reference areas corresponding to the second hole from a plurality of depth images corresponding to the plurality of color images converted to the target plane;

detecting color reference areas of the plurality of color images converted to the target plane, the color reference areas corresponding to the depth reference areas; and

restoring the first hole based on the color reference areas.

10. The method of claim 9, wherein the detecting of the depth reference areas comprises:

detecting areas having the same coordinates as coordinates of the second hole from the plurality of depth images; and

determining, to be the depth reference areas, areas of depth values among the detected areas, each having a difference from the depth value allocated to the second hole that is less than or equal to a threshold.

11. The method of claim 9, wherein the restoring of the first hole based on the color reference areas includes restoring the first hole by setting an average of color values of the color reference areas to be a color value of the first hole.

12. The method of claim 9, wherein the restoring of the first hole based on the color reference areas includes restoring the first hole by setting a color value having a greatest appearance frequency among color values of the color reference areas, to be a color value of the first hole.

13. An image processing apparatus comprising:

a memory storing computer-readable instructions; and

one or more processors configured to execute the computer-readable instructions such that the one or more processors are configured to,

generate a target color image of a second viewpoint by warping a color image of a first viewpoint to the second viewpoint based on a depth image corresponding to the color image of the first viewpoint, and

determine a conversion relationship between temporally neighboring color images among a plurality of color images of the second viewpoint, the plurality of color images of the second viewpoint including the target color image, and

restore a first hole of the target color image based on the conversion relationship.

14. An image processing method comprising:

determining a reference viewpoint based on a plurality of viewpoints of a plurality of input color images;

determining a target reference layer corresponding to the reference viewpoint based on the plurality of input color images;

determining a conversion relationship between temporally neighboring reference layers among a plurality of reference layers of the reference viewpoint, the plurality of reference layers including the target reference layer;

allocating a maximum hole to be generated through a warping of the target reference layer, to the target reference layer;

restoring the maximum hole based on the conversion relationship;

generating an output color image by warping at least one of the plurality of input color images to an output viewpoint; and

restoring a hole of the output color image based on the target reference layer.

15. The method of claim 14, further comprising:

restoring a remaining hole of the target reference layer based on the target reference layer.

16. The method of claim 14, wherein the conversion relationship indicates a homography of the temporally neighboring reference layers.

17. The method of claim 14, wherein the determining of the conversion relationship comprises:

extracting a background feature of the target reference layer; and

calculating the conversion relationship based on the background feature.

18. The method of claim 14, wherein the restoring of the maximum hole comprises:

converting the plurality of reference layers to a target plane of the target reference layer based on the conversion relationship; and

restoring the maximum hole based on a result of the converting.

19. An image processing method comprising:

receiving one or more input color images of one or more viewpoints, respectively;

setting a viewpoint from among the one or more viewpoints as a reference viewpoint;

setting a color image, from among the one or more input color images, as a target reference layer, the target reference layer corresponding to the reference viewpoint;

determining a plurality of conversion relationships by determining, for each temporally adjacent pair of reference layers among a plurality of reference layers of the reference viewpoint, a corresponding conversion relationship, the plurality of reference layers including the target reference layer;

warping the target reference layer to one or more viewpoints such that the target reference layer includes one or more holes;

restoring the target reference layer by restoring the one or more holes of the target reference layer based on the plurality of conversion relationships;

generating an output color image by warping an image from among the input color images to a first viewpoint such that the output color image includes a hole; and

restoring the hole of the output color image based on the restored target reference layer.

20. The method of claim 19, wherein the one or more input color images have a same temporal location, and the plurality of reference layers have different temporal locations.

21. The method of claim 20, wherein each conversion relationship, from among the plurality of conversion relationships, indicates a homography of the temporally adjacent pair of reference layers to which the conversion relationship corresponds.

22. The method of claim 19, wherein the restoring the one or more holes of the target reference layer comprises:

generating a plurality of converted reference layers by converting the plurality of reference layers to a target plane of the target reference layer based on the plurality of conversion relationships; and

restoring the one or more holes of the target reference layer based on the plurality of converted reference layers.

23. The method of claim 19, wherein the restoring the hole of the output color image based on the restored target reference layer comprises:

replacing the hole of the output color image with a portion of the restored target reference layer,

the portion of the restored target reference layer having a location within the restored target reference layer that corresponds to the location of the hole of the output color image within the output color image.