Image depth inference from semantic labels

• US 10,346,996 B2
• Filed: 08/21/2015
• Issued: 07/09/2019
• Est. Priority Date: 08/21/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Shen, Xiaohui, Lin, Zhe, Cohen, Scott D., Price, Brian L.
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): Rice, Elisa M
• Application Number:
  US14/832,328
• Publication Number:
  US 20170053412A1
• Time in Patent Office: 1,418 Days
• CPC Class Codes: G06F 18/2185 the supervisor being an automated module, e.g. intelligent oracle
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 10/7788 the supervisor being a human, e.g. interactive learning with a human teacher

Abstract

Image depth inference techniques and systems from semantic labels are described. In one or more implementations, a digital medium environment includes one or more computing devices to control a determination of depth within an image. Regions of the image are semantically labeled by the one or more computing devices. At least one of the semantically labeled regions is decomposed into a plurality of segments formed as planes generally perpendicular to a ground plane of the image. Depth of one or more of the plurality of segments is then inferred based on relationships of respective segments with respective locations of the ground plane of the image. A depth map is formed that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the one or more of the plurality of segments.

Claims

1. In a digital medium environment including one or more computing devices to control a determination of depth within an image, a method comprising:

decomposing one of a plurality of semantically labeled regions of an image by the one or more computing devices into a plurality of segments formed as planes of the one semantically labeled region, the plurality of segments of the one semantically labeled region being generally perpendicular to a ground plane of the image;

inferring depth of at least one of the plurality of segments by the one or more computing devices based on a depth of a location of the ground plane that is contacted by the at least one of the plurality of segments; and

forming a depth map by the one or more computing devices that describes depth for the one semantically labeled region based at least in part on the inferred depths for the at least one of the plurality of segments.

2. The method as described in claim 1, wherein semantic labels used in the semantic labeling are selected from a plurality of categories.

3. The method as described in claim 2, wherein the plurality of categories include sky, ground, building, plant, or object.

4. The method as described in claim 1, wherein the depth map is generated without use of training data.

5. The method as described in claim 1, wherein the inferring of the depth includes:

determining a coordinate of a pixel of the at least one of the plurality of segments that connects with a respective location of the ground plane by the one or more computing devices;

using a horizontal direction of the coordinate along with a depth gradient and depth offset computed based on the respective location to the ground plane to infer the depth of the pixel; and

propagating the depth of the pixel for determination of depth of other pixels in the respective said segments by the one or more computing devices.

6. The method as described in claim 5, wherein the propagating is performed using a Gaussian Random Field technique.

7. The method as described in claim 6, wherein the Gaussian Random Field technique is performed as a classification problem in which depths of one or more other segments of the one semantically labeled region are classified to a respective one of a plurality of the inferred depths of the respective said segments based on spatial locations and appearance similarities, one to another.

8. The method as described in claim 5, wherein the depth gradient and depth offset are calculated from the ground plane using least square fitting.

9. The method as described in claim 5, wherein the determining, the using, and the propagating are performed for semantically labeled regions that have a size that is determined by the one or more computing devices to be over a defined threshold.

10. The method as described in claim 9, wherein at least one of the semantically labeled regions is determined by the one or more computing devices to have a size below the threshold and the inferring of the depth for the one semantically labeled region is performed such that a single depth is inferred for an entirety of the one semantically labeled region.

11. The method as described in claim 1, further comprising inferring depth for another one of the semantically labeled regions that is not in contact with the ground plane based at least in part on an assumption that the other semantically labeled region is at least partially occluded by one or more other semantically labeled regions below the other semantically labeled region in the image and applying a random field technique to propagate within the semantically labeled region.

12. The method as described in claim 1, further comprising:

segmenting the image into a plurality of regions; and

forming a plurality of semantically labelled regions by assigning semantic labels to the plurality of regions based on a semantic meaning of respective said regions.

13. In a digital medium environment including one or more computing devices to control a determination of depth within an image, a method comprising:

semantically labeling regions of the image by one or more computing devices;

decomposing a single one of the semantically labeled regions by the one or more computing devices into a plurality of segments formed as vertical planes of the one semantically labeled region;

jointly inferring depth by the one or more computing devices for the plurality of segments within the single one of the semantically labeled regions such that information is shared, one to another, for the plurality of segments as part of the inferring, the inferring based at least in part on a depth of a location of a ground plane that is contacted by a respective segment of the plurality of segments; and

forming a depth map by the one or more computing devices that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the plurality of segments within the single one of the semantically labeled regions.

14. The method as described in claim 13, wherein the jointly inferring depth includes:

using a horizontal direction of a coordinate of a pixel of the respective said segments that connect with the ground plane along with a depth gradient and depth offset computed based on the ground plane to infer the depth of the pixel by the one or more computing devices; and

propagating the depth of the pixel for joint determination of depth of other pixels in the respective said segments by the one or more computing devices.

15. The method as described in claim 13, wherein the jointly inferring is performed without using training data.

16. In a digital medium environment having one or more computing devices to control a determination of depth within an image, a depth map system comprising:

a semantic labeling module implemented at least partially in hardware of the one or more computing devices to semantically label regions of the image;

a depth inference module implemented at least partially in hardware of the one or more computing devices to decompose at least one of the semantically labeled regions into a plurality of segments formed as planes that are generally perpendicular to a ground plane and infer depth of each of the plurality of segments based on relationship of respective said segments with respective locations of the ground plane of the image by;

determining a coordinate of a pixel of the respective said segments that connect with the respective location of the ground plane by the one or more computing devices;

using a horizontal direction of the coordinate along with a depth gradient and depth offset computed based on the respective location to the ground plane to infer the depth of the pixel; and

propagating the depth of the pixel for determination of depth of other pixels in the respective said segments by the one or more computing devices; and

a depth map generation module implemented at least partially in hardware of the one or more computing devices to form a depth map that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the one or more of the plurality of segments.

17. The system as described in claim 16, wherein the propagating is performed using a Gaussian Random Field technique.

18. The system as described in claim 17, wherein the Gaussian Random Field technique is performed as a classification problem in which depths of one or more other segments of the at least one region are classified to a respective one of a plurality of the inferred depths of the respective said segments based on spatial locations and appearance similarities, one to another.

19. The system as described in claim 16, wherein the determining, the using, and the propagating are performed for semantically labeled regions that have a size that is determined by the depth inference module to be over a defined threshold.

20. The system as described in claim 19, wherein at least one of the semantically labelled regions is determined by the depth inference module to have a size below the threshold and the inferring of the depth for the at least one semantically labeled region is performed such that a single depth is inferred for an entirety of the at least one semantically labeled region.