Image Depth Inference from Semantic Labels

• US 20170053412A1
• Filed: 08/21/2015
• Published: 02/23/2017
• Est. Priority Date: 08/21/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shen, Xiaohui, Lin, Zhe, Cohen, Scott D., Price, Brian L.
• Granted Patent: US 10,346,996 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2185 the supervisor being an automated module, e.g. intelligent oracle
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 10/7788 the supervisor being a human, e.g. interactive learning with a human teacher

Abstract

Image depth inference techniques and systems from semantic labels are described. In one or more implementations, a digital medium environment includes one or more computing devices to control a determination of depth within an image. Regions of the image are semantically labeled by the one or more computing devices. At least one of the semantically labeled regions is decomposed into a plurality of segments formed as planes generally perpendicular to a ground plane of the image. Depth of one or more of the plurality of segments is then inferred based on relationships of respective segments with respective locations of the ground plane of the image. A depth map is formed that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the one or more of the plurality of segments.

Claims

1. In a digital medium environment including one or more computing devices to control a determination of depth within an image, a method comprising:

decomposing at least one of a plurality of semantically labeled regions of an image by the one or more computing devices into a plurality of segments formed as planes that are generally perpendicular to a ground plane of the image;

inferring depth of one or more of the plurality of segments by the one or more computing devices based on relationship of respective said segments with respective locations of the ground plane of the image; and

forming a depth map by the one or more computing devices that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the one or more of plurality of segments.

2. The method as described in claim 1, wherein semantic labels used in the semantic labeling are selected from a plurality of categories.

3. The method as described in claim 2, wherein the plurality of categories include sky, ground, building, plant, or object.

4. The method as described in claim 1, wherein the depth map is generated without use of training data.

5. The method as described in claim 1, wherein the inferring of the depth includes:

determining a coordinate of a pixel of the respective said segments that connect with the respective location of the ground plane by the one or more computing devices;

using a horizontal direction of the coordinate along with a depth gradient and depth offset computed based on the respective location to the ground plane to infer the depth of the pixel; and

propagating the depth of the pixel for determination of depth of other pixels in the respective said segments by the one or more computing devices.

6. The method as described in claim 5, wherein the propagating is performed using a Gaussian Random Field technique.

7. The method as described in claim 6, wherein the Gaussian Random Field technique is performed as a classification problem in which depths of one or more other segments of the at least one region are classified to a respective one of a plurality of the inferred depths of the respective said segments based on spatial locations and appearance similarities, one to another.

8. The method as described in claim 5, wherein the depth gradient and depth offset are calculated from the ground plane using least square fitting.

9. The method as described in claim 5, wherein the determining, the using, and the propagating are performed for semantically labeled regions that have a size that is determined by the one or more computing devices to be over a defined threshold.

10. The method as described in claim 9, wherein at least one of the semantically labeled regions is determined by the one or more computing devices to have a size below the threshold and the inferring of the depth for the at least one semantically labeled region is performed such that a single depth is inferred for an entirety of the at least one semantically labeled region.

11. The method as described in claim 1, further comprising inferring depth for another one of the semantically labeled regions that is not in contact with the ground plane based at least in part on an assumption that the other semantically labeled region is at least partially occluded by one or more other semantically labeled regions below the other semantically labeled region in the image and applying a random field technique to propagate within the semantically labeled region.

12. In a digital medium environment including one or more computing devices to control a determination of depth within an image, a method comprising:

semantically labeling regions of the image by one or more computing devices;

jointly inferring depth by the one or more computing devices for a plurality of planes within a single one of the semantically labeled regions such that information is shared, one to another, for the plurality of planes as part of the inferring; and

forming a depth map by the one or more computing devices that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the plurality of planes within the single one of the semantically labeled regions.

13. The method as described in claim 12, wherein the jointly inferring depth includes:

decomposing the single one of the semantically labeled regions by the one or more computing devices into a plurality of segments formed as vertical planes;

using a horizontal direction of a coordinate of a pixel of the respective said segments that connect with the ground plane along with a depth gradient and depth offset computed based on the ground plane to infer the depth of the pixel by the one or more computing devices; and

propagating the depth of the pixel for joint determination of depth of other pixels in the respective said segments by the one or more computing devices.

14. The method as described in claim 12, wherein the jointly inferring is performed without using training data.

15. In a digital medium environment having one or more computing devices to control a determination of depth within an image, a depth map system comprising:

a semantic labeling module implemented at least partially in hardware of the one or more computing devices to semantically label regions of the image;

a depth inference module implemented at least partially in hardware of the one or more computing devices to decompose at least one of the semantically labeled regions into a plurality of segments formed as planes that are generally perpendicular to a ground plane and infer depth of each of the plurality of segments based on relationship of respective said segments with respective locations of the ground plane of the image; and

a depth map generation module implemented at least partially in hardware of the one or more computing devices to form a depth map that describes depth for the at least one semantically labeled region based at least in part on the inferred depths for the one or more of the plurality of segments.

16. The system as described in claim 15, wherein the depth inference module is configured to infer the depth by:

determining a coordinate of a pixel of the respective said segments that connect with the respective location of the ground plane by the one or more computing devices;

using a horizontal direction of the coordinate along with a depth gradient and depth offset computed based on the respective location to the ground plane to infer the depth of the pixel; and

propagating the depth of the pixel for determination of depth of other pixels in the respective said segments by the one or more computing devices.

17. The system as described in claim 16, wherein the propagating is performed using a Gaussian Random Field technique.

18. The system as described in claim 17, wherein the Gaussian Random Field technique is performed as a classification problem in which depths of one or more other segments of the at least one region are classified to a respective one of a plurality of the inferred depths of the respective said segments based on spatial locations and appearance similarities, one to another.

19. The system as described in claim 16, wherein the determining, the using, and the propagating are performed for semantically labeled regions that have a size that is determined by the depth inference module to be over a defined threshold.

20. The system as described in claim 19, wherein at least one of the semantically labelled regions is determined by the depth inference module to have a size below the threshold and the inferring of the depth for the at least one semantically labeled region is performed such that a single depth is inferred for an entirety of the at least one semantically labeled region.