Surface segmentation from RGB and depth images
First Claim
1. A computer-implemented method of image segmentation comprising:
- receiving, at a processor, an image of a scene comprising a plurality of image elements, each image element having an associated color value and an associated depth value representing a distance between from an image sensor to a scene element;
applying a weighted line scoring calculation to a plurality of straight lines extracted from the image to determine the principal directions for the image, the weighted line scoring calculation accounting for at least the surface normal of an image element, the direction of a straight line associated with the image element, and a plurality of predetermined weights;
using the depth values to derive a set of three-dimensional planes present within the scene; and
for each image element, determining whether the image element belongs to a plane from the set and labeling the image element accordingly, wherein the determining comprises evaluating a cost function over the set of planes having terms dependent on the depth value of the image element, the cost function having terms at least related to the number of planes, the set of derived planes, a unary term related to the 3D coordinates of the image element, and a unary term related to the normal of the image element, and the color value of the image element and at least one neighboring image element.
2 Assignments
0 Petitions
Accused Products
Abstract
Surface segmentation from RGB and depth images is described. In one example, a computer receives an image of a scene. The image has pixels which each have an associated color value and an associated depth value representing a distance between from an image sensor to a surface in the scene. The computer uses the depth values to derive a set of three-dimensional planes present within the scene. A cost function is used to determine whether each pixel belongs to one of the planes, and the image elements are labeled accordingly. The cost function has terms dependent on the depth value of a pixel, and the color values of the pixels and at least one neighboring pixel. In various examples, the planes can be extended until they intersect to determine the extent of the scene, and pixels not belonging to a plane can be labeled as objects on the surfaces.
190 Citations
20 Claims
-
1. A computer-implemented method of image segmentation comprising:
-
receiving, at a processor, an image of a scene comprising a plurality of image elements, each image element having an associated color value and an associated depth value representing a distance between from an image sensor to a scene element; applying a weighted line scoring calculation to a plurality of straight lines extracted from the image to determine the principal directions for the image, the weighted line scoring calculation accounting for at least the surface normal of an image element, the direction of a straight line associated with the image element, and a plurality of predetermined weights; using the depth values to derive a set of three-dimensional planes present within the scene; and for each image element, determining whether the image element belongs to a plane from the set and labeling the image element accordingly, wherein the determining comprises evaluating a cost function over the set of planes having terms dependent on the depth value of the image element, the cost function having terms at least related to the number of planes, the set of derived planes, a unary term related to the 3D coordinates of the image element, and a unary term related to the normal of the image element, and the color value of the image element and at least one neighboring image element. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. An image segmentation system, comprising:
-
an input interface arranged to receive an image of a scene from a capture device, the image comprising a plurality of image elements, each image element having an associated color value and an associated depth value representing a distance between from the capture device to a surface in the scene; at least one processor arranged to apply a weighted line scoring calculation to a plurality of straight lines extracted from the image to determine the principal directions for the image and to use the depth values to derive a set of three-dimensional planes present within the scene, and, for each image element, determine whether the image element belongs to a plane from the set and label the image element accordingly, the weighted line scoring calculation accounting for at least the surface normal of an image element, the direction of a straight line associated with the image element, and a plurality of predetermined weights, wherein the at least one processor is arranged to determine whether the image element belongs to a plane by evaluating a cost function over the set of planes, the cost function having terms at least related to the number of planes, the set of derived planes, a unary term related to the 3D coordinates of the image element, and a unary term related to the normal of the image element. - View Dependent Claims (18, 19)
-
-
20. One or more tangible device-readable media with device-executable instructions that, when executed by a computing system, direct the computing system to perform steps comprising:
-
receiving an image of a scene comprising a plurality of image elements from a combined RGB and depth camera, each image element having an associated RGB intensity value and an associated depth value representing a distance between from the depth camera to a scene element; applying a weighted line scoring calculation to a plurality of straight lines extracted from the image to determine the principal directions for the image, the weighted line scoring calculation accounting for at least the surface normal of an image element, the direction of a straight line associated with the image element, and a plurality of predetermined weights; using a random sample consensus process on the depth values to derive a set of three-dimensional planes present within the scene; and for each image element, determining whether the image element belongs to a plane from the set, and, if so, labeling the image element with a plane identifier, and, if not, labeling the image element as an object, wherein the determining comprises evaluating a cost function over the set of planes having terms dependent on the depth value of the image element, the cost function having terms at least related to the number of planes, the set of derived planes, a unary term related to the 3D coordinates of the image element, and a unary term related to the normal of the image element.
-
Specification