Object segmentation based on detected object-specific visual cues

• US 9,327,406 B1
• Filed: 08/19/2014
• Issued: 05/03/2016
• Est. Priority Date: 08/19/2014
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hinterstoisser, Stefan, Konolige, Kurt
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US14/463,156
• Time in Patent Office: 623 Days
• Field of Search: 700/253, 700/259, 382/154, 382/103, 382/276, 382/285, 345/419, 345/676, 345/619
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/31066 Virtual assembly disassembly planning
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

One or more images of a physical environment may be received, where the one or more images may include one or more objects. A type of surface feature predicted to be contained on a portion of one or more surfaces of a single object may be determined. Surface features of the type within regions of the one or more images may then be identified. The regions may then be associated to corresponding objects in the physical environment based on the identified surface features. Based at least in part on the regions associated to the corresponding objects, a virtual representation of the physical environment may be determined, the representation including at least one distinct object segmented from a remaining portion of the physical environment so as to virtually distinguish a boundary of the at least one distinct object from boundaries of objects present in the remaining portion of the physical environment.

Claims

1. A method performed by a computing device having one or more processors and memory, the method comprising:

receiving one or more images of a physical environment, wherein the one or more images include one or more objects;

based on the one or more images, determining one or more regions of the one or more images, the one or more regions representing estimated boundaries of distinct objects in the physical environment;

determining a type of surface feature predicted to be contained on a portion of one or more surfaces of a single object;

identifying, by the one or more processors, surface features of the type within the one or more regions;

determining, for each region of the determined one or more regions, a respective confidence score based on whether an identified surface feature is contained in the region, wherein the respective confidence score is indicative of a probability that the region corresponds to a distinct object in the physical environment; and

based on the determined confidence scores, determining, by the one or more processors, a virtual representation of the physical environment including at least one distinct object segmented from a remaining portion of the physical environment so as to virtually distinguish a boundary of the at least one distinct object from boundaries of objects present in the remaining portion of the physical environment.

2. The method of claim 1, wherein the type of surface feature includes one or more of:

a piece of tape adhered to an object, a barcode contained on an object, a sticker coupled to an object, text printed onto an object, and an image printed onto an object.

3. The method of claim 1, further comprising:

based on the determined virtual representation, selecting a particular object in the physical environment that corresponds to a particular distinct object segmented from the remaining portion of the physical environment; and

providing an instruction to a robotic manipulator to move the selected object.

4. The method of claim 1, further comprising:

based on the received one or more images, determining a three-dimensional (3D) virtual representation of the physical environment, wherein respective images of the one or more images correspond to respective different views of the one or more objects;

determining an average of intensity values associated with voxels of the 3D virtual representation, wherein the intensity values are representative of light reflected off portions of one or more surfaces of objects that correspond to voxels of the 3D virtual representation;

determining, for respective voxels of the 3D virtual representation, a standard deviation of the intensity value associated with the respective voxel with respect to the average;

making a determination, for the respective voxels of the 3D virtual representation, whether the determined standard deviation of the respective voxel is less than a predetermined threshold; and

decreasing respective confidence scores of the determined one or more regions that comprise a plurality of voxels that (i) have standard deviations that are less than the predetermined threshold and (ii) are located within a threshold distance of at least a portion of a border of the respective region,

wherein determining the virtual representation of the physical environment including at least one distinct object segmented from a remaining portion of the physical environment so as to virtually distinguish a boundary of the at least one distinct object from boundaries of objects present in the remaining portion of the physical environment is further based on the decreased respective confidence scores.

5. The method of claim 1, further comprising:

determining a three-dimensional (3D) virtual representation of the physical environment based on the received one or more images, wherein respective images of the one or more images correspond to respective different views of the one or more objects; and

determining one or more orthographic projections of the 3D virtual representation of the physical environment, and

wherein determining the one or more regions of the one or more images comprises determining the one or more regions to be one or more regions of the one or more orthographic projections.

6. The method of claim 1, wherein identifying surface features of the type within the one or more regions comprises identifying surface features of the type within regions of the one or more regions that correspond to predetermined locations of previously-identified surface features of the type contained on other objects.

7. The method of claim 1, wherein the determined one or more regions include multiple regions, wherein at least one region of the multiple regions overlaps at least one other region of the multiple regions, the method further comprising:

determining respective areas of the determined one or more regions, wherein the respective areas are based on a number of pixels included in respective regions of the determined one or more regions;

making a comparison between areas of at least one of the determined one or more regions which include identified surface features of the type and areas of at least one determined one or more regions that are threshold smaller than the areas of the at least one determined one or more regions which include identified surface features of the type; and

based on an output of the comparison, adjusting the respective confidence scores of the at least one of the determined one or more regions that have the threshold smaller areas.

8. The method of claim 7, wherein making the comparison comprises (i) measuring an area of overlap representative of a location where the areas of the at least one determined one or more regions that have threshold smaller areas overlap the areas of at least one of the determined one or more regions which include identified surface features of the type, and (ii) determining that the area of overlap approximately matches at least one of the threshold smaller areas, and

wherein adjusting the respective confidence scores of the at least one of the determined one or more regions that have the threshold smaller areas based on the output of the comparison comprises decreasing the respective confidence scores of the determined one or more regions that have the threshold smaller areas that approximately match the area of overlap.

9. The method of claim 8, wherein decreasing the respective confidence scores of the determined one or more regions that have the threshold smaller areas that approximately match the area of overlap comprises decreasing the respective confidence scores by a particular value, wherein the particular value is either a predetermined constant or a value based on whether the determined one or more regions that have the threshold smaller areas include identified surface features of the type.

10. The method of claim 1, wherein determining, for each region of the determined one or more regions, the respective confidence score based on whether an identified surface feature is contained in the region comprises determining a higher confidence score for regions of the one or more regions that contain the identified surface feature.

11. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

receiving one or more images of a physical environment, wherein the one or more images include one or more objects;

based on the one or more images, determining one or more regions of the one or more images, the one or more regions representing estimated boundaries of distinct objects in the physical environment;

determining a type of surface feature predicted to be contained on a portion of one or more surfaces of a single object;

identifying surface features of the type within the one or more regions;

determining, for each region of the determined one or more regions, a respective confidence score based on whether an identified surface feature is contained in the region, wherein the respective confidence score is indicative of a probability that the region corresponds to a distinct object in the physical environment; and

based on the determined confidence scores, determining a virtual representation of the physical environment including at least one distinct object segmented from a remaining portion of the physical environment so as to virtually distinguish a boundary of the at least one distinct object from boundaries of objects present in the remaining portion of the physical environment.

12. The non-transitory computer readable medium of claim 11, wherein the portion of the one or more surfaces of the single object includes a portion of at least one edge of the single object beyond a threshold distance from corners of the single object.

13. The non-transitory computer readable medium of claim 11, wherein the portion of the one or more surfaces of the single object includes a portion beyond a threshold distance from edges and corners of the single object.

14. The non-transitory computer readable medium of claim 11, wherein the one or more images include one or more of:

at least one depth map image of the one or more objects, at least one color image of the one or more objects, at least one intensity image of the one or more objects, and a gradient image of the one or more objects.

15. The non-transitory computer readable medium of claim 11, wherein identifying surface features of the type within the one or more regions comprises identifying surface features of the type within the one or more regions based on templates representative of regions corresponding to objects that are known to include surface features of the type.

16. The non-transitory computer readable medium of claim 11, wherein determining the respective confidence scores comprises (i) increasing, for each region of the one or more regions in which an identified surface feature is contained, the respective confidence score of the region and (ii) decreasing, for each region of the one or more regions having an area that is threshold smaller than an area of at least one of the regions in which an identified surface feature is contained, the respective confidence score of the region, and

wherein determining the virtual representation based on the determined confidence scores comprises determining the virtual representation based on the increased and decreased confidence scores.

17. The non-transitory computer readable medium of claim 11, wherein the identifying comprises determining that at least one region of the one or more regions contains, within a respective boundary of the respective region, a sub-region that indicates a threshold high level of light reflected off at least one object of the one or more objects.

18. A system comprising:

at least one processor; and

data storage comprising instructions executable by the at least one processor to cause the system to perform operations comprising;

receiving one or more images of a physical environment, wherein the one or more images include one or more objects,

based on the one or more images, determining one or more regions of the one or more images, the one or more regions representing estimated boundaries of distinct objects in the physical environment;

determining a type of surface feature predicted to be contained on a portion of one or more surfaces of a single object, the portion being different from a full length of a boundary of the one or more surfaces of the single object,

identifying surface features of the type within the one or more regions;

determining, for each region of the determined one or more regions, a respective confidence score based on whether an identified surface feature is contained in the region, wherein the respective confidence score is indicative of a probability that the region corresponds to a distinct object in the physical environment; and

based on the determined confidence scores, determining a virtual representation of the physical environment including at least one distinct object segmented from a remaining portion of the physical environment so as to virtually distinguish a boundary of the at least one distinct object from boundaries of objects present in the remaining portion of the physical environment.

19. The system of claim 18, wherein the portion is different from a full length of a boundary of the one or more surfaces of the single object.

20. The system of claim 18, wherein the type of surface feature includes one or more of:

a piece of tape adhered to an object, a barcode contained on an object, a sticker coupled to an object, text printed onto an object, and an image printed onto an object.