Method and apparatus for shelf feature and object placement detection from shelf images

• US 10,726,273 B2
• Filed: 05/01/2017
• Issued: 07/28/2020
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Original Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Inventors: Phan, Raymond, Zhang, Yan, Rzeszutek, Richard Jeffrey, Fu, Bo
• Primary Examiner(s): Rivera-Martinez, Guillermo M
• Application Number:
  US15/583,740
• Publication Number:
  US 20180315173A1
• Time in Patent Office: 1,184 Days
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method of detecting a back of a shelf for supporting objects includes: obtaining an image depicting a shelf having a shelf edge and a support surface extending from the shelf edge to a shelf back; decomposing the image into a plurality of patches; for each patch: generating a feature descriptor; based on the feature descriptor, assigning one of a shelf back classification and a non-shelf back classification to the patch; generating a mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches; and presenting the mask.

Claims

1. A method of detecting a back of a shelf for supporting objects, comprising:

obtaining, by an imaging controller, an image depicting a shelf having a shelf edge and a support surface extending from the shelf edge to a shelf back;

decomposing, by the imaging controller, the image into a plurality of patches;

for each patch;

generating a feature descriptor;

based on the feature descriptor, assigning one of a shelf back classification and a non-shelf back classification to the patch;

generating a mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches, wherein the indication of the classification assigned to each of the patches comprises one of a plurality of intensity values; and

presenting the mask to a gap detector for detection of gaps from the mask.

2. The method of claim 1, wherein assigning the classification to each patch includes determining a confidence value corresponding to the assigned classification.

3. The method of claim 2, wherein generating the mask comprises, for each patch:

assigning one of two intensity values to each patch based on the confidence value and a confidence threshold.

4. The method of claim 1, further comprising:

prior to decomposing the image, converting the image to greyscale.

5. The method of claim 1, further comprising:

prior to decomposing the image, performing a contrast normalization operation on the image.

6. The method of claim 1, further comprising:

generating a plurality of scaled versions of the image;

repeating the decomposing, generating a feature descriptor, and assigning a classification for each scaled version; and

wherein generating the mask includes generating a sub-mask corresponding to each scaled version, and combining the sub-masks.

7. The method of claim 1, wherein the feature descriptor is a histogram of oriented gradients (HOG) descriptor.

8. The method of claim 1, further comprising:

obtaining, by the imaging controller, a location of the shelf edge within the mask;

generating, by the imaging controller, a gap mask identifying a subset of the areas having shelf back classifications as gaps on the shelf, by;

selecting an area of the mask within a predetermined distance of the shelf edge location and having the shelf back classification; and

generating a joining area between the selected area and the shelf edge location; and

assigning a gap classification to the selected area and the joining area; and

presenting the gap mask.

19. The method of claim 1 wherein the mask is a binary intensity mask.

20. The method of claim 1, wherein patches that do not meet a back of shelf confidence threshold are assigned a first intensity value and patches that meet the back of shelf confidence threshold are assigned a second intensity value.

9. A method of detecting, by an imaging controller, gaps in an image of a shelf having a shelf edge and a support surface for supporting objects extending from the shelf edge to a shelf back, the method comprising:

obtaining, by the imaging controller, a mask generated from the image and containing indications, for each of a plurality of areas of the mask, of a classification assigned to the area and selected from a shelf back classification and a non-shelf back classification;

obtaining, by the imaging controller, a location of the shelf edge within the mask;

generating, by the imaging controller, a gap mask identifying a subset of the areas having shelf back classifications as gaps on the shelf, by;

selecting an area of the mask within a predetermined distance of the shelf edge location and having the shelf back classification; and

generating a joining area between the selected area and the shelf edge location; and

assigning a gap classification to the selected area and the joining area; and

presenting the gap mask.

10. The method of claim 9, wherein generating the gap mask further comprises:

discarding a portion of the mask having a preconfigured position relative to the shelf edge location.

11. The method of claim 10, wherein the preconfigured position includes below the shelf edge location.

12. The method of claim 10, wherein the preconfigured position includes horizontally spaced beyond an end of the shelf edge location.

13. The method of claim 9, wherein generating the gap mask further comprises:

comparing each area having a shelf back classification to a preconfigured area threshold, and discarding any areas that do not meet the threshold.

14. The method of claim 9, wherein generating the gap mask further comprises:

performing a region growth operation on the selected area with the joining area; and

assigning the gap classification to a grown region resulting from the region growth operation.

15. The method of claim 9, further comprising:

obtaining depth data captured by a depth sensor and indicating a point depth for each of a plurality of points in the gap mask;

dividing the gap mask into a plurality of vertical slices;

for each vertical slice of the gap mask;

determining a proportion of the point depths that exceed a median point depth for the vertical slice; and

assigning a non-gap classification to the vertical slice when the proportion does not meet a preconfigured depth distribution threshold.

16. The method of claim 9, wherein obtaining the mask comprises:

obtaining, by the imaging controller, the image;

decomposing, by the imaging controller, the image into a plurality of patches;

for each patch;

generating a feature descriptor;

based on the feature descriptor, assigning one of the shelf back classification and the non-shelf back classification to the patch; and

generating the mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches.

17. A server for detecting a back of a shelf for supporting objects, comprising:

a memory storing an image depicting a shelf having a shelf edge and a support surface extending from the shelf edge to a shelf back; and

an imaging controller comprising;

a pre-processor configured to decompose the image into a plurality of patches;

a back of shelf detector configured to;

(a) for each patch;

generate a feature descriptor;

based on the feature descriptor, assign one of a shelf back classification and a non-shelf back classification to the patch;

(b) generate a mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches, wherein the indication of the classification assigned to each of the patches comprises one of a plurality of intensity values;

the back of shelf detector further configured to present the mask.

18. A server for detecting gaps in an image of a shelf having a shelf edge and a support surface for supporting objects extending from the shelf edge to a shelf back, the system comprising:

a memory storing a mask generated from the image, the mask containing indications, for each of a plurality of areas of the mask, of a classification assigned to the area and selected from a shelf back classification and a non-shelf back classification; and

an imaging controller comprising;

a shelf edge detector configured to obtain a location of the shelf edge within the mask;

a gap generator configured to generate a gap mask identifying a subset of the areas having shelf back classifications as gaps on the shelf, by;

selecting an area of the mask within a predetermined distance of the shelf edge location and having the shelf back classification; and

generating a joining area between the selected area and the shelf edge location; and

assigning a gap classification to the selected area and the joining area; and

the gap generator further configured to present the gap mask.