Multi-sensor object recognition system and method

• US 10,521,914 B2
• Filed: 09/07/2017
• Issued: 12/31/2019
• Est. Priority Date: 09/07/2017
• Status: Active Grant

• Current Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Original Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Inventors: Lam, Joseph, Gong, Xinyi
• Primary Examiner(s): Mariam, Daniel G
• Application Number:
  US15/698,288
• Publication Number:
  US 20190073775A1
• Time in Patent Office: 845 Days
• Field of Search: 382190, 382225
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A method of object detection includes obtaining a set of images depicting overlapping regions of an area containing a plurality of objects. Each image includes input object indicators defined by input bounding boxes, input confidence level values, and object identifiers. The method includes identifying candidate subsets of input object indicators in adjacent images. Each candidate subset has input overlapping bounding boxes in a common frame of reference, and a common object identifier. The method includes adjusting the input confidence levels for each input object indicator in the candidate subsets; selecting clusters of the input object indicators satisfying a minimum input confidence threshold, having a common object identifier, and having a degree of overlap satisfying a predefined threshold; and detecting an object by generating a single output object indicator for each cluster, the output object indicator having an output bounding box, an output confidence level value, and the common object identifier.

Claims

1. A method of object detection in an imaging controller, comprising:

obtaining a set of images depicting overlapping regions of an area containing a plurality of objects;

each of the set of images including a plurality of input object indicators defined by respective (i) input bounding boxes, (ii) input confidence level values, and (iii) object identifiers;

identifying candidate subsets of input object indicators in adjacent ones of the set of images, each candidate subset having input bounding boxes that overlap in a common frame of reference, and a common object identifier;

adjusting the input confidence level values upwards from initial values for each input object indicator in the candidate subsets;

selecting clusters of the input object indicators, the input object indicators of each cluster satisfying a minimum input confidence threshold, having a common object identifier, and having a degree of overlap that satisfies a predefined threshold; and

detecting an object by generating a single output object indicator for each cluster, the output object indicator having (i) an output bounding box, (ii) an output confidence level value, and (iii) the common object identifier.

2. The method of claim 1, wherein identifying the candidate subsets includes determining whether at least two input object indicators have a degree of overlap satisfying an adjacency overlap threshold.

3. The method of claim 1, wherein the area contains a plurality of shelves supporting the objects;

the set of images depicting a vertical segment of the shelves.

4. The method of claim 1, wherein adjusting the input confidence level values for each object indicator in the candidate subsets includes:

determining a degree of occlusion for each input object indicator in a candidate subset based on a comparison between the input bounding box and a reference bounding box corresponding to the object identifier of the input object indicator; and

adjusting the input confidence level based on the degree of occlusion.

5. The method of claim 1, further comprising:

adjusting the input bounding boxes for each input object indicator in the candidate subsets.

6. The method of claim 5, wherein adjusting the input bounding boxes includes replacing the input bounding box for each input object in the subset with a reference bounding box corresponding to the common input object identifier.

7. The method of claim 1, wherein selecting the clusters includes:

generating a ranked list of each of the input object indicators based on the input confidence level values; and

beginning at the highest-ranked input object indicator, determining a degree of overlap for successive pairs of the input bounding boxes.

8. The method of claim 7, wherein determining the degree of overlap includes determining a ratio of the intersection of each successive pair of the input bounding boxes to the union of the pair of the input bounding boxes.

9. The method of claim 7, further comprising:

obtaining a further set of images depicting overlapping regions of an adjacent area, the further set of images registered to the common frame of reference; and

repeating the identifying candidate subsets, the adjusting, the selecting clusters, and the generating output object indicators, wherein generating the ranked list includes inserting the output object indicators into the ranked list.

10. A computing device for detecting objects, the computing device comprising:

a memory;

an imaging controller comprising;

an image preprocessor configured to obtain a set of images from the memory depicting overlapping regions of an area containing a plurality of objects;

each of the set of images including a plurality of input object indicators defined by respective (i) input bounding boxes, (ii) input confidence level values, and (iii) object identifiers;

a subset detector configured to identify candidate subsets of input object indicators in adjacent ones of the set of images, each candidate subset having input bounding boxes that overlap in a common frame of reference, and a common object identifier;

the subset detector further configured to adjust the input confidence level values upwards from initial values for each input object indicator in the candidate subsets;

a cluster detector configured to select clusters of the input object indicators, the input object indicators of each cluster satisfying a minimum input confidence threshold, having a common object identifier, and having a degree of overlap that satisfies a predefined threshold; and

an output generator configured to detect an object by generating a single output object indicator for each cluster, the output object indicator having (i) an output bounding box, (ii) an output confidence level value, and (iii) the common object identifier.

11. The computing device of claim 10, wherein the subset detector is further configured to identify the candidate subsets by determining whether at least two input object indicators have a degree of overlap satisfying an adjacency overlap threshold.

12. The computing device of claim 10, wherein the area contains a plurality of shelves supporting the objects;

the set of images depicting a vertical segment of the shelves.

13. The computing device of claim 10, wherein the subset detector is further configured to adjust the input confidence level values for each object indicator in the candidate subsets by:

determining a degree of occlusion for each input object indicator in a candidate subset based on a comparison between the input bounding box and a reference bounding box corresponding to the object identifier of the input object indicator; and

adjusting the input confidence level based on the degree of occlusion.

14. The computing device of claim 10, wherein the subset detector is further configured to adjust the input bounding boxes for each input object indicator in the candidate subsets.

15. The computing device of claim 14, wherein adjusting the input bounding boxes includes replacing the input bounding box for each input object in the subset with a reference bounding box corresponding to the common input object identifier.

16. The computing device of claim 10, wherein the cluster detector is configured to select the clusters by:

generating a ranked list of each of the input object indicators based on the input confidence level values; and

beginning at the highest-ranked input object indicator, determining a degree of overlap for successive pairs of the input bounding boxes.

17. The computing device of claim 16, wherein the cluster detector is further configured to determine the degree of overlap by determining a ratio of the intersection of each successive pair of the input bounding boxes to the union of the pair of the input bounding boxes.

18. The computing device of claim 16, wherein the imaging controller is further configured to:

obtain a further set of images depicting overlapping regions of an adjacent area, the further set of images registered to the common frame of reference; and

repeat the identifying candidate subsets, the adjusting, the selecting clusters, and the generating output object indicators, wherein generating the ranked list includes inserting the output object indicators into the ranked list.

19. A non-transitory computer readable storage medium containing a plurality of computer readable instructions executable by an imaging controller to configure the imaging controller to perform a method of object detection comprising:

obtaining a set of images depicting overlapping regions of an area containing a plurality of objects;

each of the set of images including a plurality of input object indicators defined by respective (i) input bounding boxes, (ii) input confidence level values, and (iii) object identifiers;

identifying candidate subsets of input object indicators in adjacent ones of the set of images, each candidate subset having input bounding boxes that overlap in a common frame of reference, and a common object identifier;

adjusting the input confidence level values upwards from initial values for each input object indicator in the candidate subsets;

selecting clusters of the input object indicators, the input object indicators of each cluster satisfying a minimum input confidence threshold, having a common object identifier, and having a degree of overlap that satisfies a predefined threshold; and

detecting an object by generating a single output object indicator for each cluster, the output object indicator having (i) an output bounding box, (ii) an output confidence level value, and (iii) the common object identifier.