Realogram scene analysis of images: shelf and label finding

• US 9,811,754 B2
• Filed: 03/06/2015
• Issued: 11/07/2017
• Est. Priority Date: 12/10/2014
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Schwartz, Edward
• Primary Examiner(s): CESE, KENNY A
• Application Number:
  US14/641,292
• Publication Number:
  US 20160171429A1
• Time in Patent Office: 977 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/15 Cutting or merging image elements, e.g. region growing, watershed or clustering-based techniques
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

The techniques include an image recognition system to receive a realogram image including a plurality of organized objects and to detect and identify objects in the realogram image of one or more items on a retail shelf, identify shelf fronts and labels on the shelf fronts, identify empty space under shelves, identify areas where unidentified products may be, and identify areas where products are “out of stock”.

Claims

1. A computer-implemented method for identifying shelves and labels from an image of shelves, the method comprising:

receiving the image of the shelves;

identifying features based on groups of pixels in the image of the shelves;

identifying possible shelf and label locations in the image, based on appearance using the features;

choosing the best shelf and label locations from the possible shelf and label locations in the image using context of the possible shelf and label locations;

generating a shelf model having a first geometric parameter and a first color parameter, the first geometric parameter comprising an estimate of an average thickness of previously detected shelves;

generating a label model having a second geometric parameter and a second color parameter, the second geometric parameter comprising an estimate of a width of a label in the image;

processing the image using the shelf model and the label model to update the possible shelf and label locations in the image based on features and context of the possible shelf and label locations in the image; and

choosing final shelf and label locations in the image using the updated possible shelf and label locations.

2. The computer-implemented method of claim 1, wherein identifying possible shelf and label locations further comprises determining shelf boundaries using features that correspond to a shelf edge based on gradients between neighboring groups of pixels.

3. The computer-implemented method of claim 1, wherein identifying possible shelf and label locations further comprises generating one or more label location hypotheses from the image of the shelves.

4. The computer-implemented method of claim 3, wherein generating the one or more label location hypotheses comprises:

quantizing features in the possible shelf and label locations into color clusters and non-horizontal line clusters;

generating the one or more label location hypotheses based on the color clusters and non-horizontal line clusters;

finding components within the one or more label location hypotheses that match the label model to generate label candidates; and

choosing the best label location hypothesis from the label candidates.

5. The computer-implemented method of claim 4, further comprising validating the one or more label location hypotheses based on an intersection between the color clusters and the non-horizontal line clusters.

6. The computer-implemented method of claim 3, wherein generating the one or more label location hypotheses comprises:

detecting text within the best shelf location;

generating a text bounding box for each block of text detected within the best shelf location; and

performing optical character recognition within each text bounding box.

7. The computer-implemented method of claim 3, wherein the context of the possible shelf and label locations includes one or more from the group of:

labels found within the possible shelf and label locations, prices found within the possible shelf and label locations, long horizontal line segments within the possible shelf and label locations, shelf hypotheses based on assuming regular spacing between shelves, and relationship of possible shelf and label locations to other identified objects in the image of the shelves.

8. A system for identifying shelves and labels from an image of shelves, the system comprising:

one or more processors; and

a memory, the memory storing instructions, which when executed cause the one or more processors to;

receive an image of shelves;

identify features based on groups of pixels in the image of the shelves;

identify possible shelf and label locations in the image based on appearance using the features;

choose the best shelf and label locations from the possible shelf and label locations in the image using context of the possible shelf and label locations;

generate a shelf model having a first geometric parameter and a first color parameter, the first geometric parameter comprising an estimate of an average thickness of previously detected shelves;

generate a label model having a second geometric parameter and a second color parameter, the second geometric parameter comprising an estimate of a width of a label in the image;

process the image using the shelf model and the label model to update the possible shelf and label locations in the image based on features and context of the possible shelf and label locations in the image; and

choose final shelf and label locations in the image using the updated possible shelf and label locations.

9. The system of claim 8, wherein to identify possible shelf and label locations, the instructions cause the one or more processors to determine shelf boundaries using features that correspond to a shelf edge based on gradients between neighboring groups of pixels.

10. The system of claim 8, wherein to identify possible shelf and label locations, the instructions cause the one or more processors to generate one or more label location hypotheses from the image of the shelves.

11. The system of claim 10, wherein to generate the one or more label location hypotheses, the instructions cause the one or more processors to:

quantize features in the possible shelf and label locations into color clusters and non-horizontal line clusters;

generate the one or more label location hypotheses based on the color clusters and non-horizontal line clusters;

find components within the one or more label location hypotheses that match the label model to generate label candidates; and

choose the best label location hypothesis from the label candidates.

12. The system of claim 10, wherein to generate the one or more label location hypotheses, the instructions cause the one or more processors to:

detect text within the best shelf location;

generate a text bounding box for each block of text detected within the best shelf location; and

perform optical character recognition within each text bounding box.

13. The system of claim 10, wherein the context of the possible shelf and label locations includes one or more from the group of:

labels found within the possible shelf and label locations, prices found within the possible shelf and label locations, long horizontal line segments within the possible shelf and label locations, shelf hypotheses based on assuming regular spacing between shelves, and relationship of possible shelf and label locations to other identified objects in the image of the shelves.

20. The system of claim 11, wherein the instructions further cause the one or more processors to validate the one or more label location hypotheses based on an intersection between the color clusters and the non-horizontal line clusters.

14. A computer program product comprising a non-transitory computer usable medium including a computer readable program, wherein the computer readable program, when executed on a computer causes the computer to:

receive an image of shelves;

identify features based on groups of pixels in the image of the shelves;

identify possible shelf and label locations in the image, based on appearance using the features;

choose the best shelf and label locations from the possible shelf and label locations in the image using context of the possible shelf and label locations;

generate a shelf model having a first geometric parameter and a first color parameter, the first geometric parameter comprising an estimate of an average thickness of previously detected shelves;

generate a label model having a second geometric parameter and a second color parameter, the second geometric parameter comprising an estimate of a width of a label in the image;

process the image using the shelf model and the label model to update the possible shelf and label locations in the image based on features and context of the possible shelf and label locations in the image; and

choose final shelf and label locations in the image using the updated possible shelf and label locations.

15. The computer program product of claim 14, wherein to identify possible shelf and label locations, the computer readable program causes the computer to determine shelf boundaries using features that correspond to a shelf edge based on gradients between neighboring groups of pixels.

16. The computer program product of claim 14, wherein to identify possible shelf and label locations, the computer readable program causes the computer to generate one or more label location hypotheses from the image of the shelves.

17. The computer program product of claim 16, wherein to generate the one or more label location hypotheses, the computer readable program causes the computer to:

quantize features in the possible shelf and label locations into color clusters and non-horizontal line clusters;

generate the one or more label location hypotheses based on the color clusters and non-horizontal line clusters;

find components within the one or more label location hypotheses that match the label model to generate label candidates; and

choose the best label location hypothesis from the label candidates.

18. The computer program product of claim 16, wherein to generate the one or more label location hypotheses, the computer readable program causes the computer to:

detect text within the best shelf location;

generate a text bounding box for each block of text detected within the best shelf location; and

perform optical character recognition within each text bounding box.

19. The computer program product of claim 17, wherein the computer readable program further causes the computer to validate the one or more label location hypotheses based on an intersection between the color clusters and the non-horizontal line clusters.