Planogram Matching

• US 20170178060A1
• Filed: 05/25/2016
• Published: 06/22/2017
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Schwartz, Edward L.
• Granted Patent: US 10,592,854 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Methods and systems for planogram matching are described. The methods include capturing an image, processing the image to identify a product based on a symbolic identifier, receiving a planogram, identifying a region of a planogram based on a linear grouping, determining a geometrically consistent match between the product and a sample product included in the region of the planogram, and determining an alignment between the region of the planogram and a corresponding region of the image that includes the product. The methods may further include determining a location in the corresponding region that does not include an expected product based on the planogram.

Claims

1. A computer-implemented method comprising:

capturing, by one or more processors, an image;

processing, by the one or more processors, the image to identify a product based on a symbolic identifier;

receiving, by the one or more processors, a planogram;

identifying, by the one or more processors, a region of the planogram based on a linear grouping;

determining, by the one or more processors, a geometrically consistent match between the product and a sample product included in the region of the planogram; and

determining, by the one or more processors, an alignment between the region of the planogram and a corresponding region of the image that includes the product.

2. The method of claim 1, wherein determining the alignment includes determining a location in the corresponding region that does not included an expected product based on the planogram.

3. The method of claim 1, further comprising:

determining, by the one or more processors, locations of corresponding products in the image based on expected product locations in the planogram;

identifying, by the one or more processors, an unmatched location in the image where an expected product is mapped in the planogram and where a corresponding product is not identified in the image; and

classifying, by the one or more processors, the unmatched location.

4. The method of claim 1, wherein determining the geometrically consistent match between the product and the sample product included in the region of the planogram further comprises:

identifying, by the one or more processors, a neighboring sample product in the region of the planogram;

generating, by the one or more processors, a match by matching the neighboring sample product to a portion of the image;

calculating, by the one or more processors, a matching score for the match of the portion of the image based on the matching; and

determining, by the one or more processors, the geometrically consistent match based on the matching score.

5. The method of claim 4, wherein matching the neighboring sample product to the portions of the image further comprises:

determining, by the one or more processors, whether the match satisfies a condition for discriminating a correct match; and

identifying, by the one or more processors, the match as the geometrically consistent match in response to the condition being satisfied.

6. The method of claim 1, wherein the alignment is a difference and the method further comprises:

identifying, by the one or more processors, a corrective action based on the difference between the region of the planogram and the corresponding region of the image that includes the product; and

providing, by the one or more processors, the corrective action.

7. The method of claim 1, further comprising:

identifying, by the one or more processors, an expected product included in the planogram;

determining, by the one or more processors, a similar product included in a database of products, the similar product sharing a similarity with the expected product;

presenting, by the one or more processors, the similar product;

receiving, by the one or more processors, a selection of the similar products as an identified similar product; and

storing, by the one or more processors, the identified similar product and the expected product for use in real world ground truth generation.

8. A system comprising:

a processor; and

a memory storing instructions that, when executed, cause the system to perform operations including;

capturing an image;

processing the image to identify a product based on a symbolic identifier;

receiving a planogram;

identifying a region of the planogram based on a linear grouping;

determining a geometrically consistent match between the product and a sample product included in the region of the planogram; and

determining an alignment between the region of the planogram and a corresponding region of the image that includes the product.

9. The system of claim 8, wherein determining the alignment includes determining a location in the corresponding region that does not included an expected product based on the planogram.

10. The system of claim 8, wherein the memory stores instructions that, when executed, further cause the system to perform operations including:

determining locations of corresponding products in the image based on expected product locations in the planogram;

identifying an unmatched location in the image where an expected product is mapped in the planogram and where a corresponding product is not identified in the image; and

classifying the unmatched location.

11. The system of claim 8, wherein determining the geometrically consistent match between the product and the sample product included in the region of the planogram further comprises:

identifying a neighboring sample product in the region of the planogram;

generating a match by matching the neighboring sample product to a portion of the image;

calculating a matching score for the match of the portion of the image based on the matching; and

determining the geometrically consistent match based on the matching score.

12. The system of claim 11, wherein matching the neighboring sample product to the portions of the image further comprises:

determining whether the match satisfies a condition for discriminating a correct match; and

identifying the match as the geometrically consistent match in response to the condition being satisfied.

13. The system of claim 8, wherein the alignment is a difference and wherein the memory stores instructions that, when executed, further cause the system to perform operations including:

identifying a corrective action based on the difference between the region of the planogram and the corresponding region of the image that includes the product; and

providing the corrective action.

14. The system of claim 8, wherein the memory stores instructions that, when executed, further cause the system to perform operations including:

identifying an expected product included in the planogram;

determining a similar product included in a database of products, the similar product sharing a similarity with the expected product;

presenting the similar product;

receiving a selection of the similar products as an identified similar product; and

storing the identified similar product and the expected product for use in real world ground truth generation.

20. The system of claim 8, wherein the memory stores instructions that, when executed, further cause the system to perform operations including:

identifying an expected product included in the planogram;

determining a similar product included in a database of products, the similar product sharing a similarity with the expected product;

presenting the similar product;

receiving a selection of the similar products as an identified similar product; and

storing the identified similar product and the expected product for use in real world ground truth generation.

15. A computer program product comprising a computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to perform operations including:

capturing an image;

processing the image to identify a product based on a symbolic identifier;

receiving a planogram;

identifying a region of the planogram based on a linear grouping;

determining a geometrically consistent match between the product and a sample product included in the region of the planogram; and

determining an alignment between the region of the planogram and a corresponding region of the image that includes the product.

16. The computer program product of claim 15, wherein the computer program product further causes the computer to perform operations including:

determining locations of corresponding products in the image based on expected product locations in the planogram;

identifying an unmatched location in the image where an expected product is mapped in the planogram and where a corresponding product is not identified in the image; and

classifying the unmatched location.

17. The computer program product of claim 15, wherein the computer program product further causes the computer to perform operations including:

identifying a neighboring sample product in the region of the planogram;

generating a match by matching the neighboring sample product to a portion of the image;

calculating a matching score for the match of the portion of the image based on the matching; and

determining the geometrically consistent match based on the matching score.

18. The computer program product of claim 15, wherein the computer program product further causes the computer to perform operations including:

determining whether the match satisfies a condition for discriminating a correct match; and

identifying the match as the geometrically consistent match in response to the condition being satisfied.

19. The computer program product of claim 15, wherein the alignment is a difference and wherein the computer program product further causes the computer to perform operations including:

identifying a corrective action based on the difference between the region of the planogram and the corresponding region of the image that includes the product; and

providing the corrective action.