Determining product placement compliance

• US 10,643,177 B2
• Filed: 03/21/2017
• Issued: 05/05/2020
• Est. Priority Date: 03/21/2017
• Status: Active Grant

• Current Assignee: Kellanova
• Original Assignee: Kellogg Company
• Inventors: Murthy, Ganapa Sashidhara, Yousef, Daniel Brian
• Primary Examiner(s): Zeender, Florian M
• Assistant Examiner(s): Deligi, Vanessa
• Application Number:
  US15/464,777
• Publication Number:
  US 20180276596A1
• Time in Patent Office: 1,141 Days
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method for product compliance includes receiving, at data processing hardware, a planogram defining a representative placement of a product on a display shelf and receiving at least one image from an imaging device having a field of view arranged to capture a top surface of the display shelf. The method also includes determining whether the product is disposed on the display shelf based on the at least one image. When the product is disposed on the display shelf, the method includes determining an actual placement of the product on the display shelf and comparing the actual placement of the product to the representative placement of the product defined by the planogram. The method further includes determining a planogram compliance based on the comparison of the actual placement of the product to the representative placement of the product and communicating the planogram compliance to a network.

Claims

1. A method comprising:

receiving, at data processing hardware, a planogram defining a representative placement of a product on a display shelf within a display shelf environment, the display shelf environment comprising the display shelf and a display screen disposed above the display shelf and in a user'"'"'s field of view, the display screen configured to depict media content;

receiving, at the data processing hardware, at least one image from an imaging device located above the display shelf and having a field of view arranged to capture a top surface of the display shelf, wherein the top surface of the display shelf defines a pattern of features, including at least one polygon-shaped feature, wherein the at least one image captures at least a portion of the top surface of the display shelf with exposed features;

determining, by the data processing hardware, whether the product is disposed on the display shelf based on the at least one image by;

detecting one or more edges of the exposed features;

converting the one or more detected edges to closed features; and

comparing the closed features to the at least one polygon-shaped feature to determine whether the product is disposed on the display shelf;

when the product is disposed on the display shelf;

determining, by the data processing hardware, an actual placement of the product on the display shelf;

comparing, by the data processing hardware, the actual placement of the product to the representative placement of the product defined by the planogram;

determining, by the data processing hardware, a planogram compliance based on the comparison of the actual placement of the product to the representative placement of the product; and

communicating, from the data processing hardware, the planogram compliance to a network; and

when the product is absent from the display shelf;

determining, by the data processing hardware, a stocking status of the product as being out-of-stock; and

communicating, from the data processing hardware, the stocking status of the product to the network.

2. The method of claim 1, further comprising:

when the product is disposed on the display shelf, determining, by the data processing hardware, the stocking status of the product as being in-stock; and

communicating, from the data processing hardware, the stocking status of the product to the network.

3. The method of claim 2, wherein the media content comprises advertising content related to the stocking status of the product on the display shelf.

4. The method of claim 1, further comprising, when the product is disposed on the display shelf:

determining, by the data processing hardware, a quantity of the product on the display shelf; and

communicating, from the data processing hardware, the quantity of the product to the network.

5. The method of claim 1, further comprising, communicating, from the data processing hardware, the actual placement of the product to the network.

6. The method of claim 1, wherein comparing the closed features to the at least one polygon-shaped feature comprises, for each closed feature, comparing a number of edges of a respective closed feature to a number of edges of the at least one polygon-shaped feature, and

wherein determining whether the product is disposed on the display shelf further comprises determining that the product is disposed on the display shelf when the number of edges of the respective closed feature is different than the number of edges of the at least one polygon-shaped feature.

7. The method of claim 1, wherein determining the actual placement of the product on the display shelf comprises identifying corresponding locations of the exposed features on the top surface of the display shelf captured in the at least one image.

8. The method of claim 1, further comprising executing, by the data processing hardware, an image correction routine on the at least one image, the image correction routine correcting at least barrel distortion of the at least one image.

9. The method of claim 1, wherein the media content comprises advertising content related to the product on the display shelf, a manufacturer of the product on the display shelf, or a retail store selling the product on the display shelf.

10. The method of claim 1, wherein the media content comprises at least one image from the imaging device.

11. The method of claim 1, wherein comparing the closed features to the at least one polygon-shaped feature to determine whether the product is disposed on the display shelf comprises, for each of the closed features, comparing an area of a respective closed feature to the area of the at least one polygon-shaped feature, and wherein the product is determined to be disposed on the display shelf when the area of the respective closed feature is different than the area of the at least one polygon-shaped feature.

12. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising;

receiving a planogram defining a representative placement of a product on a display shelf within a display shelf environment, the display shelf environment comprising the display shelf and a display screen disposed above the display shelf and in a user'"'"'s field of view, the display screen configured to depict media content;

receiving at least one image from an imaging device located above the display shelf and having a field of view arranged to capture a top surface of the display shelf, wherein the top surface of the display shelf defines a pattern of features, including at least one polygon-shaped feature, wherein the at least one image captures at least a portion of the top surface of the display shelf with exposed features;

determining, by the data processing hardware, whether the product is disposed on the display shelf based on the at least one image by;

detecting one or more edges of the exposed features;

converting the one or more detected edges to closed features; and

comparing the closed features to the at least one polygon-shaped feature to determine whether the product is disposed on the display shelf; and

when the product is disposed on the display shelf;

determining an actual placement of the product on the display shelf;

comparing the actual placement of the product to the representative placement of the product defined by the planogram;

determining a planogram compliance based on the comparison of the actual placement of the product to the representative placement of the product; and

communicating the planogram compliance from the data processing hardware to a network.

13. The system of claim 12, wherein the operations further comprise:

when the product is absent from the display shelf, determining a stocking status of the product as being out-of-stock;

when the product is disposed on the display shelf, determining the stocking status of the product as being in-stock; and

communicating the stocking status of the product from the data processing hardware to the network.

14. The system of claim 13, wherein the media content comprises advertising content related to the stocking status of the product on the display shelf.

15. The system of claim 12, further comprising, when the product is disposed on the display shelf:

determining a quantity of the product on the display shelf; and

communicating the quantity of the product to the network.

16. The system of claim 12, wherein the operations further comprise, communicating the actual placement of the product from the data processing hardware to the network.

17. The system of claim 12, wherein comparing the closed features to the at least one polygon-shaped feature comprises, for each closed feature, comparing a number of edges of a respective closed feature to a number of edges of the at least one polygon-shaped feature, and

wherein determining whether the product is disposed on the display shelf comprises determining that the product is disposed on the display shelf when the number of edges of the respective closed feature is different than the number of edges of the at least one polygon-shaped feature.

18. The system of claim 12, wherein determining the actual placement of the product on the display shelf comprises identifying corresponding locations of the exposed features on the top surface of the display shelf captured in the at least one image.

19. The system of claim 12, wherein the operations further comprise executing an image correction routine on the at least one image, the image correction routine correcting at least barrel distortion of the at least one image.

20. The system of claim 12, wherein the media content comprises advertising content related to the product on the display shelf, a manufacturer of the product on the display shelf, or a retail store selling the product on the display shelf.

21. The system of claim 12, wherein the media content comprises at least one image from the imaging device.

22. The system of claim 12, wherein comparing the closed features to the at least one polygon-shaped feature to determine whether the product is disposed on the display shelf comprises, for each of the closed features, comparing an area of a respective closed feature to the area of the at least one polygon-shaped feature, and wherein the product is determined to be disposed on the display shelf when the area of the respective closed feature is different than the area of the at least one polygon-shaped feature.