Determining Product Placement Compliance

• US 20180276596A1
• Filed: 03/21/2017
• Published: 09/27/2018
• Est. Priority Date: 03/21/2017
• Status: Active Grant

• Current Assignee: Kellanova
• Original Assignee: Kellogg Company
• Inventors: Murthy, Ganapa Sashidhara, Yousef, Daniel Brian
• Granted Patent: US 10,643,177 B2
• Time in Patent Office: Days
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method for product compliance includes receiving, at data processing hardware, a planogram defining a representative placement of a product on a display shelf and receiving at least one image from an imaging device having a field of view arranged to capture a top surface of the display shelf. The method also includes determining whether the product is disposed on the display shelf based on the at least one image. When the product is disposed on the display shelf, the method includes determining an actual placement of the product on the display shelf and comparing the actual placement of the product to the representative placement of the product defined by the planogram. The method further includes determining a planogram compliance based on the comparison of the actual placement of the product to the representative placement of the product and communicating the planogram compliance to a network.

Claims

1. A method comprising:

receiving, at data processing hardware, a planogram defining a representative placement of a product on a display shelf;

receiving, at the data processing hardware, at least one image from an imaging device having a field of view arranged to capture a top surface of the display shelf;

determining, by the data processing hardware, whether the product is disposed on the display shelf based on the at least one image; and

when the product is disposed on the display shelf;

determining, by the data processing hardware, an actual placement of the product on the display shelf;

comparing, by the data processing hardware, the actual placement of the product to the representative placement of the product defined by the planogram;

determining, by the data processing hardware, a planogram compliance based on the comparison of the actual placement of the product to the representative placement of the product; and

communicating, from the data processing hardware, the planogram compliance to a network.

2. The method of claim 1, further comprising:

when the product is absent from the display shelf, determining, by the data processing hardware, a stocking status of the product as being out-of-stock;

when the product is disposed on the display shelf, determining, by the data processing hardware, the stocking status of the product as being in-stock; and

communicating, from the data processing hardware, the stocking status of the product to the network.

3. The method of claim 1, further comprising, when the product is disposed on the display shelf:

determining, by the data processing hardware, a quantity of the product on the display shelf; and

communicating, from the data processing hardware, the quantity of the product to the network.

4. The method of claim 1, further comprising, communicating, from the data processing hardware, the actual placement of the product to the network.

5. The method of claim 1, wherein the top surface of the display shelf defines a pattern of features, the at least one image depicting at least a portion of the top surface of the display shelf.

6. The method of claim 5, wherein determining whether the product is disposed on the display shelf comprises determining a quantity of exposed features on the top surface of the display shelf captured in the at least one image.

7. The method of claim 6, wherein determining the actual placement of the product on the display shelf comprises identifying corresponding locations of the exposed features on the top surface of the display shelf captured in the at least one image.

8. The method of claim 7, further comprising:

detecting, by the data processing hardware, edges of the exposed features in the at least one image;

approximating, by the data processing hardware, the exposed features based on the detected edges of the exposed features;

comparing, by the data processing hardware, the approximated exposed features to the pattern of features;

determining, by the data processing hardware, whether the approximated exposed features correspond to actual features of the pattern features; and

for each approximated exposed feature, when the approximated exposed feature corresponds to a respective actual feature of the pattern of features;

incrementing, by the data processing hardware, a count of the quantity of exposed features; and

identifying, by the data processing hardware, a corresponding location of the actual feature within the pattern of features.

9. The method of claim 8, wherein at least one feature defines a polygonal shape.

10. The method of claim 8, further comprising executing, by the data processing hardware, an image correction routine on the at least one image, the image correction routine correcting at least barrel distortion of the at least one image.

11. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising;

receiving a planogram defining a representative placement of a product on a display shelf;

receiving at least one image from an imaging device having a field of view arranged to capture a top surface of the display shelf;

determining whether the product is disposed on the display shelf based on the at least one image; and

when the product is disposed on the display shelf;

determining an actual placement of the product on the display shelf;

comparing the actual placement of the product to the representative placement of the product defined by the planogram;

determining a planogram compliance based on the comparison of the actual placement of the product to the representative placement of the product; and

communicating the planogram compliance from the data processing hardware to a network.

12. The system of claim 11, wherein the operations further comprise:

when the product is absent from the display shelf, determining a stocking status of the product as being out-of-stock;

when the product is disposed on the display shelf, determining the stocking status of the product as being in-stock; and

communicating the stocking status of the product from the data processing hardware to the network.

13. The system of claim 11, further comprising, when the product is disposed on the display shelf:

determining a quantity of the product on the display shelf; and

communicating the quantity of the product to the network.

14. The system of claim 11, wherein the operations further comprise, communicating the actual placement of the product from the data processing hardware to the network.

15. The system of claim 11, wherein the top surface of the display shelf defines a pattern of features, the at least one image depicting at least a portion of the top surface of the display shelf.

16. The system of claim 15, wherein determining whether the product is disposed on the display shelf comprises determining a quantity of exposed features on the top surface of the display shelf captured in the at least one image.

17. The system of claim 16, wherein determining the actual placement of the product on the display shelf comprises identifying corresponding locations of the exposed features on the top surface of the display shelf captured in the at least one image.

18. The system of claim 17, wherein the operations further comprise:

detecting edges of the exposed features in the at least one image;

approximating the exposed features based on the detected edges of the exposed features;

comparing the approximated exposed features to the pattern of features;

determining whether the approximated exposed features correspond to actual features of the pattern features; and

for each approximated exposed feature, when the approximated exposed feature corresponds to a respective actual feature of the pattern of features;

incrementing a count of the quantity of exposed features; and

identifying a corresponding location of the actual feature within the pattern of features.

19. The system of claim 18, wherein at least one feature defines a polygonal shape.

20. The system of claim 18, wherein the operations further comprise executing an image correction routine on the at least one image, the image correction routine correcting at least barrel distortion of the at least one image.