Store planogram to image registrations

• US 10,552,787 B2
• Filed: 09/04/2016
• Issued: 02/04/2020
• Est. Priority Date: 09/04/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Marder, Mattias
• Primary Examiner(s): Obaid, Fateh M
• Application Number:
  US15/256,570
• Publication Number:
  US 20180068256A1
• Time in Patent Office: 1,248 Days
• Field of Search: None
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Embodiments of the present invention may provide automated techniques for checking store shelves for compliance with planograms that can handle unknown arrangements in a uniform way, with little user involvement, and with relatively low processing complexity. For example, a computer-implemented method for determining compliance with a planogram may comprise receiving at least one image of a plurality of shelves containing objects, receiving at least one planogram representing desired positions of the objects on the shelves, wherein each row of the planogram corresponds to one shelf, adjusting positions of objects in the planogram to reflect a size of the shelves, determining a tolerance for inaccuracy in object position in the image using object dimension data, detecting object positions in the image, aligning each row of the planogram individually to the detected object positions, and comparing the detected object positions with the adjusted planogram positions to generate compliance information.

Claims

1. A method for determining positions of physical objects implemented in a computer comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising:

capturing, with a photographic device, at least one image of a plurality of physical shelves containing physical objects, and transmitting the at least one image to the computer system;

receiving, at the computer system, the at least one image;

detecting, by computer program instructions executed by the processor, physical objects shown in the at least one image and positions of the physical objects shown in the at least one image;

determining, by computer program instructions executed by the processor, estimates of movement of the detected physical objects from an initial placement of the physical objects;

determining, by computer program instructions executed by the processor, a tolerance for inaccuracy in physical object position in the at least one image using physical object dimension data;

receiving, at the computer system, at least one planogram representing desired positions of the physical objects on the shelves, wherein each row of the planogram corresponds to one shelf;

adjusting, by computer program instructions executed by the processor, margins or dimensions of objects shown in the planogram using physical object dimension data and adjusting positions of physical objects in the planogram to reflect a size of the physical shelves;

detecting, by computer program instructions executed by the processor, matches between physical objects shown in the planogram and physical objects shown in at least one image;

aligning, by computer program instructions executed by the processor, each row of the planogram individually to the determined positions of physical objects shown in the at least one image by, for each row in the planogram;

searching for detected objects, for each detected object, performing consensus voting on the object, determining a detection confidence value for the object based on the tolerance for inaccuracy in object position, and scaling the consensus vote according to the detection confidence value;

outputting, from the computer system, shelf labeling information indicating shelf positions of the physical objects shown in the at least one image;

determining, by computer program instructions executed by the processor, differences between the determined physical object positions and the adjusted planogram positions to generate compliance information representing a level of compliance of the detected object positions with the planogram.

2. The method of claim 1, wherein aligning each row of the planogram individually to the detected object positions further comprises:

for detected objects having low detection confidence values, aligning the detected objects using predetermined adjustment values.

3. The method of claim 2, wherein the determining the differences comprises:

comparing the aligned detected objects having low detection confidence values and the detected objects having high detection confidence values with the adjusted planogram positions; and

generating the compliance information representing a level of compliance of the detected object positions with the planogram.

4. The method of claim 3, wherein the generated compliance information comprises summary information.

5. The method of claim 3, wherein the generated compliance information comprises identifications of objects that are not positioned in accordance with the planogram.

6. A computer program product for determining positions of physical objects, the computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:

capturing, with a photographic device, at least one image of a plurality of physical shelves containing physical objects, and transmitting the at least one image to the computer system;

receiving, at the computer, the at least one image;

detecting, at the computer, physical objects shown in the at least one image and positions of the physical objects shown in the at least one image;

determining, at the computer, estimates of movement of the detected physical objects from an initial placement of the physical objects;

determining, at the computer, a tolerance for inaccuracy in physical object position in the at least one image using physical object dimension data;

receiving, at the computer system, at least one planogram representing desired positions of the physical objects on the shelves, wherein each row of the planogram corresponds to one shelf;

adjusting, at the computer, margins or dimensions of objects shown in the planogram using physical object dimension data and adjusting positions of physical objects in the planogram to reflect a size of the physical shelves;

detecting, at the computer, matches between physical objects shown in the planogram and physical objects shown in at least one image;

aligning, at the computer, each row of the planogram individually to the determined positions of physical objects shown in the at least one image by, for each row in the planogram;

searching for detected objects, for each detected object, performing consensus voting on the object, determining a detection confidence value for the object based on the tolerance for inaccuracy in object position, and scaling the consensus vote according to the detection confidence value;

outputting, from the computer system, shelf labeling information indicating shelf positions of the physical objects shown in the at least one image;

determining, at the computer, differences between the determined physical object positions and the adjusted planogram positions to generate compliance information representing a level of compliance of the detected object positions with the planogram.

7. The computer program product of claim 6, wherein aligning each row of the planogram individually to the detected object positions further comprises:

for detected objects having low detection confidence values, aligning the detected objects using predetermined adjustment values.

8. The computer program product of claim 7, wherein the determining the differences comprises:

comparing the aligned detected objects having low detection confidence values and the detected objects having high detection confidence values with the adjusted planogram positions; and

generating the compliance information representing a level of compliance of the detected object positions with the planogram.

9. The computer program product of claim 8, wherein the generated compliance information comprises summary information.

10. The computer program product of claim 8, wherein the generated compliance information comprises identifications of objects that are not positioned in accordance with the planogram.

11. A system for determining positions of physical objects, the system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:

capturing, with a photographic device, at least one image of a plurality of physical shelves containing physical objects, and transmitting the at least one image to the computer system;

receiving, at the system, the at least one image;

detecting, at the system, physical objects shown in the at least one image and positions of the physical objects shown in the at least one image;

determining, at the system, estimates of movement of the detected physical objects from an initial placement of the physical objects;

determining, at the system, a tolerance for inaccuracy in physical object position in the at least one image using physical object dimension data;

receiving, at the computer system, at least one planogram representing desired positions of the physical objects on the shelves, wherein each row of the planogram corresponds to one shelf;

adjusting, at the system, margins or dimensions of objects shown in the planogram using physical object dimension data and adjusting positions of physical objects in the planogram to reflect a size of the physical shelves;

detecting, at the system, matches between physical objects shown in the planogram and physical objects shown in at least one image;

aligning, at the system, each row of the planogram individually to the determined positions of physical objects shown in the at least one image by, for each row in the planogram;

searching for detected objects, for each detected object, performing consensus voting on the object, determining a detection confidence value for the object based on the tolerance for inaccuracy in object position, and scaling the consensus vote according to the detection confidence value;

outputting, from the computer system, shelf labeling information indicating shelf positions of the physical objects shown in the at least one image;

determining, at the system, differences between the determined physical object positions and the adjusted planogram positions to generate compliance information representing a level of compliance of the detected object positions with the planogram.

12. The system of claim 11, wherein aligning each row of the planogram individually to the detected object positions further comprises:

for detected objects having low detection confidence values, aligning the detected objects using predetermined adjustment values.

13. The system of claim 12, wherein the determining the differences comprises:

comparing the aligned detected objects having low detection confidence values and the detected objects having high detection confidence values with the adjusted planogram positions; and

generating the compliance information representing a level of compliance of the detected object positions with the planogram.

14. The system of claim 13, wherein the generated compliance information comprises summary information.

15. The system of claim 13, wherein the generated compliance information comprises identifications of objects that are not positioned in accordance with the planogram.