Method for tracking placement of products on shelves in a store

• US 10,467,587 B2
• Filed: 05/19/2017
• Issued: 11/05/2019
• Est. Priority Date: 05/19/2016
• Status: Active Grant

• Current Assignee: Simbe Robotics, Inc.
• Original Assignee: Simbe Robotics, Inc.
• Inventors: Bogolea, Bradley, Shah, Mirza Akbar, Vandegrift, Lorin, Fraser, Luke, Safi, Jariullah, Gee, Jeffrey
• Primary Examiner(s): Rudy, Andrew Joseph
• Application Number:
  US15/600,527
• Publication Number:
  US 20170337508A1
• Time in Patent Office: 900 Days
• Field of Search: 700216, 700217, 705 28
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0282 generated in a local control room
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 1/00 Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]

Abstract

One variation of a method for tracking placement of products in a store includes: accessing an image recorded by a mobile robotic system within a store; detecting a shelf in a region of the image; based on an address of the shelf, retrieving a list of products assigned to the shelf by a planogram of the store; retrieving a set of template images—from a database of template images—defining visual features of products specified in the list of products; extracting a set of features from the region of the image; determining that a unit of the product is mis-stocked on the shelf in response to deviation between the set of features and features in a template image, in the set of template images, representing the product; and in response to determining that the unit of the product is mis-stocked on the shelf, generating a restocking prompt for the product.

Claims

1. A method for tracking placement of products on shelves in a store comprising:

accessing a first image comprising visual data recorded by a mobile robotic system at a first time, the first image associated with a first location and a first orientation of the mobile robotic system within the store at the first time;

detecting a first shelf represented proximal a first region of the first image;

determining an address of the first shelf based on the first location, the first orientation, and a vertical position of the first shelf within the first image;

based on the address of the first shelf, retrieving a first list of products assigned to the first shelf by a planogram of the store;

retrieving a first set of template images from a database of template images, each template image in the first set of template images comprising visual features of a product specified in the first list of products;

extracting a first set of features from the first region of the first image;

confirming presence of a unit of a first product, in the first list of products, on the first shelf in response to correlation between features in the first set of features and features in a first template image, in the first set of template images, representing the first product;

determining that a unit of the first product is improperly stocked on the first shelf in response to deviation between features in the first set of features and features in the first template image; and

in response to determining that the unit of the first product is improperly stocked on the first shelf, generating a first restocking prompt for the first product on the first shelf.

2. The method of claim 1:

wherein extracting the first set of features from the first region of the first image comprises;

from the first region of the first image, detecting a set of discrete objects arranged across the first shelf;

identifying a first subregion in the first region of the first image corresponding to a first discrete object, in the set of discrete objects, located at a first position on the first shelf; and

extracting the first set of features from the first subregion in the first region of the first image; and

wherein confirming presence of the unit of the first product on the first shelf comprises identifying the first discrete object as the unit of the first product in response to correlation between features in the first set of features and features in the first template image representing the first product.

3. The method of claim 2:

further comprising;

identifying a second subregion in the first region of the first image corresponding to a second discrete object, in the set of discrete objects, located at a second position on the first shelf;

extracting a second set of features from the second subregion in the first region of the first image;

identifying the second discrete object as a unit of a second product, in the first list of products, in response to correlation between features in the second set of features and features in a second template image, in the first set of template images, representing the second product; and

in response to identifying the first discrete object as the unit of the first product and identifying the second discrete object as the unit of the second product, detecting improper arrangement of the unit of the first product and the unit of the second product in response to a lateral offset between the first position and the second position deviating from a target lateral offset assigned to the first product and the second product on the first shelf by the planogram; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt specifying correction of relative positions of the unit of the first product and the unit of the second product in response to detecting improper arrangement of the first product and the second product on the first shelf.

4. The method of claim 1:

wherein confirming presence of the unit of a first product on the first shelf comprises confirming presence of a set of discrete units of the first product on the first shelf in response to discrete clusters of features in the first set of features exhibiting strong correlation to features in the first template image representing the first product;

further comprising;

generating a count of units of the first product in the set of discrete units of the first product on the first shelf;

confirming proper stocking of the first product on the first shelf in response to the count equaling a quantity of facings of the first product assigned to the first shelf by the planogram; and

detecting improper stocking of the first product on the first shelf in response to the count deviating from the quantity of facings of the first product assigned to the first shelf by the planogram; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt specifying correction of facings of the first product on the first shelf in response to detecting improper stocking of the first product on the first shelf.

5. The method of claim 1, further comprising:

detecting a second shelf represented in a second region of the first image below the first region;

determining an address of the second shelf based on the first location, the first orientation, and a vertical position of the second shelf within the first image;

based on the address of the second shelf, retrieving a second list of products assigned to the second shelf by the planogram;

retrieving a second set of template images from the database of template images, each template image in the second set of template images comprising visual features of a product specified in the second list of products;

extracting a second set of features from the second region of the first image;

confirming presence of a unit of a second product, in the second list of products, on the second shelf in response to a first degree of correlation between features in the second set of features and features in a second template image, in the second set of template images, representing the second product;

determining that a unit of the second product is improperly stocked on the second shelf in response to a second degree of correlation, less than the first degree, between features in the second set of features and features in the second template image;

in response to determining that the unit of the second product is improperly stocked on the first shelf, generating a second restocking prompt for the second product on the second shelf; and

compiling the first restocking prompt and the second restocking prompt into a global electronic restocking list.

6. The method of claim 1:

wherein confirming presence of the unit of the first product on the first shelf comprises;

matching features, in the first set of features extracted from the first region of the first image, to a first group of features defined in the first template image of the first product;

confirming absence of a second group of features, defined in the first template image, from the first set of features extracted from the first region of the first image; and

detecting orientation of the unit of the first product on the shelf in other than a target orientation in response to the first set of features comprising features matching the first group of features and omitting the second group of features in the first template image representing the first product; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt to correct orientation of the unit of the first product on the first shelf in response to detecting orientation of the unit of the first product on the shelf in other than the target orientation.

7. The method of claim 1:

further comprising;

identifying a first shelf tag proximal a bottom edge of the first region of the first image;

in response to confirming presence of the unit of the first product on the first shelf, extracting a horizontal distance from a left vertical edge of the first shelf tag to a left vertical edge of the unit of the first product; and

in response to the horizontal distance exceeding a threshold distance, determining that the unit of the first product is misoriented on the first shelf; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt to correct an orientation of the unit of the first product on the first shelf in response to determining that the unit of the first product is misoriented on the first shelf.

8. The method of claim 1:

wherein extracting the first set of features from the first region of the first image comprises;

identifying a first subregion of the first region of the first image corresponding to a first slot on the first shelf assigned the first product by the planogram; and

extracting a first subset of features from the first subregion of the first image;

wherein determining that the unit of the first product is improperly stocked on the first shelf comprises;

detecting omission of the unit of the first product from the first slot on the first shelf in response to the first subset of features exhibiting less than a threshold correlation to features in the first template image; and

identifying the first slot on the first shelf as empty in response to correlation between features in the first subset of features and a second set of features representing empty shelves; and

wherein generating the first restocking prompt for the first product on the first shelf comprises generating the first restocking prompt for filling the first slot on the first shelf with units of the first product in response to identifying the first slot on the first shelf as empty.

9. The method of claim 1:

wherein determining the address of the first shelf comprises identifying the first shelf, a first shelving segment containing the first shelf and a second shelf, a first shelving structure containing the first shelving segment and a second shelving segment, and a first aisle facing the first shelving structure and a second shelving structure;

wherein extracting the first set of features from the first region of the first image comprises;

identifying a first subregion in the first region of the first image corresponding to a first slot on the first shelf assigned the first product by the planogram; and

extracting the first set of features from the first subregion of the first image;

wherein determining that the unit of the first product is improperly stocked on the first shelf comprises;

detecting presence of an object in the first slot based on the first set of features; and

identifying the object as other than a product in the first list of products in response to less than a threshold similarity between the first set of features and features in template images in the first set of template images;

further comprising, in response to detecting presence of the object in the first slot and identifying the object as other than a product in the first list of products;

retrieving a second list of products assigned to the first shelving segment by the planogram, the second list of products excluding the first list of products;

retrieving a second set of template images from the database of template images, each template image in the second set of template images comprising visual features of a product specified in the second list of products; and

identifying the object as a unit of a second product, in the second list of products, in response to correlation between features in the first set of features and features in a second template image, in the second set of template images, representing the second product; and

wherein generating the first restocking prompt comprises, in response to identifying the object as the unit of the second product, generating a prompt to replace the unit of the second product currently occupying the first slot with a unit of the first product.

10. The method of claim 9, further comprising, in response to identifying the object as the unit of the second product:

scanning the planogram for a second slot nearest the first slot and assigned the second product; and

generating a second prompt to move the unit of the second product to the second slot.

11. The method of claim 9, further comprising, in response to identifying the object as the unit of the second product:

accessing available inventory of units of the first product at the store;

in response to lack of availability of units of the first product at the store;

withholding the prompt to replace the unit of the second product currently occupying the first slot with a unit of the first product; and

temporarily writing the second product to the first slot, in replacement of the first product, in the planogram.

12. The method of claim 9:

further comprising;

identifying the object as other than a product in the second list of products in response to less than a threshold similarity between the first set of features and features in template images in the second set of template images;

in response to identifying the object as other than a product in the second list of products;

retrieving a third list of products assigned to the first shelving structure by the planogram, the third list of products excluding the first list of products and the second list of products;

retrieving a third set of template images from the database of template images, each template image in the third set of template images comprising visual features of a product specified in the third list of products; and

identifying the object as a unit of a third product, in the third list of products, in response to correlation between features in the first subset of features and features in a third template image, in the third set of template images, representing the third product; and

wherein generating the first restocking prompt comprises, in response to identifying the object as the unit of the third product, generating a prompt to replace the unit of the third product currently occupying the first slot with a unit of the first product.

13. The method of claim 12:

further comprising;

identifying the object as other than a product in the third list of products in response to less than a threshold similarity between the first set of features and features in template images in the third set of template images;

in response to identifying the object as other than a product in the third list of products;

retrieving a fourth list of products assigned to the first aisle by the planogram, the fourth list of products excluding the first list of products, the second list of products, and the third list of products;

retrieving a fourth set of template images from the database of template images, each template image in the fourth set of template images comprising visual features of a product specified in the fourth list of products; and

identifying the object as a unit of a fourth product, in the fourth list of products, in response to correlation between features in the first set of features and features in a fourth template image, in the fourth set of template images, representing the fourth product; and

wherein generating the first restocking prompt comprises, in response to identifying the object as the unit of the fourth product, generating a prompt to replace the unit of the fourth product currently occupying the first slot with a unit of the first product.

14. The method of claim 1:

wherein retrieving the first set of template images from the database of template images comprises retrieving the first template image of the first product in a current packaging and a second template image of the first product in a previous packaging differing from the current packaging;

wherein confirming presence of the unit of the first product on the first shelf comprises;

confirming presence of the unit of the first product, in the current packaging, on the first shelf in response to correlation between features in the first set of features and features in the first template image; and

confirming presence of the unit of the first product, in the previous packaging, on the first shelf in response to correlation between features in the first set of features and features in the second template image;

wherein generating the first restocking prompt for the first product on the first shelf comprises, in response to confirming presence of the unit of the first product in the previous packaging on the first shelf, generating the first restocking prompt to replace the unit of the first product in the previous packaging with a unit of the first product in the current packaging.

15. The method of claim 14, further comprising:

predicting presence of the unit of the first product, in a new packaging, on the first shelf in response to correlation between features in the first set of features and features in the first template image and the second template image and in response to the first set of features comprising features unique to the first template image and the second template image;

storing a subregion of the first region of the first image corresponding to the unit of the first product in the new packaging as a new template image of the first product in the new packaging;

writing the new template image to the database of template images; and

generating a prompt to supply a stock image of the first product in the new packaging.

16. The method of claim 1:

wherein retrieving the first set of template images from the database of template images comprises retrieving a set of multi-dimensional vectors, each multi-dimensional vector in the set of multi-dimensional vectors defining features representative of a product in the first list of products;

wherein extracting the first set of features from the first region of the first image comprises;

identifying a first subregion in the first region of the first image bounding a first discrete object; and

implementing computer vision techniques to detect and extract the first set of features comprising geometry, color, and text features from the first subregion in the first region of the first image; and

wherein confirming presence of the unit of the first product on the first shelf comprises;

calculating a composite score for correlations between features in the first set of features and features defined by a first multi-dimensional vector, in the set of multi-dimensional vectors, representing the first product; and

identifying the first object as the unit of the first product in response to the composite score exceeding a threshold score.

17. The method of claim 1, wherein detecting the first shelf represented proximal the first region of the first image comprises:

detecting a first linear feature extending laterally across the first image;

correlating the first linear feature with the first shelf in a first shelving segment;

detecting a second linear feature extending laterally across the first image above the first linear feature;

correlating the second linear feature with a second shelf above the first shelf in the first shelving segment;

bounding the first region of the first image extending from proximal the first linear feature to proximal the second linear feature in the first image, the first region corresponding to an accessible volume of the first shelf.

18. The method of claim 1:

wherein determining that the unit of the first product is improperly stocked on the first shelf comprises detecting omission of the unit of the first product from a first slot on the first shelf assigned a first number of units of the first product by the planogram;

wherein generating the first restocking prompt for the first product on the first shelf comprises generating a notification specifying an identifier of the first product, the first number of units of the first product, a location of the first slot on the first shelf, and a location of the first shelf within the store defined by the planogram; and

transmitting the notification to a mobile computing device assigned to an associate of the store.

19. The method of claim 1, wherein accessing the first image comprises:

receiving a set of digital photographic images recorded, at approximately the first time, by a set of discrete color cameras integrated into the mobile robotic system, the mobile robotic system occupying the first location and arranged in the first orientation within the store at the first time, the first location adjacent a first shelving segment containing the first shelf;

assembling the set of digital photographic images into the first image based on known positions of the set of discrete color cameras within the mobile robotic system; and

storing the first location, the first orientation, and the first time in metadata of the first image.

20. The method of claim 1, further comprising:

accessing a second image comprising visual data recorded by the mobile robotic system at a second time succeeding the first time, the second image associated with a second location and a second orientation of the mobile robotic system within the store at the second time, the second location offset from the first location;

detecting a second shelf represented proximal a second region of the second image;

determining an address of the second shelf based on the second location, the second orientation, and a vertical position of the second shelf within the second image;

based on the address of the second shelf, retrieving a second list of products assigned to the second shelf by the planogram;

retrieving a second set of template images from the database of template images, each template image in the second set of template images comprising visual features of a product specified in the second list of products;

extracting a second set of features from the second region of the second image;

confirming presence of a unit of a second product, in the second list of products, on the second shelf in response to a first degree of correlation between features in the second set of features and features in a second template image, in the second set of template images, representing the second product;

determining that a unit of the second product is improperly stocked on the second shelf in response to a second degree of correlation, less than the first degree of correlation, between features in the second set of features and features in the second template image;

in response to determining that the unit of the second product is improperly stocked on the second shelf, generating a second restocking prompt for the second product on the second shelf; and

assembling the first restocking prompt and the second restocking prompt into a global restocking list for the store.