Planogram Assisted Inventory System and Method

• US 20170286773A1
• Filed: 03/28/2017
• Published: 10/05/2017
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: Shanghai Hanshi Information Technology Company Limited
• Original Assignee: Bossa Nova Robotics IP, Inc.
• Inventors: Skaff, Sarjoun, Taylor, Jonathan Davis, Williams, Stephen Vincent, Dube, Simant
• Granted Patent: US 10,565,548 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A manually assisted robot inventory monitoring method provides for detecting and reading shelf labels using an autonomous robot. Bounding boxes around possible products in a panoramic image can be taken with at least one camera associated with the autonomous robot. Products in the bounding boxes are automatically identified, with those that are not being later manually identified.

Claims

1. A planogram assisted inventory method, comprising the steps of:

providing an initial planogram with identified products;

creating a panoramic image including products on a shelf, with possible products being surrounded by bounding boxes;

detecting and reading shelf labels to localize and associate possible products with both the initial planogram and the possible products in the bounding boxes; and

identifying the possible products with a product classifier.

2. The planogram assisted inventory method of claim 1, further comprising the step of

segmenting images to extract product images and identifiers and associate with the initial planogram.

3. The planogram assisted inventory method of claim 1, further comprising the step of

template matching possible products in the bounding boxes without product segmentation.

4. The planogram assisted inventory method of claim 1, further comprising the step of

using vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.

5. The planogram assisted inventory method of claim 1, further comprising the step of

using deep learning method to build classifiers that identify products or gaps.

6. The planogram assisted inventory method of claim 1, further comprising the step of

allowing an autonomous robot to move beside the shelf, with the autonomous robot acting as a movable base capable of both autonomously detecting and reading shelf labels and creating the panoramic image.

7. The planogram assisted inventory method of claim 1, further comprising the step of

allowing an autonomous robot to move beside a shelf, with the autonomous robot acting as a movable base to capture a depth map of shelves and of products positioned on shelves using 3D cameras, a LIDAR system or structure from motion and multiple cameras.

8. The planogram assisted inventory method of claim 1, further comprising the step of using manual input to identify possible products designated in the bounding boxes.

9. The planogram assisted inventory method of claim 1, wherein the product classifier can include at least one of a machine learning system or a deep learning system.

10. The planogram assisted inventory method of claim 1, further comprising the step of identifying stock depletion level for a product using a product depth detection system.

11. A manually assisted robot inventory monitoring method, comprising the steps of:

detecting and reading shelf labels using an autonomous robot;

defining bounding boxes around possible products in a panoramic image taken with at least one camera associated with the autonomous robot;

automatically identifying products in the bounding boxes; and

manually identifying products in the bounding boxes that were not automatically identified.

12. The manually assisted robot inventory monitoring method of claim 11, further comprising the step of using manual input to identify potential inventory designated in the multiple bounding boxes in the image taken with at least one camera

13. The manually assisted robot inventory monitoring method of claim 11, wherein the bounding boxes can surround gaps between possible products.

14. The planogram assisted inventory method of claim 11, wherein the step of automatically identifying products can further comprise use of at least one of a machine learning or a deep learning system.

15. The planogram assisted inventory method of claim 11, further comprising the step of identifying stock depletion level for an automatically or manually identified product using a product depth detection system.

16. An inventory system comprising:

a planogram with identified products;

an autonomous robot able to capture a panoramic image including products on a shelf, with possible products being identified by bounding boxes;

a shelf label detector and reader attached to the autonomous robot and able to localize and associate possible products with both the planogram and the possible products in the bounding boxes; and

a product classifier to use information from the planogram and the possible products in the bounding boxes to identify products in the panoramic image.

17. The inventory system of claim 16, further comprising

an image segmentation system able to extract product images and identifiers and associate them with the planogram.

18. The inventory system of claim 16, further comprising

a template matching system to match possible products in the bounding boxes without product segmentation.

19. The inventory system of claim 16, further comprising

a system for vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.

20. The inventory system of claim 16, further comprising

a deep learning system to build classifiers that identify products.