System and Method for Locating, Identifying and Counting Items

• US 20170286901A1
• Filed: 03/28/2017
• Published: 10/05/2017
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: Shanghai Hanshi Information Technology Company Limited (Hanshow Technology Co., Ltd.)
• Original Assignee: Bossa Nova Robotics IP, Inc.
• Inventors: Skaff, Sarjoun, Taylor, Jonathan Davis, Williams, Stephen Vincent, Dube, Simant
• Granted Patent: US 11,087,272 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A system for building a product library without requiring a planogram includes an image capture unit operated to provide images of items. Taking data from a shelf label detector and depth map creation unit, a processing module can be used to compare detected shelf labels to a depth map, define a product bounding box, and associate the bounding box with an image provided by the image capture unit to build image descriptors. The system can include one or more autonomous robots for supporting and moving the image capture unit and other components.

Claims

1. A inventory monitoring method, comprising the steps of:

detecting and reading shelf labels on at least one shelf;

defining one or more bounding boxes around potential inventory in an image taken with at least one camera; and

associating the bounding boxes in the image to shelf labels to build at least one of a product classifier or a product library.

2. The inventory monitoring method of claim 1, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base capable of autonomously reading shelf labels and defining multiple bounding boxes.

3. The inventory monitoring method of claim 1, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base having multiple cameras and a LIDAR range sensing system.

4. The inventory monitoring method of claim 1, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base to capture a depth map of shelves and products positioned on shelves.

5. The inventory monitoring method of claim 1, further comprising the step of building a product library with a product scope containing at least one or more of the following:

one product identifier;

one or more sets of descriptors;

confidence levels for each set of descriptors;

dates at which each set of descriptors was generated;

one or more shelf position metric estimates;

confidence levels for each shelf position metric estimate;

one or more shelf position topological estimates;

a count number for each shelf position topological estimate;

one or more image templates of the product; and

dimensions of the product.

6. The inventory monitoring method of claim 1, further comprising the step of using manual input to identify potential inventory designated in the multiple bounding boxes in the image taken with at least one camera

7. The inventory monitoring method of claim 1, wherein bounding boxes around both potential inventory and gaps between potential inventory are automatically defined.

8. The inventory monitoring method of claim 1, wherein bounding boxes are manually associated with at least one of a shelf label or an on-product marker.

9. The inventory monitoring method of claim 1, wherein bounding boxes are automatically associated with at least one of a shelf label or an on-product marker.

10. The inventory monitoring method of claim 1, further comprising the step of building the product library without use of an initial planogram.

11. A system for building a product classifier, comprising:

an image capture unit;

a shelf label detector and reader to capture shelf label data;

a processing module to define multiple bounding boxes, and

a trainable image classifier to associate images within the bounding boxes with the shelf label data, the images being provided by the image capture unit.

12. The system of claim 11, wherein the image capture unit is mounted on an autonomous robot and is connected to the shelf label detector and reader.

13. The system of claim 11, wherein the trainable image classifier is trained using machine learning.

14. The system of claim 11, wherein the trainable image classifier is trained using a deep learning system.

15. A inventory monitoring method, comprising the steps of:

detecting and reading shelf labels;

detecting distance from at least one camera to at least one shelf;

defining bounding boxes around potential inventory in an image taken with at least one camera; and

associating selected bounding boxes in the image to selected shelf labels to build at least one of a product classifier or a product library.

16. The inventory monitoring method of claim 15, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base capable of detecting distance, reading shelf labels, and defining bounding boxes.

17. The inventory monitoring method of claim 15, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base having multiple cameras and at least one of a LIDAR range sensing system or a depth camera.

18. The inventory monitoring method of claim 15, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base to capture a depth map of shelves and of products positioned on shelves.

19. The inventory monitoring method of claim 15, further comprising the step of building a product library with a product scope containing at least one or more of the following:

one product identifier;

one or more sets of descriptors;

confidence levels for each set of descriptors;

dates at which each set of descriptors was generated;

one or more shelf position metric estimates;

confidence levels for each shelf position metric estimate;

one or more shelf position topological estimates;

a count number for each shelf position topological estimate;

one or more image templates of the product; and

dimensions of the product.

20. The inventory monitoring method of claim 15, further comprising the step of building a product library using manual input to identify potential inventory designated in the multiple bounding boxes in the image taken with at least one camera

21. The inventory monitoring method of claim 15, wherein the bounding boxes can surround gaps between potential inventory.

22. The inventory monitoring method of claim 15, further comprising the step of building the product library without use of an initial planogram.