Image-based inventory control system using advanced image recognition

• US 9,147,174 B2
• Filed: 06/12/2009
• Issued: 09/29/2015
• Est. Priority Date: 08/08/2008
• Status: Active Grant

• Current Assignee: Snap-On Incorporated
• Original Assignee: Snap-On Incorporated
• Inventors: Glickman, Stephen L., Jackson, David A., Lipsey, Matthew J., Phillips, Preston, Rogers, Frederick J.
• Primary Examiner(s): Lee, Michael G
• Assistant Examiner(s): CHEDEKEL, TABITHA F
• Application Number:
  US12/484,138
• Publication Number:
  US 20100046791A1
• Time in Patent Office: 2,300 Days
• Field of Search: 235/385, 235/375, 235/380, 235/487, 235/494, 235/491, 382/141, 382/152, 382/153, 705/22, 705/28, 705/29, 348/373, 348/207.99
• US Class Current: 1/1
• CPC Class Codes: B25H 3/00 Storage means or arrangements for workshops facilitating access to, or handling of, work tools or instruments containers or packages with special means for dispensing contents B65D83/00; nail dispensers B25C3/00
• CPC Class Codes: B25H 3/028 by sliding extraction from within a common frame
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Systems for monitoring an inventory condition of objects based on captured images are described. An exemplary system includes at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects; and a data storage device storing reference data for each storage drawer. The reference data includes, for each region of interest corresponding to each respective storage drawer, at least one of a reference image signature representing attributes of the region of interest when a stored object is present, and a reference image signature representing attributes of the region of interest when the stored object is not present. A data processor of the system is configured to derive an image signature representing attributes of a region of interest corresponding to a drawer using a captured image of the drawer; and determine an inventory condition of the drawer according to the stored reference data corresponding to the region of interest and the derived image signature representing attributes of the captured image of the region of interest.

Claims

1. An inventory control system for determining an inventory condition of objects stored in the system, the system comprising:

at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects;

a data storage device storing reference data for each storage drawer, wherein the reference data includes, for each region of interest corresponding to each respective storage drawer, a reference image signature representing a pixel number distribution of the region of interest when a stored object is present, wherein the pixel number distribution includes a plurality of pixel numbers; and

a data processor configured to;

derive an image signature representing a pixel number distribution of a region of interest corresponding to a drawer using a captured image of the drawer; and

determine an inventory condition of the drawer according to the stored reference data corresponding to the region of interest and the derived image signature representing the pixel number distribution of the region of interest in the captured image by matching each of the plurality of pixel numbers of the pixel number distribution in the stored reference image signature with the pixel number distribution of the region of interest in the captured image of the drawer.

2. The system of claim 1, wherein the pixel number distribution of the reference data includes a histogram.

3. The system of claim 2, wherein the data processor generates the image signature of the image by:

accessing data including the pixel number distribution relative to preset image attributes;

identifying a group of pixels sharing a set of common image attributes based on a number of pixels sharing the common image attributes;

identifying one or more groups of pixels sharing a different set of common image attributes based on a number of pixels sharing the different set of common image attributes;

generating the image signature based on the common image attributes of the identified pixel groups.

4. The system of claim 1, wherein:

each storage location is configured to store a pre-designated tool;

each storage location is associated with one or more pre-defined regions of interest; and

the data processor determines an inventory condition of the pre-designated tool of a respective storage location based on a correlation of the image signature of each region of interest associated with the storage location, and the reference image signature corresponding to each region of interest associated with the storage location.

12. The system of claim 1, wherein the reference data further includes a reference image signature representing a pixel number distribution of the region of interest when the stored object is not present, and the data processor determines that the object associated with the region of interest is missing from the drawer upon matching the derived image signature to the stored reference image signature representing the pixel number distribution of the region of interest when the stored object is not present.

15. The system of claim 1, wherein the data processor determines that the object associated with the region of interest is present in the drawer when the pixel number distribution of the derived image signature matches the pixel number distribution of the stored reference image signature within a preset percentage tolerance.

5. A method for using in an inventory control system for determining an inventory condition of objects stored in the system, the system comprising at least one storage drawer for storing objects, a data storage device storing reference data for each storage drawer, wherein the reference data includes, for each region of interest corresponding to each respective storage drawer, a reference image signature representing a pixel number distribution of the region of interest when a stored object is present wherein the pixel number distribution includes a plurality of pixel numbers, the method comprising:

capturing an image of a drawer;

deriving an image signature representing a pixel number distribution of at least one region of interest corresponding to the drawer; and

determining an inventory condition of the drawer according to the stored reference data corresponding to the region of interest corresponding to the drawer and the derived image signature representing the pixel number distribution of the region of interest in the captured image by matching each of the plurality of pixel numbers of the pixel number distribution in the stored reference image signature with the pixel number distribution of the region of interest in the captured image of the storage drawer.

6. The method of claim 5, wherein the pixel number distribution of the reference data includes a histogram.

7. The method of claim 6, wherein the image signature of the image is derived by:

accessing data including the pixel number distribution relative to preset image attributes;

identifying a group of pixels sharing a set of common image attributes based on a number of pixels sharing the common image attributes;

identifying one or more groups of pixels sharing a different set of common image attributes based on a number of pixels sharing the different set of common image attributes;

generating the image signature based on the common image attributes of the identified pixel groups.

8. The method of claim 5, wherein:

each storage location is configured to store a pre-designated tool;

each storage location is associated with one or more pre-defined regions of interest; and

the data processor determines an inventory condition of the pre-designated tool of a respective storage location based on a correlation of the image signature of each region of interest associated with the storage location, and the reference image signature corresponding to each region of interest associated with the storage location.

13. The method of claim 5, wherein the reference data further includes a reference image signature representing a pixel number distribution of the region of interest when the stored object is not present, and the determining the inventory condition of the drawer includes determining that the object associated with the region of interest is missing from the drawer upon matching the derived image signature to the stored reference image signature representing the pixel number distribution of the region of interest when the stored object is not present.

16. The method of claim 5, wherein the object associated with the region of image signature matches the pixel number distribution of the stored reference image signature within a preset percentage tolerance.

9. An inventory control system for determining an inventory condition of objects stored in the system, the system comprising:

at least one storage drawer, each storage drawer including a foam layer forming a plurality of storage locations for storing objects, wherein each storage location is configured to store a pre-designated object and each storage location is formed by a cutout of the foam layer corresponding

an object stored in the storage location;

an image sensing device configured to capture an image of one of the storage drawers;

a data storage system storing reference data corresponding to each storage drawer, wherein the reference data includes information of one or more pre-defined regions of interest corresponding to each storage location in the drawer, and objects stored in each storage location in the drawer; and

a data processor configured to;

access the captured image of the drawer;

access the reference data corresponding to the drawer;

determine an inventory condition of the drawer based on the stored reference data corresponding to the drawer and the captured image of the storage drawer;

wherein the reference data is adapted from a data file based on which cutouts of the foam layer are created;

wherein the reference data includes, for each region of interest corresponding to each respective storage drawer, a reference image signature representing a pixel number distribution of the region of interest when a stored object is present, wherein the pixel number distribution includes a plurality of pixel numbers; and

wherein the data processor determines the inventory condition of the drawer by matching each of the plurality of pixel numbers of the pixel number distribution in the stored reference image signature with a pixel number distribution of a region of interest in the captured image of the drawer.

10. The system of claim 9, wherein the data processor imports the data file based on which cutouts of the foam layer are created and derive the reference data from the data file.

14. The system of claim 9, wherein the reference data further includes a reference image signature representing a pixel number distribution of the region of interest when the stored object is not present, and the data processor determines that the object associated with the region of interest is missing from the drawer upon matching the derived image signature to the stored reference image signature representing the pixel number distribution of the region of interest when the stored object is not present.

17. The system of claim 9, wherein the data processor determines that the object associated with the region of interest is present in the drawer when the pixel number distribution of the derived image signature matches the pixel number distribution of the stored reference image signature within a preset percentage tolerance.

11. A method for preparing reference data for use in an inventory control system for determining an inventory condition of objects stored in the system, the system including at least one storage drawer, each storage drawer including a foam layer forming a plurality of storage locations for storing objects, wherein each storage location is configured to store a pre-designated object and each storage location is formed by a cutout of the foam layer corresponding to an object stored in the storage location, the system determining the inventory condition of a respective drawer based on a captured image of the respective drawer and the reference data, the method comprising:

importing, from a database, a data file based on which cutouts of the foam layer for a respective drawer is created, wherein the data files includes positional information of each cutout in the respective drawer;

importing, from the database, a data file based on, for each cutout corresponding to each respective storage drawer, a reference image signature representing a pixel number distribution of the cutout when a stored object is present, wherein the pixel number distribution includes a plurality of pixel numbers; and

using a data processing device to create a data structure specifying positional information of each cutout in each respective drawer and at least one region of interest corresponding to each cutout, according to the imported data files,

wherein the pixel number distribution stored in the data structure created by the data processing device is sufficient to determine inventory conditions of objects stored in the system from the captured image of the respective drawer.