Image-based inventory control system with automatic calibration and image correction

• US 8,842,183 B2
• Filed: 06/12/2009
• Issued: 09/23/2014
• Est. Priority Date: 08/08/2008
• Status: Active Grant

• Current Assignee: Snap-On Incorporated
• Original Assignee: Snap-On Incorporated
• Inventors: Glickman, Stephen L., Jackson, David A., Lipsey, Matthew J., Phillips, Preston, Fly, David C., Weeks, Jason L.
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): WILLIAMS, JEFFERY A
• Application Number:
  US12/484,139
• Publication Number:
  US 20100039513A1
• Time in Patent Office: 1,929 Days
• Field of Search: 382/165, 348/143
• US Class Current: 348/165
• CPC Class Codes: B25H 3/00 Storage means or arrangements for workshops facilitating access to, or handling of, work tools or instruments containers or packages with special means for dispensing contents B65D83/00; nail dispensers B25C3/00
• CPC Class Codes: B25H 3/028 by sliding extraction from within a common frame
• CPC Class Codes: G06K 7/10009 sensing by radiation using wavelengths larger than 0.1 mm, e.g. radio-waves or microwaves
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/60 Type of objects
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Systems for monitoring an inventory condition of objects based on captured images are described. An exemplary system includes at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects, wherein each drawer is associated with an identifier with known color attributes; and an image sensing device configured to capture an image of one of the storage drawers along with the associated identifier. A data storage device of the system stores, for each storage drawer, information of the known color attributes of the associated identifier. A data processor of the system is configured to access information of the known color attributes of the identifier associated with the drawer corresponding to the captured image; determine color attributes of the identifier in the captured image; determine a correction factor based on the color attributes of the identifier in the captured image and the known color attributes of the identifier; and apply the correction factor to subsequent images captured by the image sensing device.

Claims

1. An inventory control system for determining an inventory condition of objects stored in the system, the system comprising:

at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects, wherein each drawer is associated with an identifier that is separate from the stored objects and has known color attributes, wherein the identifier is associated with one of the storage locations and has different known color attributes depending on whether one of the objects is in the storage location;

a data storage system storing, for each storage drawer, information of the known color attributes of the associated identifier; and

a data processor configured to;

access data of a captured image of one of the storage drawers along with the associated identifier;

access information of the known color attributes of the identifier associated with the drawer corresponding to the captured image;

determine color attributes of the identifier in the captured image;

determine a correction factor based on the color attributes of the identifier in the captured image and the known color attributes of the identifier;

apply the correction factor to subsequent images captured by an image sensing device; and

determine an inventory condition of the one of the objects associated with the one storage location based on a subsequent image captured by the image sensing device and having the correction factor applied thereto.

10. The system of claim 1, wherein the applying of the correction factor to subsequent images comprises adjusting settings of the image sensing device such that color attributes of subsequent images captured by the image sensing device match the known color attributes.

2. An inventory control system for determining an inventory condition of objects stored in the system, the system comprising:

at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects, wherein each drawer is associated with at least two known reference patterns distinct from the storage locations and objects;

an image sensing device configured to capture a partial image of one of the storage drawers along with a segment of each of the associated reference patterns located within the partial image, wherein the reference patterns are provided in such a manner that at least a segment of each of the two known reference patterns is in every image taken by the image sensing device;

a data storage system storing, for each storage drawer, location information of segments of the reference patterns in each drawer; and

a data processor configured to;

access image data of the partial image of the drawer captured by the image sensing device;

extract information related to the segment of each of the two reference patterns in the partial image of the drawer;

access the location information stored in the data storage system; and

determine a portion of the drawer to which the partial image corresponds, based on the location information and the segment of each of the reference patterns, distinct from the storage locations and objects, included in the partial image.

3. The system of claim 2, wherein the image sensing device is configured to capture multiple partial images of the drawer, and the data processor determines, for each of the multiple partial images taken by the image sensing device, a portion of the drawer to which each of the multiple partial images corresponds based on the segments of each of the reference patterns in each partial image and the location information stored in the data storage system.

4. The system of claim 2, wherein the multiple partial images are stitched together to form a larger image of the drawer.

11. The system of claim 2, wherein the location information stored in the data storage system associates segments of the reference patterns in each respective drawer with corresponding portions of the respective drawer, and

the location information and the extracted information related to the segment of each of the two known reference patterns in any image taken by the image sensing device are sufficient to determine the portion of the storage drawer captured in the partial image.

16. The system of claim 2, wherein the image sensing device comprises a plurality of cameras having overlapping viewing fields jointly forming a field of view of the image sensing device, and every image taken by the image sensing device is taken by the plurality of cameras across the field of view of the image sensing device to include at least a segment of each of the two known reference patterns.

5. An inventory control system comprising:

at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects, wherein each drawer is associated with at least two known reference patterns distinct from the storage locations and objects;

an image sensing device configured to capture partial images of a storage drawer along with segments of each of the associated reference patterns located within the captured partial images, wherein the reference patterns are provided in such a manner that at least a segment of each of the two known reference patterns is in every image taken by the image sensing device;

a data storage system storing, for each storage drawer, location information of segments of the reference patterns in each drawer; and

a data processor configured to;

access image data of a first partial image and a second partial image of the drawer captured by the image sensing device;

extract information related to the segment of each of the two reference patterns in each of the first partial image and the second partial image;

access the location information stored in the data storage system;

determine portions of the drawer to which the first partial image and the second partial image respectively correspond based on the location information and the segments of each of the reference patterns, distinct from the storage locations and objects, included in the first and second partial images; and

determine a moving speed of the drawer based on the portions of the drawer to which the first partial image and the second partial image correspond, and a time difference between the first partial image and the second partial image.

6. The system of claim 5, wherein the data processor determines an imaging frequency according to the moving speed, and the image sensing device captures images of the drawer based on the determined imaging frequency.

12. The system of claim 5, wherein the location information stored in the data storage system associates segments of the reference patterns in each respective drawer with corresponding portions of the respective drawer, and

the location information and the extracted information related to the segments of each of the two known reference patterns in the first and second partial images are sufficient to determine the portion of the storage drawer captured in the first and second partial images.

17. The system of claim 5, wherein the image sensing device comprises a plurality of cameras having overlapping viewing fields jointly forming a field of view of the image sensing device, and every image taken by the image sensing device is taken by the plurality of cameras across the field of view of the image sensing device to include at least a segment of each of the two known reference patterns.

7. An inventory control system comprising:

at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects, wherein each drawer is associated with at least two known reference patterns distinct from the storage locations and objects;

an image sensing device configured to capture at least a partial image of a storage drawer along with a segment of each of the associated reference patterns located within the captured image, wherein the reference patterns are provided in such a manner that at least a segment of each of the two known reference patterns is in every image taken by the image sensing device;

a data storage system storing, for each storage drawer, location information of segments of the reference patterns in each drawer; and

a data processor configured to;

access image data of the captured image of the drawer captured by the image sensing device;

extract information related to the segment of each of the two reference patterns in the captured image;

access the location information stored in the data storage system;

determine a portion of the drawer to which the partial image corresponds based on the location information and the segment of each of the reference patterns, distinct from the storage locations and objects, included in the captured image;

determine a needed adjustment to the captured image based on the determined portion of the drawer; and

apply the needed adjustment to the captured image.

8. The system of claim 7, wherein the determined adjustment is applied to subsequent images taken by the image sensing device.

13. The system of claim 7, wherein the location information stored in the data storage system associates segments of the reference patterns in each respective drawer with corresponding portions of the respective drawer, and

the location information and the extracted information related to the segment of each of the two known reference patterns in any image taken by the image sensing device are sufficient to determine the portion of the storage drawer captured in the partial image.

14. The system of claim 7, wherein the needed adjustment applied to the captured image is one of a resealing, rotation, or perspective adjustment of the captured image to remove distortion therefrom.

18. The system of claim 7, wherein the image sensing device comprises a plurality of cameras having overlapping viewing fields jointly forming a field of view of the image sensing device, and every image taken by the image sensing device is taken by the plurality of cameras across the field of view of the image sensing device to include at least a segment of each of the two known reference patterns.

9. An inventory control system comprising:

at least one storage drawer, each storage drawer including a plurality of storage locations for storing objects;

an image sensing device configured to capture an image of a target including known reference patterns, wherein relative spatial relationships of the target and the reference patterns are known;

a data storage system storing information of the known spatial relationships of the reference patterns; and

a data processor configured to;

access image data of the image of the target captured by the image sensing device;

extract information related to the reference patterns in the captured image;

access the information of the known spatial relationships of the reference patterns stored in the data storage system;

determine an orientation of the target in the image based on a difference between the reference patterns in the captured image and the known spatial relationships of the target and the reference patterns;

determine a needed adjustment to the captured image based on the determined orientation of the target; and

apply the needed adjustment to subsequent images captured by the image sensing device.

15. The system of claim 9, wherein the needed adjustment applied to the captured image is one of a resealing, rotation, or perspective adjustment of the captured image to remove distortion therefrom.