System and Method for Performing Inventory Using a Mobile Inventory Robot

• US 20080077511A1
• Filed: 09/21/2006
• Published: 03/27/2008
• Est. Priority Date: 09/21/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Zimmerman, Thomas Guthrie
• Granted Patent: US 7,693,757 B2
• Time in Patent Office: Days
• US Class Current: 705/28
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0261 using magnetic plots
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 2201/0216 Vehicle for transporting goods in a warehouse, factory or similar
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

A mobile inventory robot system generates an inventory map of a store and a product database when a mobile inventory robot is manually navigated through the store to identify items on shelves, a location for each of the items on the shelves, and a barcode for each of the items. The system performs inventory of the items by navigating through the store via the inventory map, capturing a shelf image, decoding a product barcode from the captured shelf image, retrieving a product image for the decoded product barcode from the product database, segmenting the captured shelf image to detect an image of an item on the shelves, determining whether the detected image matches the retrieved image and, if not, setting an out-of-stock flag for an the item.

Claims

1. A processor-implemented method for performing inventory of products with identifiers disposed on a plurality of shelves in a store, using a mobile inventory robot, the method comprising:

navigating the mobile inventory robot to a shelf through the store;

capturing an image of the shelf;

decoding a product barcode from the captured shelf image;

segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

retrieving a product visual descriptor from the decoded product barcode;

determining whether the detected image of the at least one of the products matches the retrieved product visual descriptor; and

upon determination that the detected image of the at least one of the products does not match the retrieved product visual descriptor, setting an out-of-stock flag for the product corresponding to the decoded product identifier.

2. The method of claim 1, wherein navigating the mobile inventory robot through the store comprises locating at least one of a plurality of tracking tags.

3. The method of claim 2, wherein the tracking tags are secured to a fixed structure of the store to assist the mobile inventory robot in determining a location of the mobile inventory robot within the store.

4. The method of claim 1, wherein navigating through the store further comprises using an inventory map of the store.

5. The method of claim 4, further comprising generating the inventory map of the store by manually navigating the mobile inventory robot through the store, in order to identify the identifiers and locations of the products on the shelves.

6. The method of claim 1, wherein retrieving the product visual descriptor from the decoded product identifier comprises generating a product database that contains a listing of:

the products, the location of the products, and the identifiers of the products.

7. The method of claim 1, wherein generating the product database comprises:

capturing an image of a shelf;

decoding the product barcode from the captured shelf image;

segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

extracting the product visual descriptor of the products image, wherein the product visual descriptor is selected from any one or more of;

a list of images, sizes, shapes, colors, bar code and textures; and

placing the product visual descriptor in a product database.

8. The method of claim 1, wherein capturing the shelf image is performed by an imaging device.

9. The method of claim 8, further comprising a light source that is located in the vicinity of the imaging device;

and

a retro-reflecting material located against a back wall of the shelf, resulting in a reflected optical signal when no products are located on the shelf, indicating an out-of-stock condition.

10. The method of claim 8, wherein the imaging device is mounted on a flexible section capable of moving the imaging device vertically, in order to image the products on the shelves at different heights.

11. The method of claim 1, further comprising using an RFID reader to read RFID tags of objects selected from any of:

a list of products, boxes, cartons, and pallets.

12. The method of claim 11, wherein the RFID reader is mounted on a flexible section capable of moving the RFID reader vertically, in order to read RFID tags located at different heights.

13. The method of claim 1, wherein the mobile inventory robot communicates via a wireless network with an inventory control system at a remote location.

14. The method of claim 1, wherein the mobile inventory robot is controlled by a remote operator.

15. The method of claim 14, wherein the mobile inventory robot comprises an audio input and an audio output to enable the remote operator to converse with a person located in proximity of the mobile inventory robot.

16. The method of claim 1, wherein the product visual descriptor is selected from any one or more of:

a list of images, sizes, colors, and textures.

17. A processor-implemented system for performing inventory of products with identifiers disposed on a plurality of shelves in a store, using a mobile inventory robot, the system comprising:

a tracking system for navigating the mobile inventory robot to a shelf through the store;

an imaging device for capturing an image of the shelf;

a decoder for decoding a product barcode from the captured shelf image;

a product image segmentation module for segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

an inventory control module for retrieving a product visual descriptor from the decoded product barcode;

an out-of-stock detector for determining whether the detected image of the at least one of the products matches the retrieved product visual descriptor; and

upon determination by a product visual descriptor that the detected image of the at least one of the products does not match the retrieved product visual descriptor, the inventory control module setting an out-of-stock flag for the product corresponding to the decoded product identifier.

18. The system of claim 17, wherein the tracking system navigates the mobile inventory robot-through the store by locating at least one of a plurality of tracking tags.

19. The system of claim 18, wherein the tracking tags are secured to a fixed structure of the store to assist the mobile inventory robot in determining a location of the mobile inventory robot within the store.

20. The system of claim 17, wherein the tracking system navigates through the store using an inventory map of the store.

21. The system of claim 20, further comprising a mapping system for generating the inventory map of the store by manually navigating the mobile inventory robot through the store using a manual control system, in order to identify the identifiers and locations of the products on the shelves.

22. The system of claim 17, wherein the inventory control module retrieves the product visual descriptor from the decoded product identifier by generating a product database that contains a listing of:

the products, the location of the products, and the identifiers of the products.

23. The system of claim 17, wherein the imaging device captures an image of a shelf;

the decoder decodes the product barcode from the captured shelf image;

the product image segmentation module segments the captured shelf image to detect an image of at least one of the products on the shelves;

the product visual descriptor is selected from any one or more of;

a list of images, sizes, shapes, colors, bar code and textures; and

a product datastore that contains the product visual descriptor.

24. The system of claim 17, further comprising a light source located in the vicinity of the imaging device;

and

a retro-reflecting material located against a back wall of the shelf, resulting in a reflected optical signal when no products are located on the shelf, indicating an out-of-stock condition.

25. The system of claim 17, wherein the imaging device is mounted on a flexible section capable of moving the imaging device vertically, in order to image the products on the shelves at different heights.

26. The system of claim 17, further comprising an RFID reader for reading RFID tags of objects selected from any of:

a list of products, boxes, cartons, and pallets.

27. The system of claim 26, wherein the RFID reader is mounted on a flexible section capable of moving the RFID reader vertically, in order to read RFID tags located at different heights.

28. The system of claim 17, wherein the mobile inventory robot communicates via a wireless network with an inventory control system at a remote location.

29. The system of claim 17, wherein the mobile inventory robot is controlled by a remote operator.

30. The system of claim 29, wherein the mobile inventory robot comprises an audio input and an audio output to enable the remote operator to converse with a person located in proximity of the mobile inventory robot;

and

wherein the product visual descriptor is selected from any one or more of;

a list of images, sizes, colors, and textures.

31. A computer program product that includes executable instruction codes stored on a computer readable medium, for performing inventory of products with identifiers disposed on a plurality of shelves in a store, using a mobile inventory robot, the computer program product comprising:

a set of instruction codes for navigating the mobile inventory robot to a shelf through the store;

a set of instruction codes for capturing an image of the shelf;

a set of instruction codes for decoding a product barcode from the captured shelf image;

a set of instruction codes for segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

a set of instruction codes for retrieving a product visual descriptor from the decoded product barcode;

a set of instruction codes for determining whether the detected image of the at least one of the products matches the retrieved product visual descriptor; and

a set of instruction codes for setting an out-of-stock flag for the product corresponding to the decoded product identifier, upon determination by a product visual descriptor that the detected image of the at least one of the products does not match the retrieved product visual descriptor.