System and method for performing inventory using a mobile inventory robot

• US 7,693,757 B2
• Filed: 09/21/2006
• Issued: 04/06/2010
• Est. Priority Date: 09/21/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Zimmerman, Thomas Guthrie
• Primary Examiner(s): Gart; Matthew S
• Assistant Examiner(s): GLASS, RUSSELL S
• Application Number:
  US11/534,162
• Publication Number:
  US 20080077511A1
• Time in Patent Office: 1,293 Days
• Field of Search: None
• US Class Current: 705/28
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0261 using magnetic plots
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

A mobile inventory robot system generates an inventory map of a store and a product database when a mobile inventory robot is manually navigated through the store to identify items on shelves, a location for each of the items on the shelves, and a barcode for each of the items. The system performs inventory of the items by navigating through the store via the inventory map, capturing a shelf image, decoding a product barcode from the captured shelf image, retrieving a product image for the decoded product barcode from the product database, segmenting the captured shelf image to detect an image of an item on the shelves, determining whether the detected image matches the retrieved image and, if not, setting an out-of-stock flag for an the item.

Claims

1. A processor-implemented method for performing inventory of products with identifiers disposed on a plurality of shelves in a store, using a mobile inventory robot, the method comprising:

emitting ultrasonic signals from the mobile inventory robot;

detecting a returned ultrasonic signal reflected from obstacles in the store;

navigating the mobile inventory robot through aisles in the store and avoiding obstacles based on the detected ultrasonic signals wherein the mobile inventory robot arrives at a shelf in the store;

wherein navigating the mobile inventory robot through the store comprises locating at least one of a plurality of retro-reflective tracking tags, and

wherein the retro-reflective tracking tags are secured to a fixed structure of the store to assist the mobile inventory robot in determining a location of the mobile inventory robot within the store;

capturing an image of the shelf;

decoding a product barcode from the captured shelf image;

segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

retrieving a product visual descriptor from the decoded product barcode;

determining whether the detected image of the at least one of the products matches the retrieved product visual descriptor; and

upon determination that the detected image of the at least one of the products does not match the retrieved product visual descriptor, setting an out-of-stock flag for the product corresponding to the decoded product identifier;

determining whether the at least one of the products corresponds to the decoded product barcode; and

generating a flag when the at least one of the products on the shelf is located in a wrong location if the at least one of the products does not correspond to the decoded product barcode for the shelf.

2. The method of claim 1, wherein navigating through the store further comprises using an inventory map of the store.

3. The method of claim 2, further comprising generating the inventory map of the store by manually tracking the mobile inventory robot through the store, in order to identify the identifiers and locations of the products on the shelves.

4. The method of claim 1, wherein retrieving the product visual descriptor from the decoded product identifier comprises generating a product database that contains a listing of:

the products, the location of the products, and the identifiers of the products.

5. The method of claim 4, wherein generating the product database comprises:

capturing an image of a shelf

decoding the product barcode from the captured shelf image;

segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

extracting the product visual descriptor of the products image, wherein the product visual descriptor is selected from any one or more of;

a list of images, sizes, shapes, colors, bar code and textures; and

placing the product visual descriptor in a product database.

6. The method of claim 1, wherein capturing the shelf image is performed by an imaging device.

7. The method of claim 6, further comprising a light source that is located in the vicinity of the imaging device;

and

a retro-reflecting material located against a back wall of the shelf, resulting in a reflected optical signal when no products are located on the shelf, indicating an out-of-stock condition.

8. The method of claim 6, wherein the imaging device is mounted on a flexible section capable of moving the imaging device vertically, in order to image the products on the shelves at different heights.

9. The method of claim 1, further comprising using an RFID reader to read RFID tags of objects selected from any of:

a list of products, boxes, cartons, and pallets.

10. The method of claim 9, wherein the RFID reader is mounted on a flexible section capable of moving the RFID reader vertically, in order to read RFID tags located at different heights.

11. The method of claim 1, wherein the mobile inventory robot communicates via a wireless network with an inventory control system at a remote location.

12. The method of claim 1, wherein the mobile inventory robot is controlled by a remote operator.

13. The method of claim 12, wherein the mobile inventory robot comprises an audio input and an audio output to enable the remote operator to converse with a person located in proximity of the mobile inventory robot.

14. The method of claim 1, wherein the product visual descriptor is selected from any one or more of:

a list of images, sizes, colors, and textures.

15. A processor-implemented system for performing inventory of products with identifiers disposed on a plurality of shelves in a store, using a mobile inventory robot, comprising:

a tracking system, including ultrasonic range finders, for navigating the mobile inventory robot to a shelf through aisles in the store;

wherein the tracking system navigates the mobile inventory robot through the store by locating at least one of a plurality of retro-reflective tracking tags, and

wherein the tracking tags are secured to a fixed structure of the store to assist the mobile inventory robot in determining a location of the mobile inventory robot within the store;

an imaging device mounted on the mobile inventory robot for capturing an image of the shelf;

a decoder for decoding a product barcode from the captured shelf image;

a product image segmentation module for segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

an inventory control module for retrieving a product visual descriptor from the decoded product barcode;

an out-of-stock detector for determining whether the detected image of the at least one of the products matches the retrieved product visual descriptor; and

upon determination by a product visual descriptor that the detected image of the at least one of the products does not match the retrieved product visual descriptor, the inventory control module setting an out-of-stock flag for the product corresponding to the decoded product identifier,

and upon determination that the at least one of the products does not match the product barcode, the inventory control module setting a misplaced item flag for at least one of the products.

16. The system of claim 15, wherein the tracking system tracks through the store using an inventory map of the store.

17. The system of claim 16, further comprising a mapping system for generating the inventory map of the store by manually navigating the mobile inventory robot through the store using a manual control system, in order to identify the identifiers and locations of the products on the shelves.

18. The system of claim 15, wherein the inventory control module retrieves the product visual descriptor from the decoded product identifier by generating a product database that contains a listing of:

the products, the location of the products, and the identifiers of the products.

19. The system of claim 15, wherein the imaging device captures an image of a shelf;

the decoder decodes the product barcode from the captured shelf image;

the product image segmentation module segments the captured shelf image to detect an image of at least one of the products on the shelves;

the product visual descriptor is selected from any one or more of;

a list of images, sizes, shapes, colors, bar code and textures; and

a product datastore that contains the product visual descriptor.

20. The system of claim 15, further comprising a light source located in the vicinity of the imaging device;

and

a retro-reflecting material located against a back wall of the shelf, resulting in a reflected optical signal when no products are located on the shelf, indicating an out-of-stock condition.

21. The system of claim 15, wherein the imaging device is mounted on a flexible section capable of moving the imaging device vertically, in order to image the products on the shelves at different heights.

22. The system of claim 15, further comprising an RFID reader for reading RFID tags of objects selected from any of:

a list of products, boxes, cartons, and pallets.

23. The system of claim 22, wherein the RFID reader is mounted on a flexible section capable of moving the RFID reader vertically, in order to read RFID tags located at different heights.

24. The system of claim 15, wherein the mobile inventory robot communicates via a wireless network with an inventory control system at a remote location.

25. The system of claim 15, wherein the mobile inventory robot is controlled by a remote operator.

26. The system of claim 25, wherein the mobile inventory robot comprises an audio input and an audio output to enable the remote operator to converse with a person located in proximity of the mobile inventory robot;

and

wherein the product visual descriptor is selected from any one or more of;

a list of images, sizes, colors, and textures.

27. A computer program product that includes executable instruction codes stored on a computer readable medium, for performing inventory of products with identifiers disposed on a plurality of shelves in a store, using a mobile inventory robot, the computer program product comprising:

a set of instruction codes for navigating the mobile inventory robot around obstacles in the store using ultrasonic signals wherein the mobile inventory robot arrives at a shelf in through the store;

wherein navigating the mobile inventory robot around obstacles in the store comprises locating at least one of a plurality of retro-reflective tracking tags, and

wherein the retro-reflective tracking tags are secured to a fixed structure of the store to assist the mobile inventory robot in determining a location of the mobile inventory robot within the store;

a set of instruction codes for capturing an image of the shelf;

a set of instruction codes for decoding a product barcode from the captured shelf image;

a set of instruction codes for segmenting the captured shelf image to detect an image of at least one of the products on the shelves;

a set of instruction codes for retrieving a product visual descriptor from the decoded product barcode;

a set of instruction codes for determining whether the detected image of the at least one of the products matches the retrieved product visual descriptor;

a set of instruction codes for setting an out-of-stock flag for the product corresponding to the decoded product identifier, upon determination by a product visual descriptor that the detected image of the at least one of the products does not match the retrieved product visual descriptor, and

a set of instruction codes generating a flag that the at least one of the products on the shelf is located in a wrong location if the at least one of the products does not correspond to the decoded product barcode for the shelf.