STORE SHELF IMAGING SYSTEM AND METHOD

• US 20180108120A1
• Filed: 10/17/2016
• Published: 04/19/2018
• Est. Priority Date: 10/17/2016
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Venable, Dennis L., Wu, Wencheng, Wade, Thomas F., Shen, Ethan, Rizzolo, Charles D.
• Granted Patent: US 10,210,603 B2
• Time in Patent Office: Days
• CPC Class Codes: B65G 1/0492 with cars adapted to travel in storage aisles
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A store profile generation system includes a mobile base and an image capture assembly mounted on the base. The assembly includes at least one image capture device for acquiring images of product display units in a retail environment. A control unit acquires the images captured by the at least one image capture device at a sequence of locations of the mobile base in the retail environment. The control unit extracts product-related data from the acquired images and generates a store profile indicating locations of products and their associated tags throughout the retail environment, based on the extracted product-related data. The store profile can be used for generating new product labels for a sale in an appropriate order for a person to match to the appropriate locations in a single pass through the store.

Claims

1. A robotic system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within an operating environment, the operating environment including a plurality of objects substantially horizontally aligned parallel to one or more aisles along which the mobile base travels and the location of the mobile base determined by the primary navigation component including a distance of the mobile base along an aisle being traveled by the mobile base;

one or more image capture devices operatively attached to the mobile base and configured to acquire images of the plurality of objects at a predefined resolution and field of view (FOV) as the mobile base travels along the one or more aisles;

a master control unit operatively associated with the mobile base and the one or more image capture devices, the master control unit including an operatively associated memory configured to store instructions and an operatively associated processor configured to execute the instructions, the master control unit instructions configured to instruct the mobile base to travel along a first aisle, instruct the one or more image capture devices to acquire sequential overlapping images including an overlapping portion and a non-overlapping portion of one or more objects along the first aisle, and processing the sequential overlapping images to perform image-based accuracy correction to generate a corrected location of the mobile base, the corrected location of the mobile base calculated by determining a shift distance associated with aligning the overlapping portions of the sequential overlapping images, the shift distance applied to the location of the mobile base along the aisle provided by the primary navigation component.

2. The robotic system according to claim 1, wherein the primary navigation system is based on one of LIDAR (Light Detection and Ranging) and Kinect®

.

3. The robotic system according to claim 1, wherein the robotic system is configured as a SLAM (Simultaneous Localization and Mapping) platform.

4. The robotic system according to claim 1, wherein the image-based accuracy correction performs one or more of a cross correlation function between the acquired sequentially overlapping images to calculate the shift distance, and a Harris Corners Detector function to identify a most frequent shift distance associated with the acquired sequentially overlapping images to calculate the shift distance.

5. The robotic system according to claim 1, wherein the primary navigation component is configured to determine the location of the mobile base with a positional error substantially equal to or greater than +/−

40 mm and the image-based accuracy correction process is configured to generate the corrected location of the mobile base with a positional error less than +/−

40 mm.

6. The robotic system according to claim 1, wherein the mobile base acquires the sequentially overlapping images during one of a continuous movement mode of the mobile base, and a stop and go movement mode of the mobile base.

7. The robotic system according to claim 1, wherein the image-based accuracy correction performs a cross correlation function configured to:

overlap the sequential overlapping images by a horizontal separation distance traveled by the mobile base from a first imaging location associated with acquiring a first of the sequential overlapping images to a second imaging location associated with acquiring a second of the sequential overlapping images, the horizontal separation distance calculated based on the location of the mobile base along the first aisle provided by the primary navigation system;

determine an optimal shift that yields a minimum sum of absolute differences associated with the overlapped sequential overlapping images to horizontally align the overlapping portions of the sequential overlapping images; and

apply the optimal shift as a correction to the location of the mobile base along the aisle at the second imaging location provided by the primary navigation component.

8. The robotic system according to claim 7, wherein the cross correlation function is performed for a plurality of images captured at one or both of the first imaging location and the second imaging location, and the correction to the location of the mobile base along the aisle is based on an average or median of the optimal shifts associated with the plurality of images captured at one or both of the first imaging location and the second imaging location.

9. The robotic system according to claim 7, wherein a search range of S_opt associated with finding the optimal shift that yields the minimum sum of absolute differences is limited based on a priori information associated with the robotic system and/or operating environment.

10. The robotic system according to claim 1, further comprising:

a configuration component configured to spatially characterize the one or more image capture devices to map pixels of a captured image to locations in a real space.

11. The robotic system according to claim 10, wherein the image-based accuracy correction performs one or more of a cross correlation function between the acquired sequentially overlapping images to calculate the shift distance, and a Harris Corners Detector function to identify a most frequent shift distance associated with the acquired sequentially overlapping images to calculate the shift distance.

12. A continuous motion store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

an image capture assembly mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units;

a master control unit including an operatively associated memory which stores instructions and a processor for executing the instructions, the control unit instructions including instructions for processing the images acquired by the plurality of image capture devices at a sequence of locations during a continuous movement of the mobile base in the retail environment, thereafter extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the continuous movement of the mobile base, and generating a store profile indicating locations of the products and/or tags throughout the retail environment based on the extracted product-related data, a spatial characterization of the image capture assembly, and the locations of the mobile base at the sequence of locations provided by the primary navigation component during continuous movement of the mobile base during an interval of time that the images were acquired.

13. The continuous motion store profile generation system according to claim 12, wherein the image-based accuracy correction performs one or more of a cross correlation function between the acquired sequentially overlapping images to calculate the shift distance, and a Harris Corners Detector function to identify a most frequent shift distance associated with the acquired sequentially overlapping images to calculate the shift distance.

14. The continuous motion store profile generation system according to claim 12, wherein the primary navigation component is configured to determine the location of the mobile base with a positional error substantially equal to or greater than +/−

40 mm and the image-based accuracy correction process is configured to generate the corrected location of the mobile base with a positional error less than +/−

40 mm.

15. The continuous motion store profile generation system according to claim 12, wherein the image-based accuracy correction performs a cross correlation function configured to:

overlap the sequential overlapping images by a horizontal separation distance traveled by the mobile base from a first imaging location associated with acquiring a first of the sequential overlapping images to a second imaging location associated with acquiring a second of the sequential overlapping images, the horizontal separation distance calculated based on the location of the mobile base along the first aisle provided by the primary navigation system;

determine an optimal shift that yields a minimum sum of absolute differences associated with the overlapped sequential overlapping images to horizontally align the overlapping portions of the sequential overlapping images; and

apply the optimal shift as a correction to the location of the mobile base along the aisle at the second imaging location provided by the primary navigation component.

16. The continuous motion store profile generation system according to claim 15, wherein the cross correlation function is performed for a plurality of images captured at one or both of the first imaging location and the second imaging location, and the correction to the location of the mobile base along the aisle is based on an average or median of the optimal shifts associated with the plurality of images captured at one or both of the first imaging location and the second imaging location.

17. The continuous motion store profile generation system according to claim 15, wherein a search range of S_opt associated with finding the optimal shift that yields the minimum sum of absolute differences is limited based on a priori information associated with the robotic system and/or operating environment.

18. The continuous motion store profile generation system according to claim 12, further comprising:

a configuration component configured to spatially characterize the one or more image capture devices to map pixels of a captured image to locations in a real space.

19. A store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

a high resolution (HR) image capture assembly mounted on the mobile base, the image capture assembly including a plurality of high resolution (HR) image capture devices configured at a first resolution and a first field of view (FOV) to acquire HR images of the product display units; and

a master control unit including an operatively associated memory which stores instructions and a processor for executing the instructions, the control unit instructions including instructions for processing the images acquired by the plurality of image capture devices at a sequence of locations during a continuous movement of the mobile base in the retail environment, thereafter extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the continuous movement of the mobile base, and generating a store profile indicating locations of the products and/or tags throughout the retail environment based on the extracted product-related data, a spatial characterization of the image capture assembly, and the locations of the mobile base at the sequence of locations provided by the primary navigation component during continuous movement of the mobile base during an interval of time that the images were acquired.

20. The store profile generation system according to claim 19, wherein the image-based accuracy correction performs one or more of a cross correlation function between the acquired sequentially overlapping images to calculate the shift distance, and a Harris Corners Detector function to identify a most frequent shift distance associated with the acquired sequentially overlapping images to calculate the shift distance.

21. The store profile generation system according to claim 19, wherein the image-based accuracy correction performs a cross correlation function configured to:

overlap the sequential overlapping images by a horizontal separation distance traveled by the mobile base from a first imaging location associated with acquiring a first of the sequential overlapping images to a second imaging location associated with acquiring a second of the sequential overlapping images, the horizontal separation distance calculated based on the location of the mobile base along the first aisle provided by the primary navigation system;

determine an optimal shift that yields a minimum sum of absolute differences associated with the overlapped sequential overlapping images to horizontally align the overlapping portions of the sequential overlapping images; and

apply the optimal shift as a correction to the location of the mobile base along the aisle at the second imaging location provided by the primary navigation component.

22. The store profile generation system according to claim 21, wherein the cross correlation function is performed for a plurality of images captured at one or both of the first imaging location and the second imaging location, and the correction to the location of the mobile base along the aisle is based on an average or median of the optimal shifts associated with the plurality of images captured at one or both of the first imaging location and the second imaging location.

23. The store profile generation system according to claim 21, wherein a search range of S_opt associated with finding the optimal shift that yields the minimum sum of absolute differences is limited based on a priori information associated with the robotic system and/or operating environment.