Store shelf imaging system and method

• US 10,289,990 B2
• Filed: 10/17/2016
• Issued: 05/14/2019
• Est. Priority Date: 10/17/2016
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Rizzolo, Charles D., Venable, Dennis L., Wu, Wencheng, Wade, Thomas F., Paul, Peter, Shen, Ethan, Williams, Lee Anne, Hann, Daniel S.
• Primary Examiner(s): Bekele, Mekonen T
• Application Number:
  US15/295,031
• Publication Number:
  US 20180107999A1
• Time in Patent Office: 939 Days
• CPC Class Codes: B65G 1/0492 with cars adapted to travel in storage aisles
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G06K 19/06028 using bar codes
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/201 Price look-up processing, e.g. updating
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04W 4/30 Services specially adapted for particular environments, situations or purposes

Abstract

A store profile generation system includes a mobile base and an image capture assembly mounted on the base. The assembly includes at least one image capture device for acquiring images of product display units in a retail environment. A control unit acquires the images captured by the at least one image capture device at a sequence of locations of the mobile base in the retail environment. The control unit extracts product-related data from the acquired images and generates a store profile indicating locations of products and their associated tags throughout the retail environment, based on the extracted product-related data. The store profile can be used for generating new product labels for a sale in an appropriate order for a person to match to the appropriate locations in a single pass through the store.

Claims

1. A store profile generation system comprising:

a robotic mobile base including an operatively associated primary navigation component configured to determine a location of the robotic mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

a high resolution (HR) digital image capture assembly mounted on the robotic mobile base, the HR digital image capture assembly including a plurality of high resolution (HR) digital image capture devices configured at a first resolution and a first field of view (FOV) to acquire HR digital images of the product display units;

a low resolution (LR) digital image capture device configured at a second resolution and a second FOV to acquire LR digital images of the product display units, the second resolution less than the first resolution associated with the HR digital image capture devices and the second FOV horizontally wider and vertically higher than the first FOV associated with the HR digital image capture devices; and

a master control unit operatively associated with the robotic mobile base, the plurality of HR digital image capture devices; and

the LR digital image capture device, the master control unit including an operatively associated memory configured to store instructions and an operatively associated processer configured to execute the instructions, the master control unit instructions configured to process the acquired HR digital images from the plurality of HR digital image capture devices and the acquired LR digital images from the LR digital image capture device at one or more sequential locations of the robotic mobile base along one or more of the plurality of aisles, the process using the acquired HR digital images to identify the barcodes associated with the plurality of tags and generate a store profile indicating locations of the plurality of tags and their respective products in the retail environment, and the process using the acquired LR digital image to generate a full planogram image of the product display units, the full planogram image including a display facing profile indicating the presence or non-presence of the plurality of products and their respective locations in the retail environment,

wherein the master control unit is configured to perform image-based accuracy corrections of a location of the robotic mobile base reported by the primary navigation component operatively associated with the robotic mobile base, and

wherein the image-based accuracy correction includes processing two sequentially acquired digital images of a product display unit which include an overlapping portion and a non-overlapping portion, and determining a shift distance associated with aligning the overlapping portions of the two sequentially acquired digital images, the shift distance applied to the location of the robotic mobile base reported by the primary navigation component operatively associated with the robotic mobile base to generate a corrected location of the robotic mobile base.

2. The store profile generation system according to claim 1, wherein the robotic mobile base, HR digital image capture assembly, LR digital image capture assembly and master control unit operate in one or both of a continuous movement mode and a stop/go movement mode.

3. The store profile generation system according to claim 1, wherein the product display fixtures includes one or more of shelving and pegboards.

4. The store profile generation system according to claim 1, wherein the master control unit instructions are configured to acquire the HR digital images at a first sequence of locations with the robotic mobile base traveling at a first speed and acquire the LR digital image at a second sequence of locations with the robotic mobile base traveling at a second speed, the second speed greater than the first speed.

5. The store profile generation system according to claim 1, wherein the master control unit instructions are configured to use a spatial characterization of one or both of the plurality of HR digital image capture devices and the LR digital image capture device to generate the locations of the plurality of tags and their respective products, the spatial characterization mapping locations of pixels associated with acquired HR digital images and/or acquired LR digital images to locations in a real space.

6. The store profile generation system according to claim 1, wherein the master control unit generates an order for a set of replacement tags based on the generated store profile.

7. The store profile generation system according to claim 1, wherein the image-based accuracy correction performs one or more of a cross correlation function between the two sequentially acquired digital images to calculate the shift distance and a Harris Corners Detector function to identify a most frequent shift distance associated with the two sequentially acquired digital images.

8. The store profile generation system according to claim 1, further comprising:

a plurality of distributed controllers, each of the distributed controllers operatively associated with one of the plurality of HR digital image capture devices and operatively associated with the master control unit, the distributed controllers configured to operate in parallel and independently drive a respective digital image capture device, store acquired digital images of the product display units, and communicate the stored acquired digital images of the product display units to the master control unit.

9. The store profile generation system according to claim 1, further comprising:

one or more lamps attached to the robotic mobile base and configured to illuminate the product display units.

10. The store profile generation system according to claim 1, wherein the system is configured to control a focal distance and illumination associated with one or more of the plurality of HR digital image capture devices and the LR digital image capture device.

11. The store profile generation system according to claim 1, wherein the first resolutions associated with the plurality of HR digital image capture devices is 200 ppi (pixels per inch) or greater, and the second resolution associated with the LR digital image capture device is less than 200 ppi and equal to or greater than 20 ppi.

12. The store profile generation system according to claim 1, wherein the primary navigation component is configured to provide the location of the robotic mobile base with a positional error substantially equal to or greater than +/−

40 mm and a secondary navigation component is configured to generate the image-based corrected location of the robotic mobile base with a positional error less than +1-40 mm.

13. A store profile generation method performed with the store profile generation system of claim 1, the method comprising:

moving the robotic mobile base around the retail environment;

the digital image capture assembly acquiring digital images of the product display units at a sequence of locations of the robotic mobile base; and

a plurality of distributed controllers, each of the distributed controllers operatively associated with one of the plurality of digital image capture devices and operatively associated with the master control unit, the distributed controllers configured to operate in parallel and independently drive a respective digital image capture device, store acquired digital images of the product display units, and communicate the stored acquired digital images of the product display units to the master control unit.

14. A store profile generation system comprising:

a robotic mobile base including an operatively associated primary navigation component configured to determine a location of the robotic mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on the product display fixtures and a plurality of tags attached to the product display fixtures, each tag including product related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

a digital image capture assembly mounted on the robotic mobile base, the digital image capture assembly including a plurality of digital image capture devices vertically arranged to acquire digital images of the product display fixture and cover an entire vertical height of the product display fixture;

a plurality of distributed controllers, each of the distributed controllers operatively associated with one of the plurality of digital image capture devices and operatively associated with a master control unit, the distributed controllers configured to operate in parallel and independently drive a respective digital image capture device, store acquired digital images of the product display units, and communicate the stored acquired digital images of the product display units to the master control unit;

a secondary navigation component, the secondary navigation component configured to perform an image-based accuracy correction of a location of the robotic mobile base along an aisle of the retail environment reported by the primary navigation component operatively associated with the robotic mobile base, the image-based accuracy corrections including processing sequentially acquired digital images of a product display unit which includes an overlapping portion and a non-overlapping portion, and determining a shift distance associated with aligning the overlapping portions of the two sequentially acquired images, the shift distance applied to the robotic mobile base location reported by the primary navigation component operatively associated with the robotic mobile base to generate a corrected location of the robotic mobile base.

15. The store profile generation system according to claim 14, wherein the image-based accuracy correction performs one or more of a cross correlation function between two sequentially acquired digital images to determine the shift distance and a Harris Corners Detector function to identify a most frequent shift distance associated with the two sequentially acquired digital images.

16. The store profile generation system according to claim 14, wherein the master control unit instructions are configured to use a spatial characterization of the plurality of digital image capture device to generate the locations of the plurality of tags and their respective products, the spatial characterization mapping locations of pixels associated with the acquired digital images to locations in a real space.

17. A store profile generation system comprising:

a robotic mobile base including an operatively associated primary navigation component configured to determine a location of the robotic mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

a high resolution (HR) digital image capture assembly mounted on the robotic mobile base, the HR digital image capture assembly including a plurality of high resolution (HR) digital image capture devices configured at a first resolution and a first field of view (FOV) to acquire HR digital images at the product display units;

a low resolution (LR) digital image capture device configured at a second resolution and a second FOV to acquire LR digital images of the product display units, the second resolution less than the first resolution associated with the HR digital image capture devices and the second FOV horizontally wider and vertically higher than the first FOV associated with the HR digital image capture devices; and

a master control unit operatively associated with the robotic mobile base, the plurality of HR digital image capture devices; and

the LR digital image capture device, the master control unit including an operatively associated memory configured to store instructions and an operatively associated processer configured to execute the instructions, the master control unit instructions configured to process the acquired HR digital images from the plurality of HR digital image capture devices and the acquired LR digital image from the LR digital image capture device at one or more sequential locations of the robotic mobile base along one or more of the plurality of aisles, the process using the acquired HR digital images to identify the barcodes associated with the plurality of tags and generate a store profile indicating locations of the plurality of tags and their respective products in the retail environment, and the process using the acquired digital LR image to generate a full planogram image of the product display units, the full planogram image including a display facing profile indicating the presence or non-presence of the plurality of products and their respective locations in the retail environment,

wherein the product display units include one or more of shelving and pegboards,

wherein a secondary navigation component performs an image-based accuracy correction including processing two sequentially acquired images of a product display unit which include an overlapping portion and a non-overlapping portion, and determining a shift distance associated with aligning the overlapping portions of the two sequentially acquired digital images, the shift distance applied to the location of the robotic mobile base reported by the primary navigation component operatively associated with the robotic mobile base to generate a corrected location of the robotic mobile base, and

wherein the image-based accuracy correction performs one or more of a cross correlation function between the two sequentially acquired digital images and a Harris Corners Detector function to identify a most frequent shift distance associated with the two sequentially acquired digital images.