Store shelf imaging system and method

• US 10,176,452 B2
• Filed: 10/17/2016
• Issued: 01/08/2019
• Est. Priority Date: 06/13/2014
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Rizzolo, Charles D., Venable, Dennis L., Wu, Wencheng, Wade, Thomas F., Paul, Peter, Shen, Ethan, Williams, Lee Anne S., Hann, Daniel S.
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US15/295,306
• Publication Number:
  US 20170032311A1
• Time in Patent Office: 813 Days
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/201 Price look-up processing, e.g. updating
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/2247 Characters composed of bars, e.g. CMC-7

Abstract

A store profile generation system includes a mobile base and an image capture assembly mounted on the base. The assembly includes at least one image capture device for acquiring images of product display units in a retail environment. A control unit acquires the images captured by the at least one image capture device at a sequence of locations of the mobile base in the retail environment. The control unit extracts product-related data from the acquired images and generates a store profile indicating locations of products and their associated tags throughout the retail environment, based on the extracted product-related data. The store profile can be used for generating new product labels for a sale in an appropriate order for a person to match to the appropriate locations in a single pass through the store.

Claims

1. A continuous motion store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

an image capture assembly mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units; and

a master control unit including an operatively associated memory which stores instructions and a processor for executing the instructions, the control unit instructions including instructions for processing the images acquired by the plurality of image capture devices at a sequence of locations during a continuous movement of the mobile base in the retail environment, thereafter extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the continuous movement of the mobile base, and generating a store profile indicating locations of the products or tags throughout the retail environment based on the extracted product-related data, a spatial characterization of the image capture assembly, and the locations of the mobile base at the sequence of locations provided by the primary navigation component during continuous movement of the mobile base during an interval of time that the images were acquired,

wherein the master control unit is configured to perform image-based accuracy corrections of a location of the mobile base reported by the primary navigation component operatively associated with the mobile base, and

wherein the image-based accuracy correction includes processing two sequentially acquired images of a product display unit which include an overlapping portion and a non-overlapping portion, and determining a shift distance associated with aligning the overlapping portions of the two sequentially acquired images, the shift distance applied to the location of the mobile base reported by the navigation component operatively associated with the mobile base to generate a corrected location of the mobile base.

2. The continuous motion store profile generation system of claim 1, wherein the product-related data includes at least one of printed barcodes and text.

3. The continuous motion store profile generation system of claim 1, wherein at least one of the image capture devices has a first position and a second position, vertically spaced from the first position, for acquiring images of the product display units in the first and second positions.

4. The continuous motion store profile generation system of claim 3, wherein the at least one image capture device includes a plurality of image capture devices, each having a first position and a second position, vertically spaced from the first position, for acquiring images of the product display units in the first and second positions of each of the image capture devices.

5. The continuous motion store profile generation system of claim 1, wherein the master control unit translates at least one of at least two vertically spaced acquired images based on a computed spatial profile of each of the image capture devices.

6. The continuous motion store profile generation system of claim 5, wherein the computed spatial profile of the at least one image capture device includes a mapping between locations for pixels of an acquired image and locations in a real space.

7. The continuous motion store profile generation system of claim 1, wherein the master control unit generates an image of one of the product display units using a low resolution full fixture field of view (FFFOV) camera independent from the plurality of image capture devices, the FFFOV camera acquiring images at a resolution less than the resolution of the plurality of image capture devices.

8. The continuous motion store profile generation system of claim 1 wherein the master control unit generates a packaging order and/or printing order for a set of replacement product labels based on the generated store profile.

9. The continuous motion store profile generation system according to claim 1, wherein the image-based accuracy correction performs one or more of a cross correlation function between the two sequentially acquired images and a Harris Corners Detector function to identify a most frequent shift distance associated with the two sequentially acquired images.

10. The continuous motion store profile generation system according to claim 1, further comprising:

a plurality of distributed controllers, each of the distributed controllers operatively associated with one of the plurality of image capture devices and operatively associated with the master control unit, the distributed controllers configured to operate in parallel and independently drive a respective image capture device, store acquired images of the product display units, and communicate the stored acquired images of the product display units to the master control unit.

11. A continuous motion store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

an image capture assembly mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units;

a master control unit including an operatively associated memory which stores instructions and a processor for executing the instructions, the control unit instructions including instructions for processing the images acquired by the plurality of image capture devices at a sequence of locations during a continuous movement of the mobile base in the retail environment, thereafter extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the continuous movement of the mobile base, and generating a store profile indicating locations of the products or tags throughout the retail environment based on the extracted product-related data, a spatial characterization of the image capture assembly, and the locations of the mobile base at the sequence of locations provided by the primary navigation component during continuous movement of the mobile base during an interval of time that the images were acquired; and

a secondary navigation component operatively associated with the continuous motion store profile generation system, the secondary navigation component supplementing the primary navigation component to generate a corrected location of the mobile base,

wherein the primary navigation component is configured to provide the location of the mobile base with a positional error substantially equal to or greater than +/−

40 mm and the secondary navigation component is configured to generate the corrected location of the mobile base with a positional error less than +/−

40 mm.

12. A continuous motion store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

an image capture assembly mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units;

a master control unit including an operatively associated memory which stores instructions and a processor for executing the instructions, the control unit instructions including instructions for processing the images acquired by the plurality of image capture devices at a sequence of locations during a continuous movement of the mobile base in the retail environment, thereafter extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the continuous movement of the mobile base, and generating a store profile indicating locations of the products or tags throughout the retail environment based on the extracted product-related data, a spatial characterization of the image capture assembly, and the locations of the mobile base at the sequence of locations provided by the primary navigation component during continuous movement of the mobile base during an interval of time that the images were acquired; and

a secondary navigation component operatively associated with the continuous motion store profile generation system, the secondary navigation component supplementing the primary navigation component to generate a corrected location of the mobile base along the aisle within the retail environment,

wherein the primary navigation component is configured to provide the location of the mobile base along the aisle with a positional error substantially equal to or greater than +/−

40 mm and the secondary navigation component is configured to generate the corrected location of the mobile base along the aisle with a positional error less than +/−

40 mm.

13. A continuous motion store profile generation method performed with a continuous motion store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

an image capture assembly mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units; and

a master control unit including an operatively associated memory which stores instructions and a processor for executing the instructions, the control unit instructions including instructions for processing the images acquired by the plurality of image capture devices at a sequence of locations during a continuous movement of the mobile base in the retail environment, thereafter extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the continuous movement of the mobile base, and generating a store profile indicating locations of the products or tags throughout the retail environment based on the extracted product-related data, a spatial characterization of the image capture assembly, and the locations of the mobile base at the sequence of locations provided by the primary navigation component during continuous movement of the mobile base during an interval of time that the images were acquired,

wherein the continuous motion store profile generation method comprises;

continuously moving the mobile base around the retail environment;

the image capture assembly mounted on the mobile base acquiring images of the product display units at a continuous sequence of locations of the mobile base while the mobile base is continuously moving; and

the master control unit extracting product-related data from the acquired images and generating a store profile indicating locations of products or tags throughout the retail environment based on the extracted product-related data and the locations of the mobile base provided by the primary navigation component during an interval of time that the images were acquired.

14. The continuous motion store profile generation method of claim 13, wherein the product-related data includes at least one of printed barcodes and text.

15. The continuous motion store profile generation method of claim 13, wherein acquiring images of the product display units includes acquiring images at each of a plurality of vertically-spaced positions for each of the continuous sequence of locations of the mobile base.

16. The continuous motion store profile generation method of claim 15, wherein the master control unit generates an image of one of the product display units using a low resolution full fixture field of view (FFFOV) camera independent from the plurality of image capture devices, the FFFOV camera acquiring images at a resolution less than the plurality of image capture devices.

17. The continuous motion store profile generation method of claim 13, further comprising:

generating a packaging order and/or printing order for a set of replacement product labels based on the constructed store profile.

18. A store profile generation system comprising:

a mobile base including an operatively associated primary navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more of the product display fixtures, and a plurality of tags attached to the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture near the tag;

an image capture assembly mounted on the mobile base, the image capture assembly including at least one image capture device for acquiring images of the product display units in the retail environment;

a configuration component which spatially characterizes the image capture assembly by computing a spatial profile for each of a plurality of vertically-spaced image capture device positions from captured images of a printed calibration target that includes identifiable landmarks;

a master control unit which processes images captured by the at least one image capture device at a sequence of locations during a continuous movement of the mobile base in the retail environment, the master control unit extracting the product-related data from the acquired images and generating a store profile indicating locations of one or both of the products and tags throughout the retail environment based on the extracted product-related data, the spatial profiles of the image capture devices, and the locations of the mobile base during an interval of time that the images were acquired; and

a secondary navigation component, the secondary navigation component supplementing the primary navigation component to generate a corrected location of the mobile base, the corrected location of the mobile base more accurate than the location of the mobile base determined by the primary navigation component,

wherein the secondary navigation component is configured to perform image-based accuracy corrections of a location of the mobile base reported by the primary navigation component operatively associated with the mobile base, and

wherein the image-based accuracy correction includes processing two sequentially acquired images of a product display unit which include an overlapping portion and a non-overlapping portion, and determining a shift distance associated with aligning the overlapping portions of the two sequentially acquired images, the shift distance applied to the location of the mobile base reported by the navigation component operatively associated with the mobile base to generate a corrected location of the mobile base.

19. The store profile generation system according to claim 18, wherein the image-based accuracy correction performs one or more of a cross correlation function between the two sequentially acquired images and a Harris Corners Detector function to identify a most frequent shift distance associated with the two sequentially acquired images.

20. The store profile generation system according to claim 19, further comprising:

a plurality of distributed controllers, each of the distributed controllers operatively associated with one of the plurality of image capture devices and operatively associated with the master control unit, the distributed controllers configured to operate in parallel and independently drive a respective image capture device, store acquired images of the product display units, and communicate the stored acquired images of the product display units to the master control unit.