Store shelf imaging system and method using a vertical LIDAR

• US 10,019,803 B2
• Filed: 10/17/2016
• Issued: 07/10/2018
• Est. Priority Date: 10/17/2016
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Venable, Dennis L., Wu, Wencheng, Rizzolo, Charles D., Wade, Thomas F., Shen, Ethan, Pennington, Graham, Williams, Lee Anne, Hann, Daniel S.
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US15/295,634
• Publication Number:
  US 20180108134A1
• Time in Patent Office: 631 Days
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

This disclosure provides a retail environment robotic system including a vertically oriented/scanning LIDAR (Light Detection and Ranging) unit mounted to a mobile base which travels along the aisles of a retail environment such as a store. According to an exemplary embodiment, the LIDAR unit vertically scans product display units along the aisles to generate a depth map associated with detected objects which is further processed to detect missing products, product/shelving tags, shelves, etc.

Claims

1. A retail environment robotic system comprising:

a mobile base including an operatively associated navigation component configured to determine a location of the mobile base within a retail environment, the retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures and a plurality of products arranged on one or more of the product display fixtures;

a vertical scanning LIDAR (Light Detection and Ranging) unit operatively mounted to the mobile base, the LIDAR unit configured to scan in a vertical direction and generate a vertical array of distance and angle values representative of a plurality of object distances and angles from the LIDAR unit to a plurality of objects detected along the vertical scan of the LIDAR unit;

a LIDAR data processing component operatively associated with the LIDAR unit, the LIDAR data processing component configured to process the plurality of vertical arrays of object distances and angles to generate a depth map associated with one or more product display units as the mobile base travels along an aisle, the depth map including a plurality of columns and a plurality of rows where each of the plurality of columns is associated with a single vertical scan of the LIDAR unit at one of a plurality of locations along the aisle, and each of the plurality of rows is associated with one of a plurality of vertical distances; and

a depth map processing component configured to identify regions of the depth map associated with regions of the one or more product display units which include one or more of a missing product, the product, a shelf, a product tag attached to the product, and a shelf tag attached to the shelf.

2. The retail environment robotic system according to claim 1, wherein the depth map processing component is configured to convert the depth map to a pixel image including a plurality of pixels, each pixel associated with a pixel location determined by a column position and row position of the corresponding LIDAR data and a pixel value determined by the object distance of the corresponding LIDAR data at the respective column positions and row positions of the corresponding LIDAR data.

3. The retail environment robotic system according to claim 2, wherein a polar to Cartesian conversion is used to convert the LIDAR data to the pixel image.

4. The retail environment robotic system according to claim 1, wherein the depth map processing component is configured to identify one or more shelf locations associated with the depth map by identifying a plurality of closest detected objects associated with the LIDAR data, the closest detected objects closest to one or both of the mobile base and LIDAR.

5. The retail environment robotic system according to claim 4, wherein the depth map processing component is configured to identify each shelf location by processing the LIDAR data using a sliding window to identify the closest detected objects within a plurality of localized regions.

6. The retail environment robotic system according to claim 4, wherein the depth map processing component is configured to generate a plot of the one or more identified shelf locations and process the plot to reduce noise by performing one or more of chain code processing, line joining processing and morphological processing.

7. The retail environment robotic system according to claim 1, wherein the depth map processing component is configured to initially identify one or more shelf locations associated with the depth map and subsequently identify regions of the depth map proximately above the identified shelf location associated with missing products.

8. The retail environment robotic system according to claim 1, wherein the depth map processing component is configured to initially identify one or more shelf locations associated with the depth map and subsequently identify regions of the depth map proximately below the identified shelf location associated with a shelf tag.

9. A retail environment robotic system comprising:

a mobile base including an operatively associated navigating component configured to determine a location of the mobile base within a retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more shelves of the product display fixtures, and a plurality of tags attached to the shelves of the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture shelf near the tag;

an image capture assembly operatively mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units;

a vertical scanning LIDAR (Light Detection and Ranging) unit operatively mounted to one of the mobile base and the image capture assembly, the LIDAR unit configured to scan in a vertical direction and generate LIDAR data including a vertical array of distance and angle values representative of a plurality of object distances and angles from the LIDAR unit to a plurality of objects detected along the vertical scan of the LIDAR unit; and

a master control unit including an operatively associated memory which stores instruction and a processor for executing the instructions, the control unit instructions configured to process the images acquired by the plurality of image capture devices at a sequence of locations during a movement of the mobile base in the retail environment, thereby extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the movement of the mobile base, and generating a profile of the retail environment indicating locations of one or more of the products, missing products and the tags throughout the retail environment based on the extracted product-related data, the LIDAR data, and the locations of the mobile base at the sequence of locations provided by the navigation component.

10. The retail environment robotic system according to claim 9, wherein the product-related data includes at least one of printed barcodes and text.

11. The retail environment robotic system according to claim 9, wherein the master control unit generates one or more of a packaging order, a product stocking order and a set of replacement tags based on the generated profile of the retail environment.

12. The retail environment robotic system according to claim 9, wherein the image capture devices acquire images and the LIDAR unit scans in a continuous movement mode of the mobile base or a stop and go movement mode of the mobile base.

13. The retail environment robotic system according to claim 9, further comprising:

a LIDAR data processing component operatively associated with the LIDAR unit and master control unit, the LIDAR data processing component configured to process a plurality of vertical arrays of object distances and angles to generate a depth map associated with the one or more product display units as the mobile base travels along an aisle, the depth map including a plurality of columns and a plurality of rows where each of the plurality of columns is associated with a single scan of the LIDAR unit at one of a plurality of locations along the aisle, and each of the plurality of rows is associated with one of a plurality of vertical distances.

14. The retail environment robotic system according to claim 13, further comprising:

a depth map processing component configured to identify regions of the depth map associated with regions of the one or more product display units which include one or more of the missing products, the products, the shelves, and the tags.

15. The retail environment robotic system according to claim 14, wherein the depth map processing component is configured to identify one or more shelf locations associated with the depth map by identifying a plurality of closest detected objects associated with the LIDAR data, the closest detected objects closest to one or both of the mobile base and LIDAR.

16. The retail environment robotic system according to claim 15, wherein the depth map processing component is configured to identify each shelf location by processing the LIDAR data using a sliding window to identify the closest detected objects within a plurality of localized regions.

17. The retail environment robotic system according to claim 15, wherein the depth map processing component is configured to generate a plot of the one or more identified shelf locations and process the plot to reduce noise by performing one or more of chain code processing, line joining processing and morphological processing.

18. The retail environment robotic system according to claim 14, wherein the depth map processing component is configured to initially identify one or more shelf locations associated with the depth map and subsequently identify regions of the depth map proximately above the identified shelf location associated with missing products.

19. A retail environment robotic system comprising:

a mobile base including an operatively associated navigating component configured to determine a location of the mobile base within a retail environment including a plurality of product display units arranged in a plurality of aisles, each product display unit including a plurality of product display fixtures, a plurality of products arranged on one or more shelves of the product display fixtures, and a plurality of tags attached to the shelves of the product display fixtures, each tag including product-related data including a barcode associated with a respective product proximately located on the product display fixture shelf near the tag;

an image capture assembly operatively mounted on the mobile base, the assembly including a plurality of image capture devices for acquiring images of the product display units;

a vertical scanning LIDAR (Light Detection and Ranging) unit operatively mounted to one of the mobile base and the image capture assembly, the LIDAR unit configured to scan in a vertical direction and generate LIDAR data including a vertical array of distance and angle values representative of a plurality of object distances and angles from the LIDAR unit to a plurality of objects detected along the vertical scan of the LIDAR unit;

a master control unit including an operatively associated memory which stores instruction and a processor for executing the instructions, the control unit instructions configured to process the images acquired by the plurality of image capture devices at a sequence of locations during a movement of the mobile base in the retail environment, thereby extracting the product-related data from the images of the product display units captured by the plurality of image capture devices during the movement of the mobile base, and generating a profile of the retail environment indicating locations of one or more of the products, missing products and the tags throughout the retail environment based on the extracted product-related data, the LIDAR data, and the locations of the mobile base at the sequence of locations provided by the navigation component;

a LIDAR data processing component operatively associated with the LIDAR unit and master control unit, the LIDAR data processing component configured to process the plurality of vertical arrays of object distances and angles to generate a depth map associated with the one or more product display units as the mobile base travels along an aisle, the depth map including a plurality of columns and a plurality of rows where each of the plurality of columns is associated with a single scan of the LIDAR unit at one of a plurality of locations along the aisle, and each of the plurality of rows is associated with one of a plurality of vertical distances; and

a depth map processing component configured to identify regions of the depth map associated with regions of the one or more product display units which include one or more of the missing products, the products, the shelves, and the tags.

20. The retail environment robotic system according to claim 19,

wherein the depth map processing component is configured to identify one or more shelf locations associated with the depth map by identifying a plurality of closest detected objects associated with the LIDAR data, the closest detected objects closest to one or both of the mobile base and LIDAR;

wherein the depth map processing component is configured to identify each shelf location by processing the LIDAR data using a sliding window to identify the closest detected objects within a plurality of localized regions; and

wherein the depth map processing component is configured to generate a plot of the one or more identified shelf locations and process the plot to reduce noise by performing one or more of chain code processing, line joining processing and morphological processing.

21. The retail environment robotic system according to claim 20, wherein the depth map processing component is configured to initially identify one or more shelf locations associated with the depth map and subsequently identify regions of the depth map proximately above the identified shelf location associated with missing products.