Methods and systems for automatic fullness estimation of containers

• US 9,940,730 B2
• Filed: 11/18/2015
• Issued: 04/10/2018
• Est. Priority Date: 11/18/2015
• Status: Active Grant

• Current Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Original Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Inventors: Zhang, Yan, Williams, Jay, O'Connell, Kevin J., Taskiran, Cuneyt M.
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): RICE, ELISA M
• Application Number:
  US14/944,860
• Publication Number:
  US 20170140550A1
• Time in Patent Office: 874 Days
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume

Abstract

A method and apparatus for (a) receiving a three-dimensional (3D) point cloud from a depth sensor that is oriented towards an open end of a shipping container, the point cloud comprising a plurality of points that each have a respective depth value, (b) segmenting the received 3D point cloud among a plurality of grid elements, (c) calculating a respective loaded-container-portion grid-element volume for each grid element, (d) calculating a loaded-container-portion volume of the shipping container by aggregating the calculated respective loaded-container-portion grid-element volumes, (e) calculating an estimated fullness of the shipping container based on the loaded-container-portion volume and a capacity of the shipping container; and (f) outputting the calculated estimated fullness of the shipping container.

Claims

1. A method comprising:

receiving a three-dimensional (3D) point cloud from a depth sensor that is oriented towards an open end of a shipping container, the point cloud comprising a plurality of points that each have a respective depth value;

segmenting the received 3D point cloud among a plurality of grid elements;

calculating a respective loaded-container-portion grid-element volume for each grid element;

calculating a loaded-container-portion volume of the shipping container by aggregating the calculated respective loaded-container-portion grid-element volumes;

calculating an estimated fullness of the shipping container based on the loaded-container-portion volume and a capacity of the shipping container; and

outputting the calculated estimated fullness of the shipping container, wherein the plurality of grid elements collectively forms a two-dimensional (2D) grid image that corresponds to a plane that is parallel to the open end of the shipping container, each grid element having a respective grid-element area, the method further comprising;

determining a respective loaded-container-portion grid-element depth value for each grid element, wherein calculating the respective loaded-container-portion grid-element volume for each grid element is based on at least the respective grid-element area and the respective loaded-container-portion grid-element depth value for each respective grid element;

determining an unloaded-container-portion depth value for each grid element, wherein determining a respective loaded-container-portion grid-element depth value for each grid element is based at least in part on the difference between (i) a depth dimension of the shipping container and (ii) the determined unloaded-container-portion depth value for the corresponding grid element, wherein determining the unloaded-container-portion depth value for a given grid element comprises;

assigning a grid-element depth value to the given grid element based on the depth values of the points in the point cloud that correspond to the given grid element; and

determining the unloaded-container-portion depth value for the given grid element based at least in part on the difference between (i) the assigned grid-element depth value for the given grid element and (ii) an offset depth value corresponding to a depth between the depth sensor and a front plane of the shipping container.

2. The method of claim 1, wherein assigning the grid-element depth value for the given grid element based on the depth values of the points in the point cloud that correspond to the given grid element comprises assigning as the grid-element depth value for the given grid element a minimum value from among the depth values of the points in the point cloud that correspond to the given grid element.

3. The method of claim 1, wherein assigning the grid-element depth value for the given grid element based on the depth values of the points in the point cloud that correspond to the given grid element comprises assigning as the grid-element depth value for the given grid element an average value of the depth values of the points in the point cloud that correspond to the given grid element.

4. The method of claim 1, wherein the depth dimension of the shipping container is a grid-element-specific depth dimension that is based on a corresponding grid element in a reference empty-container point cloud.

5. The method of claim 4, wherein the reference empty-container point cloud reflects a back wall of the shipping container being a flat surface.

6. The method of claim 4, wherein the reference empty-container point cloud reflects a back wall of the shipping container being a curved surface.

7. The method of claim 1, further comprising cleaning up the 2D grid image prior to determining a respective loaded-container-portion grid-element depth value for each grid element.

8. The method of claim 1, wherein the depth sensor has an optical axis and an image plane, the method further comprising:

prior to segmenting the received point cloud among the plurality of grid elements, rotating the received 3D point cloud to align (i) the optical axis with a ground level and (ii) the image plane with an end plane of the shipping container.

9. The method of claim 8, wherein the rotating the point cloud is based on an offline calibration process using the ground level and the end plane as reference.

10. The method of claim 1, further comprising determining the capacity of the shipping container based at least in part on the received 3D point cloud.

11. The method of claim 1, further comprising:

receiving an optical image of the shipping container; and

determining the capacity of the shipping container based at least in part on the received optical image.

12. The method of claim 11, wherein determining the capacity of the shipping container based at least in part on the received optical image comprises:

determining at least one physical dimension of the shipping container from the received optical image; and

determining the capacity of the shipping container based on the at least one determined physical dimension.

13. The method of claim 11, wherein determining the capacity of the shipping container based at least in part on the received optical image comprises:

using optical character recognition (OCR) on the at least one received optical image to ascertain at least one identifier of the shipping container; and

using the at least one ascertained identifier of the shipping container to determine the capacity of the shipping container.

14. The method of claim 1, wherein each grid element has sides substantially equal to 5 millimeters (mm) in length.

15. The method of claim 1, wherein each grid element is substantially square in shape, and wherein a grid-element side length is an adjustable parameter.

16. A system comprising:

a depth sensor that is oriented towards an open end of a shipping container;

a communication interface;

a processor; and

data storage containing instructions executable by the processor for causing the system to carry out a set of functions, the set of functions including;

receiving a three-dimensional (3D) point cloud from the depth sensor, the point cloud comprising a plurality of points that each have a respective depth value;

segmenting the received 3D point cloud among a plurality of grid elements;

calculating a respective loaded-container-portion grid-element volume for each grid element;

calculating a loaded-container-portion volume of the shipping container by aggregating the calculated respective loaded-container-portion grid-element volumes;

calculating an estimated fullness of the shipping container based on the loaded-container-portion volume and a capacity of the shipping container; and

outputting the calculated estimated fullness of the shipping container, wherein the plurality of grid elements collectively forms a two-dimensional (2D) grid image that corresponds to a plane that is parallel to the open end of the shipping container, each grid element having a respective grid-element area, the set of functions further comprising;

determining a respective loaded-container-portion grid-element depth value for each grid element, wherein calculating the respective loaded-container-portion grid-element volume for each grid element is based on at least the respective grid-element area and the respective loaded-container-portion grid-element depth value for each respective grid element, the set of functions further comprising;

determining an unloaded-container-portion depth value for each grid element, wherein determining a respective loaded-container-portion grid-element depth value for each grid element is based at least in part on the difference between (i) a depth dimension of the shipping container and (ii) the determined unloaded-container-portion depth value for the corresponding grid element, wherein determining the unloaded-container-portion depth value for a given grid element comprises;

assigning a grid-element depth value to the given grid element based on the depth values of the points in the point cloud that correspond to the given grid element; and

determining the unloaded-container-portion depth value for the given grid element based at least in part on the difference between (i) the assigned grid-element depth value for the given grid element and (ii) an offset depth value corresponding to a depth between the depth sensor and a front plane of the shipping container.

17. The system of claim 16, wherein:

the plurality of grid elements collectively forms a two-dimensional (2D) grid image that corresponds to a plane that is parallel to the open end of the shipping container, each grid element having a respective grid-element area;

the set of functions further comprises determining a respective loaded-container-portion grid-element depth value for each grid element; and

calculating the respective loaded-container-portion grid-element volume for each grid element is based on at least the respective grid-element area and the respective loaded-container-portion grid-element depth value for each respective grid element.

18. A method comprising:

receiving a three-dimensional (3D) point cloud from a depth sensor that is oriented towards an open end of a shipping container, the point cloud comprising a plurality of points that each have a respective depth value, wherein the depth sensor has an optical axis and an image plane;

rotating the received 3D point cloud to align (i) the optical axis with a ground level and (ii) the image plane with an end plane of the shipping container;

segmenting the rotated 3D point cloud among a plurality of grid elements that collectively form a two-dimensional (2D) grid image that corresponds to a plane that is parallel to the open end of the shipping container, each grid element having a respective grid-element area;

determining a respective loaded-container-portion grid-element depth value for each grid element at least in part by comparing (i) the respective portion of the projected point cloud that is overlaid by the respective grid-element area of the respective grid element with (ii) a respective corresponding portion of a reference empty-container point cloud;

calculating a respective loaded-container-portion grid-element volume for each grid element based on at least the respective grid-element area and the respective loaded-container-portion grid-element depth value for each respective grid element;

calculating a loaded-container-portion volume of the shipping container by aggregating the calculated respective loaded-container-portion grid-element volumes;

determining a capacity of the shipping container based at least in part on the received depth data calculating an estimated fullness of the shipping container based on the loaded-container-portion volume and the determined capacity of the shipping container;

outputting the calculated estimated fullness of the shipping container; and

determining an unloaded-container-portion depth value for each grid element, wherein determining a respective loaded-container-portion grid-element depth value for each grid element is based at least in part on the difference between (i) a depth dimension of the shipping container and (ii) the determined unloaded-container-portion depth value for the corresponding grid element, wherein determining the unloaded-container-portion depth value for a given grid element comprises;

assigning a grid-element depth value to the given grid element based on the depth values of the points in the point cloud that correspond to the given grid element; and

determining the unloaded-container-portion depth value for the given grid element based at least in part on the difference between (i) the assigned grid-element depth value for the given grid element and (ii) an offset depth value corresponding to a depth between the depth sensor and a front plane of the shipping container.