Automatic visual fact extraction

• US 9,478,030 B1
• Filed: 03/19/2014
• Issued: 10/25/2016
• Est. Priority Date: 03/19/2014
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lecky, Ned
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US14/219,384
• Time in Patent Office: 951 Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

By evaluating readily available facts regarding an object, an item may be identified, or one or more characteristics of the item may be determined, and a destination for the item may be selected. An extraction module including a depth sensor may capture depth imaging data regarding the item, which may then be processed in order to estimate one or more dimensions of the item, and an appropriate container or storage area for the item may be selected. Additionally, the extraction module may further include a scale for determining a mass of the item, or digital cameras for capturing one or more images of the item. The images of the item may be analyzed in order to interpret any markings, labels or identifiers disposed on the item, and a destination for the item may be selected based on the mass, the analyzed images or the depth imaging data.

Claims

1. A visual fact extraction system comprising:

an extraction module including a depth sensor mounted to a frame, wherein the extraction module defines an imaging region adapted to receive an object therein, and wherein the depth sensor is configured to capture depth imaging data from the imaging region; and

a computing device in communication with the depth sensor,

wherein the computing device is configured to at least;

capture baseline depth imaging data from the imaging region using the depth sensor, wherein the baseline depth imaging data comprises information regarding distances between the depth sensor and the imaging region without an object therein;

receive an object at the imaging region;

capture depth imaging data from the imaging region using the depth sensor, wherein the depth imaging data comprises information regarding distances between the depth sensor and the imaging region with the object therein;

generate net depth imaging data based at least in part on the baseline depth imaging data and the depth imaging data, wherein the net depth imaging data comprises a plurality of pixels corresponding to the imaging region;

estimate a first dimension of the object based at least in part on the net depth imaging data;

determine a two-dimensional size of at least one of the plurality of pixels based at least in part on the first dimension;

identify a portion of the plurality of pixels corresponding to the object;

estimate a second dimension of the object based at least in part on the two-dimensional size and the portion of the pixels;

estimate a third dimension of the object based at least in part on the two-dimensional size and the portion of the pixels; and

select a container for the object based at least in part on the first dimension, the second dimension and the third dimension.

2. The visual fact extraction system of claim 1, wherein the first dimension is a maximum height of the object.

3. The visual fact extraction system of claim 1, wherein the visual fact extraction system further comprises:

at least one ingress conveyor configured to transport objects from an origin to the imaging region; and

at least one egress conveyor configured to transport objects from the imaging region to one of a plurality of destinations.

4. The visual fact extraction system of claim 3, wherein the computing device is further configured to at least:

cause the object to be transported from the origin to the imaging region by way of the at least one ingress conveyor;

select one of the plurality of destinations for the object based at least in part on the first dimension;

cause the object to be transported from the imaging region to the selected one of the plurality of destinations by way of the at least one egress conveyor; and

cause the object to be packed into the selected container at the selected one of the plurality of destinations.

20. The visual fact extraction system of claim 1, wherein the second dimension is a maximum length of the object, and

wherein the third dimension is a maximum width of the object.

5. A method for extracting facts comprising:

capturing a first set of depth imaging data of a defined region using a first depth sensor;

capturing a second set of depth imaging data of the defined region using the first depth sensor, wherein the second set of depth imaging data is captured with an object present within the defined region;

determining a net depth profile of the object based at least in part on the first set of depth imaging data and the second set of depth imaging data using at least one computer processor, wherein the net depth profile comprises a plurality of pixels corresponding to the defined region;

estimating a first dimension of the object based at least in part on the net depth profile;

determining a two-dimensional size of at least one of the plurality of pixels based at least in part on the first dimension;

identifying a portion of the plurality of pixels corresponding to the object;

estimating a second dimension of the object based at least in part on the two-dimensional size and the portion of the pixels;

estimating a third dimension of the object based at least in part on the two-dimensional size and the portion of the pixels;

storing an association of the net depth profile and the object in at least one data store; and

selecting a container for the object based at least in part on the first dimension, the second dimension and the third dimension.

6. The method of claim 5, further comprising:

estimating a bounding volume of the object based at least in part on the first dimension, the second dimension and the third dimension,

wherein the container is selected for the object based at least in part on the bounding volume.

7. The method of claim 5, wherein determining the net depth profile of the object based at least in part on the first set of depth imaging data and the second set of depth imaging data comprises:

defining an array having a plurality of cells based at least in part on a difference between the second set of depth imaging data and the first set of depth imaging data, wherein each of the cells corresponds to one of the plurality of pixels, and

wherein the net depth profile is determined based at least in part on the array.

8. The method of claim 7,

further comprising:

populating the plurality of cells with ranging data for at least some of the pixels of the defined region; and

estimating the first dimension of the object based at least in part on the ranging data for at least one of the pixels of the defined region.

9. The method of claim 7,

further comprising:

populating the plurality of cells with ranging data for at least some of the pixels of the defined region; and

estimating the two-dimensional size of the at least one of the plurality of pixels based at least in part on the ranging data.

10. The method of claim 8, further comprising:

determining a number of the pixels of the defined region corresponding to the object;

estimating the second dimension of the object based at least in part on the number of the pixels and the first dimension of the object; and

estimating the third dimension of the object based at least in part on the number of the pixels and the first dimension of the object.

11. The method of claim 8,

wherein the first dimension corresponds to a largest value of the ranging data for the plurality of the pixels of the defined region corresponding to the object.

12. The method of claim 5, wherein the first depth sensor comprises an infrared projector and an infrared sensor, and

wherein the capturing the first set of depth imaging data of the defined region using the first depth sensor comprises:

projecting a first plurality of infrared beams into the defined region using the infrared projector without the object present within the defined region;

capturing a second plurality of infrared beams reflected from the defined region using the infrared sensor; and

generating the first set of depth imaging data based at least in part on the second plurality of infrared beams using the at least one computer processor.

13. The method of claim 12, wherein the method further comprises:

projecting a third plurality of infrared beams into the defined region using the infrared projector with the object present within the defined region;

capturing a fourth plurality of infrared beams reflected from the defined region using the infrared sensor; and

generating the second set of depth imaging data based at least in part on the fourth plurality of infrared beams using the at least one computer processor.

14. The method of claim 5, further comprising:

determining a mass of the object using at least one scale, wherein a weighing surface of the at least one scale comprises the defined region, and wherein the computing device is in communication with the at least one scale,

wherein selecting the container for the object based at least in part on the net depth profile further comprises;

selecting the container for the object based at least in part on the first dimension, the second dimension, the third dimension and the mass.

15. The method of claim 5, further comprising:

causing placement of the object into the defined region by causing transport of the object from an origin to the defined region on a first conveyor;

causing transport of the object from the defined region to a destination on a second conveyor; and

causing the object to be deposited into the selected container at the destination.

21. The method of claim 5, wherein the first dimension is a maximum height of the object,

wherein the second dimension is a maximum length of the object, and

wherein the third dimension is a maximum width of the object.

16. A system comprising:

an electronic scale;

a depth sensor;

a plurality of imaging devices; and

a computing device in communication with the electronic scale, the depth sensor and at least one of the plurality of the imaging devices,

wherein the computing device is configured to at least;

capture at least one image of a visible surface of an object on a surface of the electronic scale using one or more of the imaging devices, wherein the visible surface comprises at least one identifier disposed thereon;

determine a mass of the object using the electronic scale;

capture a depth image of the object using the depth sensor, wherein the depth image comprises a plurality of pixels representing distances between the depth sensor and the electronic scale with the object thereon;

analyze the at least one image to interpret the at least one identifier;

estimate a maximum height of the object based at least in part on the depth image;

determine a two-dimensional area of at least one of the plurality of pixels based at least in part on the maximum height of the object;

identify a portion of the depth image corresponding to the object;

estimate a maximum length and a maximum width of the object based at least in part on the two-dimensional area and the portion of the depth image corresponding to the object; and

select a container for the object based at least in part on the mass of the object, the at least one identifier, the maximum height, the maximum length and the maximum width.

17. The system of claim 16, further comprising at least one conveyor in communication with the computing device,

wherein the computing device is further configured to:

select a destination associated with the container; and

cause a delivery of the object to the selected destination using the at least one conveyor.

18. The system of claim 16, wherein the identifier is one of a one-dimensional bar code, a two-dimensional bar code, a bokode, a character, a number, a symbol or a color.

19. The system of claim 16, wherein at least one of the plurality of imaging devices is a line scan camera.