Counting inventory items using image analysis and depth information

• US 9,996,818 B1
• Filed: 04/21/2016
• Issued: 06/12/2018
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ren, Xiaofeng, Manyam, Ohil Krishnamurthy, Misra, Avishkar, Bo, Liefeng, Raghavan, Sudarshan Narasimha, Towers, Christopher Robert, Gopal, Gopi Prashanth, Asmi, Yasser Baseer
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US15/135,384
• Time in Patent Office: 782 Days
• Field of Search: 382103, 382236, 348169-172, 348352
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Described is a system for counting stacked items using image analysis. In one implementation, an image of an inventory location with stacked items is obtained and processed to determine the number of items stacked at the inventory location. In some instances, the item closest to the camera that obtains the image may be the only item viewable in the image. Using image analysis, such as depth mapping or Histogram of Oriented Gradients (HOG) algorithms, the distance of the item from the camera and the shelf of the inventory location can be determined. Using this information, and known dimension information for the item, a count of the number of items stacked at an inventory location may be determined.

Claims

1. A computing system, comprising:

a processor; and

a memory coupled to the processor and storing program instructions that when executed by the processor causes the processor to at least;

receive from a first camera a first image of an inventory location, wherein the first image includes a representation of a plurality of inventory items located at the inventory location;

determine from an inventory location data store, an item type corresponding to the inventory location;

select a plurality of histogram of oriented gradients (“

HOG”

) models corresponding to the item type;

process the first image to generate a plurality of feature vectors, each feature vector representative of at least a portion of an object of an inventory item of the plurality of inventory items represented in the first image;

compare each of the plurality of feature vectors with each of the plurality of HOG models;

determine that a first feature vector representative of a first object of a first inventory item is substantially similar to at least one of the plurality of HOG models;

determine position information representative of a position of the first object represented by the feature vector;

compare the position information with an expected position of the first object, wherein the expected position is on a top of the first inventory item;

determine that the position information of the first object represented by the feature vector corresponds with the expected position of the feature vector; and

increment an inventory count.

2. The computing system of claim 1, wherein the program instructions, that when executed by the processor to cause the processor to determine an item count, further include instructions that when executed by the processor cause the processor to at least:

not increment the item count if it is determined that the position information does not correspond with the expected position of the first object represented by the feature vector.

3. The computing system of claim 1, wherein:

the program instruction that when executed by the processor further cause the processor to determine, based on the position information, a height of the first object from a shelf of the inventory location; and

the program instructions that when executed by the processor to cause the processor to compare the position information with the expected position, further include program instructions that cause the processor to at least;

determine a plurality of heights of the first object; and

determine that the position information of the first object represented by the feature vector corresponds with the expected position if it is determined that the plurality of heights of the first object are substantially similar.

4. A computer-implemented method for counting items, comprising:

under control of one or more computing systems configured with executable instructions,

receiving from a camera an image of an inventory location;

determining a first feature from the image, wherein the first feature is potentially representative of an item positioned at the inventory location, and wherein the first feature is represented by a plurality of pixels in the image;

determining a plurality of distances, each distance corresponding to a distance between a pixel of the plurality of pixels and the camera;

determining that the first feature corresponds with a model feature stored in an item information data store for a type of item located at the inventory location;

determining that each of the plurality of distances are substantially similar; and

incrementing an item count for the inventory location based at least in part on the determination that the first feature corresponds with the model feature and the determination that each of the plurality of distances are substantially similar.

5. The computer-implemented method of claim 4, further comprising:

comparing the first feature with a histogram of oriented gradients (“

HOG”

) model to determine that the first feature potentially represents an item of an item type associated with the inventory location.

6. The computer-implemented method of claim 5, further comprising:

generating, using a HOG algorithm, a feature vector representative of the first feature.

7. The computer-implemented method of claim 5, wherein the HOG model includes a model feature vector representative of a feature of an item of the item type.

8. The computer-implemented method of claim 4, further comprising:

receiving depth information identifying distances between the camera and each pixel of the image, wherein the depth information is determined based at least in part on a comparison of a plurality of images obtained by the camera.

9. The computer-implemented method of claim 4, further comprising:

determining a second feature from the image, wherein the second feature is potentially representative of a second item positioned at the inventory location, and wherein the second feature is represented by a second plurality of pixels in the image;

determining a second plurality of distances, each distance of the second plurality of distances corresponding to a distance between a pixel of the second plurality of pixels and the camera; and

determining that the second feature does not correspond with a second model feature stored in an item information data store for the type of item located at the inventory location.

10. The computer-implemented method of claim 4, wherein the first feature is at least one of a color, a size, a shape, a pattern, a letter, a label, a logo, a texture, a gradient, a reflectivity, an edge of the item, a character, or a symbol.

11. The computer-implemented method of claim 4, wherein the plurality of distances identify a position of the first feature with respect to the camera.

12. The computer-implemented method of claim 11, wherein the position of the first feature is substantially similar to an expected position of the first feature with respect to the camera.

13. The computer-implemented method of claim 4, wherein the image includes a plurality of features, each feature of the plurality of features potentially representative of an item located at the inventory location.

14. The computer-implemented method of claim 4, wherein the item count identifies a number of items located at the inventory location.

15. A computing system, comprising:

a processor; and

a memory coupled to the processor and storing program instructions that when executed by the processor causes the processor to at least;

receive from a camera an image of an item located at an inventory location,

receive depth information corresponding to a position of the item represented in the image with respect to the camera;

determine a feature vector representative of a feature of the item represented in the image;

compare the feature vector with a model feature vector corresponding to an item type associated with the inventory location;

determine a position of the feature of the item;

compare the position of the feature with an expected position; and

determine that the item corresponds to the item type based at least in part on the comparison of the feature vector with the model feature vector and the comparison of the position of the feature with the expected position.

16. The computing system of claim 15, wherein:

the feature is visible from at least one surface of the item; and

the expected position corresponds to a position of a surface upon which the feature is expected to be.

17. The computing system of claim 16, wherein the expected position is at least one of an alignment of the at least one surface or a distance of the at least one surface from the camera.

18. The computing system of claim 15, wherein the model feature vector is a histogram of oriented gradients (“

HOG”

) model.

19. The computing system of claim 18, wherein the HOG model is generated from an image of the item at a known distance and position with respect to a second camera.

20. The computing system of claim 15, wherein the image includes a representation of a plurality of items.

21. A method, comprising:

determining an item type of an inventory item associated with an inventory location;

segmenting an image of the inventory location to produce a plurality of image segments;

processing, using at least one computing system, a first image segment of the plurality of image segments to determine a feature from the first image segment, wherein the first image segment is less than the entire image;

comparing, using the at least one computing system, the feature with a model feature corresponding to the item type;

determining, using at least one computing system, that the image segment corresponds with the item type based at least in part on the comparison; and

in response to a determination that the image segment corresponds with the item type, perform an action.

22. The method of claim 21, wherein the action includes incrementing an item count for the inventory location.

23. The method of claim 21, further comprising:

determining position information representative of a position of the feature;

determining that the position information corresponds with an expected position for the inventory location; and

in response to determining that the position information corresponds, incrementing an item count for the inventory location.

24. The method of claim 21, wherein segmenting includes:

segmenting the image into at least three segments, the first segment corresponding to a front of the inventory location, a second segment corresponding to a middle of the inventory location, and a third segment corresponding to a back of the inventory location.