Estimating features of occluded objects

• US 9,336,602 B1
• Filed: 02/19/2013
• Issued: 05/10/2016
• Est. Priority Date: 02/19/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chang, Samuel Henry, Yao, Ning
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): Sosanya, Obafemi
• Application Number:
  US13/770,327
• Time in Patent Office: 1,176 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A number of images of an environment may be obtained over time. In some images, a portion of an object included in the environment may be occluded by another object in the environment. In these images, features of the occluded portion of the object may be absent. The absent features may be estimated based on features generated from previous images including the object where the occluded portion of the object was not occluded in the previous images. In some cases, the absent features may be estimated based on data associated with a reference object that corresponds to the occluded object with the reference object being included in a catalog of reference objects.

Claims

1. A system comprising:

one or more processors;

one or more cameras to obtain a plurality of images of an environment, the environment including a first object and a second object;

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

capturing a first image of the environment at a first time, wherein a first portion of the first object is occluded by at least a portion of the second object and a second portion of the first object is non-occluded;

generating first features of the first object based, at least in part, on the first image, wherein the first features include features of the second portion of the first object and features of the first portion of the first object are absent from the first features;

capturing a second image of the environment at a second time subsequent to the first time, wherein the second portion of the first object is occluded by the second object in the second image;

generating second features of the first object based, at least in part, on the second image, wherein the features of the second portion of the first object are absent from the second features; and

estimating the features of the second portion of the first object in the second image based, at least in part, on the features of the second portion of the first object included in the first features generated based on the first image.

2. The system of claim 1, wherein the first image, the second image,

or both are 3-Dimensional (3D) images of the environment.

3. The system of claim 2, wherein the acts further comprise capturing an additional image of the environment substantially at the first time, the additional image being captured by a red-green-blue (RGB) camera.

4. The system of claim 3, wherein the generating the first features of the first object is based, at least in part, on a color of the first object in the additional image, a texture of the first object in the additional image, or a combination thereof.

5. The system of claim 1, wherein a surface of the second object contacts a surface of the first object at the first portion of the first object at the first time.

6. The system of claim 1, wherein a time interval between the first time and the second time is based, at least in part, on a speed of the second object.

7. The system of claim 1, further comprising:

capturing a previous image before the first time, the previous image including the first object and the second portion of the first object is non-occluded in the previous image; and

generating previous features of the first object based, at least in part, on the previous image, wherein the previous features of the first object include features of the second portion of the first object.

8. The system of claim 7, wherein the estimating the features of the second portion of the first object is based, at least in part, on the features of the second portion of the first object included in the previous features.

9. A method comprising:

obtaining, by sensors of a computing device, data including a plurality of images of an environment;

generating, by the computing device, first features of a first object and second features of a second object included in a particular image of the plurality of images;

determining, by the computing device, that a particular portion of the first object is occluded by the second object; and

estimating, by the computing device, the features of the particular portion of the first object based, at least in part, on at least one of information included in a catalog of reference objects or data from at least one image including the first object that is captured previous to the particular image in which the particular portion of the first object was non-occluded.

10. The method of claim 9, wherein the generating the first features of the first object and the second features of the second object includes:

segmenting surfaces in the environment to identify continuous surfaces in the environment; and

determining 3-Dimensional (3D) points located on respective peripheries of the continuous surfaces.

11. The method of claim 9, wherein the second object is in motion in the environment during a time when the particular image is captured or the second object has remained substantially stationary in a threshold number of images captured previous to the particular image.

12. The method of claim 9, further comprising determining whether the catalog of reference objects includes a particular reference object corresponding to the first object.

13. The method of claim 12, wherein the determining whether the catalog of reference objects includes a particular reference object corresponding to the first object includes at least one of:

comparing contours of non-occluded portions of a periphery of the first object with respective contours of reference objects included in the catalog of reference objects;

comparing dimensions of the first object with respective dimensions of the reference objects included in the catalog of reference objects;

or

comparing a location of the first object with respective locations associated with the reference objects included in the catalog of reference objects.

14. The method of claim 9, further comprising obtaining information about reference objects in the catalog of reference objects from one or more images of the environment, input from a user of the computing device, or a combination thereof.

15. A method comprising:

obtaining, by a camera of a computing device, a particular image of a workspace included in a portion of an environment;

identifying, by the computing device, a first object in the workspace comprising a particular portion that is occluded by a second object in the workspace in the particular image; and

estimating, by the computing device, missing features of the first object based, at least in part, on at least one of information corresponding to the first object included in a catalog of reference objects or data from at least one image that includes the first object that is captured previous to the particular image in which the particular portion of the first object was non-occluded.

16. The method of claim 15, wherein the workspace includes a table, a desk, a chair, a laptop computer, a desktop computer, a tablet computer, a telephone, one or more hands of a user of the workspace, office supplies, paper products, furniture, or combinations thereof.

17. The method of claim 15, wherein the first object is occluded by a hand of a user in the workspace.

18. The method of claim 15, wherein the estimating the missing features of the first object is based on the information corresponding to the first object included in the catalog of reference objects when the second object is substantially stationary for a predetermined period of time.

19. The method of claim 15, wherein the estimating the missing features of the first object is based on the data from at the least one image captured previous to the particular image when the second object covers different portions of the first object within a particular period of time.

20. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

obtaining a plurality of images of an environment, wherein each of the plurality of images includes multiple objects;

identifying a first object in an image of the plurality of images having portions of the first object being occluded by a second object;

generating data corresponding to features of the first object;

identifying, based on the data corresponding to the features, a reference object in a catalog of reference objects that corresponds to the first object; and

estimating missing features of occluded portions of the first object based, at least in part, on respective features of the reference object corresponding to the missing features of the occluded portions of the first object.

21. The one or more computer-readable media of claim 20, wherein the features of the first object include a location of the first object, a dimension of the first object, a shape of the first object, a texture of the first object, a color of the first object, non-occluded contours of the first object, or a combination thereof.

22. The one or more computer-readable media of claim 20, wherein the acts further comprise identifying the reference object by comparing one or more features associated with the reference object with corresponding features of the first object.

23. The one or more computer-readable media of claim 20, wherein an additional portion of the first object is occluded by an additional object, and the acts further comprise estimating occluded features of the additional portion of the periphery of the first object based on one or more previously obtained images of the environment that include the first object.

24. The one or more computer-readable media of claim 20, wherein the portion of the first object that is occluded by the second object is blocked from a field of view of a camera obtaining the plurality of images of the environment.