INTELLIGENT AUTO-CROPPING OF IMAGES

• US 20180174299A1
• Filed: 12/20/2016
• Published: 06/21/2018
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Agrawal, Amit Kumar, Davidson, Alexander Adrian Hugh, Ramu, Prakash
• Granted Patent: US 10,325,372 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/356 having separate monoscopic and stereoscopic modes

Abstract

Techniques for providing an accurate auto-crop feature for images captured by an image capture device may be described herein. For example, one or more image masks for a color image captured by an image capture device may be received by a computer system. Metadata about the color image that identifies portions of the image as foreground and the color image itself may also be received by the computer system. Further, a representation of a user and a floor region associated with a user may be extracted from the color image using the one or more image masks and the metadata. A first area of the color image may be cropped with respect to the extracted representation of the user and the floor region associated with the user to generate a second area of the color image. In embodiments, a third area of the color image may be obscured based on the received metadata.

Claims

1. A computer-implemented method, comprising:

receiving, by a computer system and from an image capture device, a first image mask of an image of a user, the image, and metadata about the image, the metadata indicating a plurality of unique values for one or more portions included in the image, the image capture device configured to capture a depth image of the user and a color mask image of the user, the depth image of the user including a three-dimensional (3D) representation of the user that is used by the image capture device to identify a depth of the user and a foreground location of the user with respect to a background of the image captured by the image capture device, the color mask image comprising a red, green, and blue (RGB) image of the user, the first image mask comprising the depth image and the color mask image;

extracting, by the computer system, a representation of the user and a representation of a floor region associated with the representation of the user from the image based at least in part on the first image mask and the metadata, the representation of the user comprised of a first subset of portions of the one or more portions included in the image and the representation of the floor region comprised of a second subset of portions of the one or more portions included in the image;

removing, by the computer system, a first area of the image with respect to the extracted representation of the user in the image based at least in part on the first image mask and the metadata thereby generating a second area of the image;

combining, by the computer system, the extracted representation of the user with the floor region of the image with respect to the second area of the image based at least in part on the first image mask and the metadata; and

displaying, by the computer system, a revised image of the user via a user interface of the computer system, the revised image of the user comprising the combination of the extracted representation of the user with the floor region of the image contained within the second area of the image.

2. The computer-implemented method of claim 1, wherein one or more of the plurality of unique values indicated by the metadata identify a pixel in the image as being in the background of the image.

3. The computer-implemented method of claim 1, further comprising obscuring a portion of the one or more portions of the image based at least in part on the metadata.

4. The computer-implemented method of claim 1, further comprising identifying an item included in the image based at least in part on an item recognition algorithm and an item catalog maintained by an electronic marketplace.

5. The computer-implemented method of claim 4, further comprising displaying, via the user interface of the computer system, an interactable object that is configured to enable ordering of the item included in the image.

6. The computer-implemented method of claim 1, further comprising updating the revised image of the user with a particular portion of another revised image of the user maintained by the computer system.

7. A computer-implemented method, comprising:

receiving, by a computer system and from an image capture device, a first image mask that comprises a two-dimensional (2D) representation of a user in an image captured by the image capture device and first metadata that identifies a first subset of regions in the image as being in a foreground of the image;

receiving, by the computer system and from the image capture device, a second image mask that comprises a representation of a floor region associated with the user in the image captured by the image capture device and second metadata that identifies a second subset of regions in the image as the foreground of the image;

receiving, by the computer system and from the image capture device, a color image of the user;

extracting, by the computer system, the representation of the user and the floor region associated with the user, from the color image of the user, based at least in part on the first image mask, the second image mask, the first metadata, and the second metadata;

cropping, by the computer system, a first area of the color image of the user with respect to the extracted representation of the user and the floor region associated with the user based at least in part on the first image mask and the second image mask thereby generating a second area of the color image; and

obscuring, by the computer system, a third area of the cropped color image based at least in part on the first metadata and the second metadata thereby generating a revised color image of the user that comprises a combination of the extracted representation of the user and the floor region associated with the user.

8. The computer-implemented method of claim 7, further comprising maintaining a plurality of cropped color images of the user.

9. The computer-implemented method of claim 7, further comprising identifying a plurality of items included in the image of the user based at least in part on an item recognition algorithm and an item catalog.

10. The computer-implemented method of claim 9, further comprising generating one or more item listing web pages for offering the identified plurality of items included in the image.

11. The computer-implemented method of claim 7, wherein receiving the color image of the user includes receiving third metadata that identifies a third subset of regions in the image as a background of the image.

12. The computer-implemented method of claim 11, wherein the first metadata, the second metadata, and the third metadata further identify a respective depth measurement for each pixel within the image captured by the image capture device.

13. The computer-implemented method of claim 7, wherein the image capture device comprises a depth sensor for capturing a three-dimensional (3D) image of the user and is further configured to convert the 3D image of the user to the 2D representation of the user using the color image of the user.

14. A computer system, comprising:

memory that stores computer-executable instructions;

a first sensor configured to capture a three-dimensional (3D) image of an object;

a second sensor configured to capture a color image of the object; and

at least one processor configured to access the memory and execute the compute-executable instructions to collectively at least;

obtain a first image mask that comprises a two-dimensional (2D) representation of a user in an image captured by the first sensor and first metadata that identifies a first subset of regions in the image as being in a foreground of the image based at least in part on a 3D image of the image captured by the first sensor;

obtain a second image mask that comprises a representation of a floor region associated with the user in the image captured by the first sensor and second metadata that identifies a second subset of regions in the image as the foreground of the image;

obtain the color image of the user from the second sensor;

extract the representation of the user and the floor region associated with the user, from the color image of the user, based at least in part on the first image mask, the second image mask, the first metadata, and the second metadata; and

remove a first area of the color image of the user with respect to the extracted representation of the user and the floor region associated with the user based at least in part on the first image mask and the second image mask thereby generating a second area of the color image.

15. The computer system of claim 14, wherein the at least one processor is further configured to display a revised image of the user that comprises a combination of the extracted representation of the user and the floor region associated with the user within the second area of the color image.

16. The computer system of claim 14, wherein the at least one processor is further configured to obscure a third area of the color image based at least in part on the first metadata and the second metadata.

17. The computer system of claim 14, wherein obtaining the first image mask includes converting the 3D image of the user to the 2D image of the user.

18. The computer system of claim 14, wherein the at least one processor is further configured to identify one or more objects in the image based at least in part on an item recognition algorithm.

19. The computer system of claim 18, wherein the at least one processor is further configured to transmit instructions to the user for removing the identified one or more objects from the first subset of regions in the image.

20. The computer system of claim 18, wherein the at least one processor is further configured to transmit instructions to the user for capturing another image of the user in response to an indication that the identified one or more objects have been removed from the first subset of regions in the image.