Enhancing resolution of single images

• US 9,058,674 B1
• Filed: 03/15/2013
• Issued: 06/16/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Noble, Isaac Scott, Bell, Matthew Paul
• Primary Examiner(s): Do, Anh
• Application Number:
  US13/837,079
• Time in Patent Office: 823 Days
• Field of Search: 382/299, 382/218, 382/219, 382/254, 382/255, 382/276, 382/118, 345/592, 345/1.3, 345/628, 340/12.25, 340/5.61, 340/5.64, 250/221, 250/208.1, 455/411, 455/418, 455/420, 455/414.1, 455/426.1, 455/456.1, 455/456.3, 704/2, 704/9, 704/10, 704/257, 715/202, 715/205, 715/206, 715/208.1, 715/231
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A computing device can capture an image using a camera of the device. The computing device can also determine or receive an image property (i.e., information related to the image). The device can transmit the image and the information related to the image to a network service. The network service can communicate with an image patch library which includes a plurality of image patches. The network service can select a subset of image patches from the library based on the information related to the image. The network service can compare image portions of the image to the image patches in the subset to determine one or more image patches that match one or more image portions. In some embodiments, the one or more image patches can have corresponding higher resolution versions. The network service can modify image portions of the image using available higher resolution versions of matching image patches.

Claims

1. A computer-implemented method, performed by at least one processor, for image enhancement at a network service, comprising:

receiving an image and geolocation information indicating where the image was acquired;

identifying, from a library including a plurality of library image portions, a subset of the plurality of library image portions that are associated with the geolocation information;

for at least one source image portion in a plurality of source image portions of the image;

determining at least one library image portion, from the subset, that substantially matches the at least one source image portion, the at least one library image portion being associated with at least one higher resolution version; and

modifying the at least one source image portion using the at least one higher resolution version associated with the at least one library image portion; and

generating an enhanced version of the image based, at least in part, on the at least one source image portion being modified using the at least one higher resolution version.

2. The computer-implemented method of claim 1, further comprising:

receiving information indicating a subject matter associated with the image, wherein the subset of the plurality of library image portions is identified based, at least in part, on the subject matter.

3. The computer-implemented method of claim 1, wherein the modifying of the at least one source image portion using the higher resolution version further comprises at least one of replacing the at least one source image portion with the higher resolution version or merging the at least one source image portion with the higher resolution version.

4. The computer-implemented method of claim 1, wherein at least some of the plurality of source image portions are included in the library and are utilized as library image portions, and wherein the library is hosted on one or more networked servers configured to communicate with the network service.

5. The computer-implemented method of claim 1, wherein the determining that the at least one library image portion substantially matches the at least one source image portion further comprises comparing a set of pixels of the at least one library image portion to a set of pixels of the at least one source image portion, wherein the comparing utilizes at least one of line detection, edge detection, feature detection, image color comparison, image texture comparison, image shape comparison, or image distance measurement.

6. A computer-implemented method, performed by at least one processor, comprising:

receiving an image and information related to the image;

selecting, from a plurality of library image portions stored at a networked image portion library, a subset of library image portions that are associated with the information;

determining, from the subset, at least one library image portion that corresponds to at least one source image portion of the image;

identifying a higher resolution version of each of the at least one library image portion; and

modifying the at least one source image portion of the image using the higher resolution version to cause the at least one source image portion to have a higher resolution than before the modifying.

7. The computer-implemented method of claim 6, further comprising:

generating a down-sampled version for the at least one source image portion;

transmitting the down-sampled version for the at least one source image portion to the networked image portion library; and

transmitting the at least one source image portion to the networked image portion library, wherein the subset of library image portions further includes the down-sampled version and the at least one source image portion, and wherein the at least one source image portion corresponds to a higher resolution version of the down-sampled version.

8. The computer-implemented method of claim 6, wherein the information related to the image indicates at least one of a geolocation of a computing device at a time when the image was acquired using a camera of the computing device, a direction at which the camera was facing at the time when the image was acquired, a temporal aspect of when the image was acquired, a subject matter associated with the image, or a description associated with the image.

9. The computer-implemented method of claim 6, further comprising:

identifying a face in the image, wherein the subset of library image portions is selected based, at least in part, on the face identified in the image.

10. The computer-implemented method of claim 9, further comprising:

receiving, from a source configured to communicate with the network service, one or more library image portions associated with the face; and

modifying a resolution of at least a portion of the face in the image using, at least in part, the one or more library image portions associated with the face.

11. The computer-implemented method of claim 10, wherein the source configured to communicate with the network service is external to the network service, the source comprising at least one of a social networking service, a networked image storage service, a computing device used to acquire the image, or a computing device associated with a user whose face is identified in the image.

12. The computer-implemented method of claim 6, further comprising:

determining that none of the library image portions from the subset corresponds to the at least one image source portion within an allowable threshold; and

performing at least one of lowering the allowable threshold or increasing the subset to include additional library image portions.

13. The computer-implemented method of claim 6, wherein the modifying of the at least one source image portion further comprises at least one of replacing the at least one source image portion with the higher resolution version or merging the at least one source image portion with the higher resolution version.

14. A system comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the system to;

receive an image and information related to the image;

select, from a plurality of library image portions stored at a networked image portion library, a subset of library image portions that are associated with the information;

determine, from the subset, at least one library image portion that corresponds to at least one source image portion of the image;

identify a higher resolution version of each of the at least one library image portion; and

modify the at least one source image portion of the image using the higher resolution version to cause the at least one source image portion to have a higher resolution than before the modifying.

15. The system of claim 14, further comprising:

a computing device including at least one of a geolocation sensor configured to determine geolocation information indicating where the image was acquired, a camera, an orientation sensor configured to determine a direction at which the camera was pointing when the image was acquired by the camera, or a system clock configured to determine a temporal aspect indicating when the image was acquired, and wherein the information related to the image includes at least one of the geolocation information, the direction at which the camera was pointing, or the temporal aspect.

16. The system of claim 14, further comprising:

a computing device including a user interface configured to receive data inputted by a user of the computing device, wherein the information related to the image includes the data inputted by the user, and wherein the data indicates at least one of a subject matter associated with the image or a description associated with the image.

17. The system of claim 14, wherein the modifying of the at least one source image portion further comprises at least one of replacing the at least one source image portion with the higher resolution version or merging the at least one source image portion with the higher resolution version.

18. A non-transitory computer-readable storage medium including instructions, the instructions when executed by a processor of a computing system causing the computing system to:

receive an image and information related to the image;

select, from a plurality of library image portions stored at a networked image portion library, a subset of library image portions that are associated with the information;

determine, from the subset, at least one library image portion that corresponds to at least one source image portion of the image;

identify a higher resolution version of each of the at least one library image portion; and

modify the at least one source image portion of the image using the higher resolution version to cause the at least one source image portion to have a higher resolution than before the modifying.

19. The non-transitory computer-readable storage medium of claim 18, wherein the instructions cause the computing system to further:

generate a down-sampled version for the at least one source image portion;

transmit the down-sampled version for the at least one source image portion to the networked image portion library; and

transmit the at least one source image portion to the networked image portion library, wherein the subset of library image portions further includes the down-sampled version and the at least one source image portion, and wherein the at least one source image portion corresponds to a higher resolution version of the down-sampled version.

20. The non-transitory computer-readable storage medium of claim 18, wherein the information related to the image indicates at least one of a geolocation of a computing device at a time when the image was acquired using a camera of the computing device, a direction at which the camera was facing at the time when the image was acquired, a temporal aspect of when the image was acquired, a subject matter associated with the image, or a description associated with the image.

21. The non-transitory computer-readable storage medium of claim 18, wherein the instructions cause the computing system to further identify a face in the image, and wherein the subset of library image portions is selected based, at least in part, on the face identified in the image.

22. The non-transitory computer-readable storage medium of claim 21, wherein the instructions cause the computing system to further:

receive one or more library image portions associated with the face; and

modify a resolution of at least a portion of the face in the image using, at least in part, the one or more library image portions associated with the face.

23. The non-transitory computer-readable storage medium of claim 22, wherein the instructions cause the computing system to further:

determine that none of the library image portions from the subset corresponds to the at least one source image portion within an allowable threshold; and

perform at least one of lowering the allowable threshold or increasing the subset to include additional library image portions.

24. The non-transitory computer-readable storage medium of claim 18, wherein the modifying of the at least one source image portion further comprises at least one of replacing the at least one source image portion with the higher resolution version or merging the at least one source image portion with the higher resolution version.