Cloud based content aware fill for images

• US 9,311,735 B1
• Filed: 11/21/2014
• Issued: 04/12/2016
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Amirghodsi, Sohrab, Darabi, Aliakbar, Schectman, Elya
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US14/550,135
• Time in Patent Office: 508 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 19/93 Run-length coding

Abstract

A region of an image on which a content aware fill operation is to be performed is identified by a device, the content aware fill operation replacing pixel values in the region based on pixel values from one or more other regions. The device generates a reduced-size version of the image by down sampling the image, and sends the reduced-size version of the image to a remote service. The remote service generates a mapping indicating which pixels in the reduced-size version of the image are to be used as the values of which other pixels in the region of the reduced-size version of the image. The mapping is compressed and returned to the device. The device decompresses the mapping and up samples the mapping. An output image is generated based on the originally obtained image, the up sampled mapping, and an indication of the region.

Claims

1. A method implemented in a computing device, the method comprising:

obtaining an image;

obtaining an indication of a first one or more regions of the image to be filled based on values of pixels in a second one or more regions of the obtained image;

down sampling the obtained image to generate a reduced-size version of the obtained image;

sending the reduced-size version of the obtained image to a remote service, the remote service being implemented in one or more devices separate from the computing device;

receiving from the remote service a map identifying, for each pixel in the first one or more regions, one or more pixels in the second one or more regions the values of which are to be used to generate the value for the pixel; and

generating an output image, the generating including using the map to determine values of pixels in the first one or more regions.

2. The method as recited in claim 1, the receiving comprising receiving the map without receiving a version of the image.

3. The method as recited in claim 1, the received map comprising a compressed map, and the method further comprising generating a decompressed map by decompressing the received map, and the generating the output image including using the decompressed map to determine the values of pixels in the first one or more regions.

4. The method as recited in claim 3, the received map including a partially compressed map and a table, the decompressing the received map comprising, for each pixel in the first one or more regions of the image:

obtaining, from the partially compressed map, an identifier of an entry in the table;

obtaining, from the entry in the table, an offset for the pixel;

applying the offset to a coordinate of the pixel to generate an offset coordinate; and

using the offset coordinate as the identifier of the pixel in the decompressed map.

5. The method as recited in claim 1, further comprising up sampling the map before generating the output image.

6. The method as recited in claim 1, further comprising:

receiving an additional version of the image that has been filtered or converted; and

the generating the output image comprising generating the output image based on the additional version of the image rather than the obtained image.

7. The method as recited in claim 6, the generating the output image comprising using the map to determine values of pixels in the first one or more regions of the additional version of the image.

8. The method as recited in claim 1, the generating comprising generating the output image on a stripe by stripe basis, each of multiple stripes comprising a block of the obtained image that is less the entire obtained image.

9. The method as recited in claim 8, further comprising, for each stripe, up sampling a portion of the map corresponding to a stripe of the obtained image before generating a stripe of the output image.

10. A method implemented in a service that includes one or more computing devices, the method comprising:

receiving, from a computing device separate from the service, a reduced-size version of an image;

receiving, from the computing device, an indication of a first one or more regions of the reduced-size version of the image to be filled based on values of pixels in a second one or more regions of the reduced-size version of the image;

determining, for each pixel in the first one or more regions, one or more pixels in the second one or more regions the values of which are to be used to generate the value for the pixel;

generating a map identifying, for each pixel in the first one or more regions, one or more pixels in the second one or more regions the values of which are to be used to generate the value for the pixel; and

returning the map to the computing device without returning a version of the image to the computing device.

11. The method as recited in claim 10, further comprising compressing the map, and the returning comprising returning the compressed map to the computing device.

12. The method as recited in claim 11, the compressing the map comprising:

generating a partially compressed map by, for each pixel in the first one or more regions of the reduced-size version of the image, including in the partially compressed map an identifier of an entry in a table, the entry in the table including an offset value that, when applied to a coordinate of the pixel, results in a coordinate of a particular pixel in the second region that is to be used to generate the value for the pixel, the entry including a gain value and a bias value for the particular pixel in the second region; and

generating the compressed map by applying run-length encoding to the partially compressed map and the table.

13. The method as recited in claim 12, further comprising generating a new entry in the table in response to no entry in the table having constraints that satisfy the particular pixel in the second region.

14. A computing device comprising:

an input module configured to obtain an image and an indication of a first one or more regions of the image to be filled based on values of pixels in a second one or more regions of the obtained image;

a content aware fill module configured to down sample the obtained image to generate a down sampled version of the obtained image;

a communication module configured to send the down sampled version of the image obtained image to a remote service, the remote service being implemented in one or more devices separate from the computing device, and further configured to receive from the remote service a map identifying, for each pixel in the first one or more regions, one or more pixels in the second one or more regions the values of which are to be used to generate the value for the pixel; and

the content aware fill module being further configured to generate an output image by using the map to determine values of pixels in the first one or more regions.

15. The computing device as recited in claim 14, the communication module being further configured to receive the map without receiving a version of the image from the remote service.

16. The computing device as recited in claim 14, the map comprising a compressed nearest-neighbor field (NNF) map, the computing device further comprising a map decompression module configured to generate a decompressed map by decompressing the received map, and the content aware fill being further configured to generate the output image by using the decompressed map.

17. The computing device as recited in claim 14, the content aware fill module being further configured to generate the output image on a stripe by stripe basis, each of multiple stripes comprising a block of the obtained image that is less the entire obtained image.

18. The computing device as recited in claim 17, the content aware fill module being further configured to, for each stripe, up sample a portion of the map corresponding to the obtained image before generating a stripe of the output image.

19. The computing device as recited in claim 14, the communication module being further configured to receive an additional version of the image that has been filtered or converted, and the content aware fill module being further configured to generate the output image based on the additional version of the image rather than the obtained image.

20. The computing device as recited in claim 14, the content aware fill module being further configured to:

receive an additional version of the image that has been filtered or converted by an additional remote service; and

generate the output image based on the additional version of the image rather than the obtained image by using the map to determine values of pixels in the first one or more regions of the additional version of the image.