Image cache for replacing portions of images

• US 10,217,222 B2
• Filed: 10/24/2016
• Issued: 02/26/2019
• Est. Priority Date: 10/28/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: James, Gavin, Lewis, Justin
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US15/332,923
• Publication Number:
  US 20170103538A1
• Time in Patent Office: 855 Days
• Field of Search: 382115, 382284, 382117, 382118, 382128, 382170, 382174, 382190, 382209, 382218, 382224, 382278, 340 581, 340 582, 340 593
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

Implementations relate to an image cache used for replacing portions of images. In some implementations, a method includes examining a received image and detecting a subject depicted in a subject portion of the received image. The method determines a likelihood that the subject portion will be used as a replacement to be inserted in place of one or more corresponding subject portions in future images examined at a later time. The subject portion, or a reference thereto, is stored in an image cache in response to the determined likelihood being greater than a predetermined threshold. The image cache is available to be searched for the subject portion and other portions of images to replace corresponding portions of future images.

Claims

1. A computer-implemented method comprising:

identifying a subject face of an image, wherein the subject face depicts a subject;

comparing the subject face of the image with one or more cached faces of one or more cached images stored in an image cache;

selecting a particular cached face of the one or more cached faces to replace the subject face, wherein the selecting comprises finding the particular cached face from the one or more cached faces in the image cache based on a number of transformations to be applied to fit the particular cached face to the image being lower than a respective number of transformations to be applied to fit other faces of the one or more cached faces to the image; and

modifying the image by stitching the particular cached face in the image to replace the subject face.

2. The computer-implemented method of claim 1,

wherein comparing the subject face of the image with the one or more cached faces includes assigning respective scores to one or more visual image characteristics of the one or more cached faces based on comparisons of the one or more visual characteristics of the one or more cached faces with one or more corresponding visual image characteristics of the subject face, wherein the one or more visual characteristics include one or more of lighting characteristics or color characteristics, and

wherein selecting the particular cached face is further based on the respective scores.

3. The computer-implemented method of claim 1, wherein selecting the particular cached face comprises:

based on the comparing, determining a candidate list of candidate cached faces;

assigning a score to each candidate cached face in the candidate list of candidate cached faces; and

identifying the particular cached face based on the score.

4. The computer-implemented method of claim 1, further comprising assigning a respective score to one or more image characteristics of the subject face of the image and one or more image characteristics of the one or more cached faces stored in the image cache,

wherein the selecting the particular cached face is in response to determining that the particular cached face has at least one image characteristic associated with a better score than a score associated with a corresponding image characteristic of the subject face of the image.

5. The computer-implemented method of claim 1 further comprising:

determining whether the subject face qualifies to be stored in the image cache based on a comparison of the subject face to the one or more cached faces; and

storing the subject face in the image cache if the subject face qualifies to be stored in the image cache.

6. The computer-implemented method of claim 1, further comprising:

determining a candidate list of one or more candidate cached faces from the image cache, wherein determining the candidate list includes determining that the one or more candidate cached faces have at least one similar characteristic to the subject face, wherein selecting the particular cached face includes selecting the particular cached face from the one or more candidate cached faces of the candidate list.

7. The computer-implemented method of claim 1, further comprising determining that the subject face does not satisfy a quality threshold based on examination of one or more particular characteristics of the image, wherein selecting the particular cached face and stitching the particular cached face are performed in response to determining that the subject face does not satisfy the quality threshold.

8. The computer-implemented method of claim 7 wherein the one or more particular characteristics of the image include one or more facial attributes depicted in the subject face, the facial attributes including smiling mouth or eyes-open.

9. The computer-implemented method of claim 1, wherein stitching the particular cached face comprises one or more of:

aligning the subject face and the particular cached face based on facial landmark points;

or

performing gradient domain compositing.

10. The computer-implemented method of claim 1, further comprising, prior to the stitching, transforming the particular cached face to fit the image, wherein the transforming comprises one or more of:

performing one or more of color correction or brightness correction on the particular cached face;

applying a face touch-up filter on the particular cached face;

or

changing an angle of the particular cached face.

11. A system comprising:

a storage device with an image cache stored thereon; and

one or more processors configured to access the storage device and to perform operations including;

identifying a subject face of an image, wherein the subject face depicts a subject;

comparing the subject face of the image with one or more cached faces stored in the image cache;

selecting a particular cached face of the one or more cached faces, wherein the selecting is based on the comparing and is based on a number of transformations to be applied to fit the particular cached face to the image being lower than a respective number of transformations to be applied to fit other faces of the one or more cached faces to the image; and

modifying the image by stitching the particular cached face in the image to replace the subject face.

12. The system of claim 11, wherein the operation of selecting the particular cached face comprises:

based on the comparing, determining a candidate list of candidate cached faces;

assigning a score to each candidate cached face in the candidate list of candidate cached faces; and

identifying the particular cached face based on the score.

13. The system of claim 12, wherein the operation of determining the candidate list includes determining the candidate cached faces of the image cache to be on the candidate list, wherein the candidate cached faces include at least one of:

a similar lighting to the subject face or a similar geometric orientation to the subject face.

14. The system of claim 11, wherein the one or more processors are configured to perform further operations comprising determining that the subject face does not satisfy a quality threshold based on examination of one or more particular characteristics of the image, wherein selecting the particular cached face and stitching the particular cached face are performed in response to determining that the subject face does not satisfy the quality threshold.

15. The system of claim 14, wherein the one or more particular characteristics of the image include one or more facial attributes depicted in the subject face, the facial attributes including smiling mouth and eyes-open.

16. The system of claim 11, wherein the one or more processors are configured to perform further operations comprising assigning respective scores for one or more visual image characteristics of the subject face of the image and for one or more visual image characteristics of the one or more cached faces stored in the image cache, wherein the comparing includes comparing the scores for the subject face of the image with the scores for the one or more cached faces,

wherein the operation of selecting the particular cached face is further based on the scores.

17. The system of claim 11, wherein the one or more processors are configured to perform further operations comprising:

determining whether the subject face qualifies to be stored in the image cache based on a comparison of the subject face to the one or more cached faces; and

storing the subject face in the image cache if the subject face qualifies to be stored in the image cache.

18. The system of claim 11, wherein stitching the particular cached face comprises one or more of:

aligning the subject face and the particular cached face based on facial landmark points;

or

performing gradient domain compositing.

19. A non-transitory computer readable medium having stored thereon program instructions that, when executed by a processor, cause the processor to perform operations comprising:

identifying a subject face of an image, wherein the subject face depicts a subject;

comparing the subject face of the image with one or more cached faces stored in an image cache;

selecting a particular cached face of the one or more cached faces, wherein the selecting is based on the comparing and is based on a number of transformations to be applied to fit the particular cached face to the image being lower than a respective number of transformations to be applied to fit other faces of the one or more cached faces to the image; and

modifying the image by stitching the particular cached face in the image to replace the subject face.

20. The non-transitory computer readable medium of claim 19,

wherein the operation of comparing the subject face of the image with the one or more cached faces includes assigning respective scores to one or more visual image characteristics of the one or more cached faces based on comparisons of the one or more visual characteristics of the one or more cached faces with one or more corresponding visual image characteristics of the subject face, wherein the one or more visual characteristics include one or more of lighting characteristics or color characteristics, and

wherein the operation of selecting the particular cached face is further based on the respective scores.