Method of identifying and replacing an object or area in a digital image with another object or area

• US 9,754,166 B2
• Filed: 11/03/2016
• Issued: 09/05/2017
• Est. Priority Date: 03/15/2013
• Status: Expired due to Fees

• Current Assignee: Pandoodle Corporation
• Original Assignee: Pandoodle Corporation
• Inventors: Ludwigsen, David M., Brown, Dirk Dewar, Glassett, Robert John, Griffith, Jason, Bradshaw, Mark
• Primary Examiner(s): Park, Edward
• Assistant Examiner(s): DELANEY, MOLLY K
• Application Number:
  US15/343,054
• Publication Number:
  US 20170161558A1
• Time in Patent Office: 306 Days
• Field of Search: None
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06T 1/00 General purpose image data processing
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A system and method for object and area detection and replacement in an image includes identifying an object or area in one or more sequential images that form a moving image sequence and replacing some or all of the identified object or areas with another image such that the image looks to be part of the original composition of the original image including lighting, shadows, placement, occlusion, orientation, position, and deformation.

Claims

1. A method of identifying and replacing a region of pixels in each image of a series of sequential images, the method comprising:

accessing, using a processor, a database of previously identified objects or areas;

identifying, using the processor, a first region of pixels in a first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database;

identifying, using the processor, the first region of pixels in a first subset of images in the series of sequential images;

generating, using the processor, a lightmap associated with the first region of pixels in the first subset of images in the series of sequential images;

in response to the lightmap being generated, determining a depth and an overall lighting condition of the first region of pixels as the identified object or area in each of the first subset of images in the series of sequential images;

selecting, using the processor, a second region of pixels to replace the first region of pixels;

adjusting, using the processor, one or more features of the second region of pixels based on the depth and the overall lighting condition determined for the first region of pixels in each of the first subset of images, the one or more features of the second region of pixels being adjusted so that the second region of pixels, when substituted in for the first region of pixels, appears to be part of the original composition of the series of sequential images;

inserting, using the processor, the second region of pixels in each of the first subset of images to replace the first region of pixels; and

rendering, using the processor, at least one new image of the series of sequential images including the second region of pixels replacing the first region of pixels in the first subset of images.

2. The method of claim 1, wherein each of the previously identified objects or areas being represented as a two-dimensional array of pixels in the images, the first region of pixels is identified using a two-dimensional array of pixels in the first image, and the second region of pixels is represented as a two-dimensional array of pixels in the image.

3. The method of claim 1, further comprising:

adjusting, using the processor, a size of the second region of pixels based on the size of the first region of pixels in each of the first subset of images.

4. The method of claim 1, wherein identifying, using the processor, the first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprises:

automatically searching and identifying the first region of pixels in the first image using computer vision methods and using the previously identified objects or areas from the database.

5. The method of claim 1, wherein identifying, using the processor, the first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprises:

automatically searching and identifying the first region of pixels in the first image by performing correlation with the previously identified objects or areas from the database.

6. The method of claim 1, wherein identifying, using the processor, a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprises:

selecting a selection area in the first image; and

automatically searching and identifying the first region of pixels in the first image by searching within the selection area.

7. The method of claim 6, wherein selecting a selection area in the first image comprises:

selecting a selection area in the first image based on objects or areas already identified in the first image.

8. The method of claim 6, wherein selecting a selection area in the first image comprises:

receiving a user input identifying the selection area.

9. The method of claim 1, wherein adjusting, using the processor, one or more features of the second region of pixels based on the depth and the overall lighting condition determined for the first region of pixels in each of the first subset of images comprises:

adjusting one or more features of the second region of pixels, the one or more features including lighting, shadows, placement, occlusion, orientation, position or deformation.

10. The method of claim 1, wherein adjusting, using the processor, one or more features of the second region of pixels based on the depth and the overall lighting condition determined for the first region of pixels in each of the first subset of images comprises:

converting a two dimensional image of a first object or area defined by the first region of pixels into a three dimensional image of the first object or area; and

extracting one of occlusion, depth or lighting information of the first object or area using the three dimensional image.

11. The method of claim 1, further comprising:

transmitting the at least one new image over a data network while continuing to render new images of the series of sequential images including the second region of pixels.

12. The method of claim 1, wherein rendering at least one new image of the series of sequential images including the second region of pixels in the first subset of images comprises:

generating a series of new sequential images being the series of sequential images with the second region of pixels replacing the first region of pixels in each of the first subset of images.

13. The method of claim 1, wherein the series of sequential images comprises a series of original sequential images and rendering at least one new image of the series of sequential images including the second region of pixels in the first subset of images comprises:

generating a set of new sequential images corresponding to the first subset of images, each image of the set of new sequential images including the second region of pixels and being transparent where the second region of pixels does not occupy any area in the new sequential image; and

generating information describing arrangement of the set of new sequential images with the series of original sequential images.

14. The method of claim 13, further comprising:

transmitting at least one image from the series of original sequential images and one image of the set of new sequential images over a data network while continuing to render new images in the set of original sequential images and the set of new sequential images.

15. The method of claim 1, wherein the second region of pixels comprises a replacement object or area and selecting a second region of pixels to replace the first region of pixels comprises:

automatically selecting a replacement object or area using information relating to a user.

16. The method of claim 15, wherein automatically selecting a replacement object or area using information relating to a user comprises:

automatically selecting a replacement object or area using information relating to the user, the information including at least one of the geographic locations of the user, the likes and dislikes of the user, the favorite brands of the user, and the favorite foods of the user.

17. The method of claim 1, further comprising:

automatically selecting the first image from the series of sequential images in response to the detection of the first object or area in the first image.

18. The method of claim 1, further comprising:

automatically selecting the first image from the series of sequential images in response to the detection of one or more predetermined spoken words in an audio track associated with the first image.

19. The method of claim 1, further comprising:

tagging one or more images of the series of sequential images using one or more keywords; and

automatically selecting the first image from the series of sequential images in response to the detection of one or more predetermined keywords tagged with the first image.

20. A system for identifying and replacing an object or area in a series of sequential images, comprising:

a processor; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to;

access a database of previously identified objects or areas;

identify a first region of pixels in a first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database;

identify the first region of pixels in a first subset of images in the series of sequential images;

generate a lightmap associated with the first region of pixels in the first subset of images in the series of sequential images;

in response to the lightmap being generated, determine a depth and an overall lighting condition of the first region of pixels as the identified object or area in each of the first subset of images in the series of sequential images;

select a second region of pixels to replace the first region of pixels;

adjust one or more features of the second region of pixels based on the depth and the overall lighting condition determined for the first region of pixels in each of the first subset of images, the one or more features of the second region of pixels being adjusted so that the second region of pixels, when substituted in for the first region of pixels, appears to be part of the original composition of the series of sequential images;

insert the second region of pixels in each of the first subset of images to replace the first region of pixels; and

render at least one new image of the series of sequential images including the second region of pixels replacing the first region of pixels in the first subset of images.