Method of identifying and replacing an object or area in a digital image with another object or area

• US 9,514,381 B1
• Filed: 03/13/2014
• Issued: 12/06/2016
• Est. Priority Date: 03/15/2013
• Status: Expired due to Fees

• Current Assignee: Pandoodle Corporation
• Original Assignee: Pandoodle Corporation
• Inventors: Griffith, Jason, Bradshaw, Mark, Ludwigsen, David M., Brown, Dirk Dewar, Glassett, Robert John
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): Delaney, Molly
• Application Number:
  US14/209,361
• Time in Patent Office: 999 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06T 1/00 General purpose image data processing
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A system and method for object and area detection and replacement in an image includes identifying an object or area in one or more sequential images that form a moving image sequence and replacing some or all of the identified object or areas with another image such that the image looks to be part of the original composition of the original image including lighting, shadows, placement, occlusion, orientation, position, and deformation.

Claims

1. A method of identifying and replacing a region of pixels in each image of a series of sequential images, the method comprising:

accessing, using a processor, a database of previously identified objects or areas, the previously identified objects or areas being discrete, identifiable elements of images that are naturally occurring in the images, each of the previously identified objects or areas being represented as a two-dimensional array of pixels in the images;

searching, using the processor, within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database, the identifiable objects or areas being discrete, identifiable elements of the first image that are naturally occurring in the first image;

identifying, using the processor, a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database, the first region of pixels being identified using a two-dimensional array of pixels in the first image;

identifying, using the processor, the first region of pixels and one or more features associated with the first region of pixels in a first subset of images in the series of sequential images using a two-dimensional array of pixels in each of the first subset of images;

selecting, using the processor, a second region of pixels to replace the first region of pixels, the second region of pixels being represented as a two-dimensional array of pixels in the image;

adjusting, using the processor, one or more features of the second region of pixels based on information derived from the one or more features of the first region of pixels in each of the first subset of images, the one or more features of the second region of pixels being adjusted so that the second region of pixels, when substituted in for the first region of pixels, appears to be part of the original composition of the series of sequential images;

inserting, using the processor, the second region of pixels represented as a two-dimensional array of pixels in each of the first subset of images to replace the first region of pixels; and

rendering, using the processor, at least one new image of the series of sequential images including the second region of pixels replacing the first region of pixels in the first subset of images.

2. The method of claim 1, further comprising:

adjusting, using the processor, a size of the second region of pixels based on the size of the first region of pixels in each of the first subset of images.

3. The method of claim 1, wherein searching, using the processor, within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database and identifying, using the processor, a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprise:

automatically searching and identifying the first region of pixels in the first image using computer vision methods and using the previously identified objects or areas from the database.

4. The method of claim 1, wherein searching, using the processor, within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database and identifying, using the processor, a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprise:

automatically searching and identifying the first region of pixels in the first image by performing correlation with the previously identified objects or areas from the database.

5. The method of claim 1, wherein identifying, using the processor, a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprises:

identifying an object in the first image or an area of the first image as the first region of pixels.

6. The method of claim 1, wherein searching, using the processor, within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database and identifying, using the processor, a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database comprise:

selecting a selection area in the first image; and

automatically searching and identifying the first region of pixels in the first image by searching within the selection area.

7. The method of claim 6, wherein selecting a selection area in the first image comprises:

selecting a selection area in the first image based on objects or areas already identified in the first image.

8. The method of claim 6, wherein selecting a selection area in the first image comprises:

receiving a user input identifying the selection area.

9. The method of claim 1, wherein adjusting, using the processor, one or more features of the second region of pixels based on information derived from the one or more features of the first region of pixels in each of the first subset of images comprises:

adjusting one or more features of the second region of pixels, the one or more features including lighting, shadows, placement, occlusion, orientation, position or deformation.

10. The method of claim 1, wherein adjusting, using the processor, one or more features of the second region of pixels based on information derived from the one or more features of the first region of pixels in each of the first subset of images comprises:

converting a two dimensional image of a first object or area defined by the first region of pixels into a three dimensional image of the first object or area; and

extracting one of occlusion, depth or lighting information of the first object or area using the three dimensional image.

11. The method of claim 1, further comprising:

transmitting the at least one new image over a data network while continuing to render new images of the series of sequential images including the second region of pixels.

12. The method of claim 1, wherein rendering at least one new image of the series of sequential images including the second region of pixels in the first subset of images comprises:

generating a series of new sequential images being the series of sequential images with the second region of pixels replacing the first region of pixels in each of the first subset of images.

13. The method of claim 1, wherein the series of sequential images comprises a series of original sequential images and rendering at least one new image of the series of sequential images including the second region of pixels in the first subset of images comprises:

generating a set of new sequential images corresponding to the first subset of images, each image of the set of new sequential images including the second region of pixels and being transparent where the second region of pixels does not occupy any area in the new sequential image; and

generating information describing arrangement of the set of new sequential images with the series of original sequential images.

14. The method of claim 13, further comprising:

transmitting at least one image from the series of original sequential images and one image of the set of new sequential images over a data network while continuing to render new images in the set of original sequential images and the set of new sequential images.

15. The method of claim 1, wherein the second region of pixels comprises a replacement object or area and selecting a second region of pixels to replace the first region of pixels comprises:

automatically selecting a replacement object or area using information relating to a user.

16. The method of claim 15, wherein automatically selecting a replacement object or area using information relating to a user comprises:

automatically selecting a replacement object or area using information relating to the user, the information including at least one of the geographic locations of the user, the likes and dislikes of the user, the favorite brands of the user, and the favorite foods of the user.

17. The method of claim 1, further comprising:

automatically selecting the first image from the series of sequential images in response to the detection of the first object or area in the first image.

18. The method of claim 1, further comprising:

automatically selecting the first image from the series of sequential images in response to the detection of one or more predetermined spoken words in an audio track associated with the first image.

19. The method of claim 1, further comprising:

tagging one or more images of the series of sequential images using one or more keywords; and

automatically selecting the first image from the series of sequential images in response to the detection of one or more predetermined keywords tagged with the first image.

20. A system for identifying and replacing an object or area in a series of sequential images, comprising:

a processor; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to;

access a database of previously identified objects or areas, the previously identified objects or areas being discrete, identifiable elements of images that are naturally occurring in the images, each of the previously identified objects or areas being represented as a two-dimensional array of pixels in the images;

search within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database, the identifiable objects or areas being discrete, identifiable elements of the first image that are naturally occurring in the first image;

identify a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database, the first region of pixels being identified using a two-dimensional array of pixels in the first image;

identify the first region of pixels and one or more features associated with the first region of pixels in a first subset of images in the series of sequential images using a two-dimensional array of pixels in each of the first subset of images;

select a second region of pixels to replace the first region of pixels, the second region of pixels being represented as a two-dimensional array of pixels in the image;

adjust one or more features of the second region of pixels based on information derived from the one or more features of the first region of pixels in each of the first subset of images, the one or more features of the second region of pixels being adjusted so that the second region of pixels, when substituted in for the first region of pixels, appears to be part of the original composition of the series of sequential images;

insert the second region of pixels represented as a two-dimensional array of pixels in each of the first subset of images to replace the first region of pixels; and

render at least one new image of the series of sequential images including the second region of pixels replacing the first region of pixels in the first subset of images.

21. The system recited in claim 20, wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:

automatically select a replacement object or area as the second region of pixels using information relating to the user, the information including at least one of the geographic locations of the user, the likes and dislikes of the user, the favorite brands of the user, and the favorite foods of the user.

22. A computer program product for identifying and replacing an object in a series of sequential images, the computer program product being embodied in a tangible and non-transitory computer readable storage medium and comprising computer instructions for:

providing a database of previously identified objects or areas, the previously identified objects or areas being discrete, identifiable elements of images that are naturally occurring in the images, each of the previously identified objects or areas being represented as a two-dimensional array of pixels in the images;

searching, using the processor, within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database, the identifiable objects or areas being discrete, identifiable elements of the first image that are naturally occurring in the first image;

identifying a first region of pixels in the first image of the series of sequential images as an identified object or area by comparing with the previously identified objects or areas from the database, the first region of pixels being identified using a two-dimensional array of pixels in the first image;

identifying and tracking the first region of pixels and one or more features associated with the first region of pixels in a first subset of images in the series of sequential images using a two-dimensional array of pixels in each of the first subset of images;

selecting a second region of pixels to replace the first region of pixel, the second region of pixels being represented as a two-dimensional array of pixels in the image;

adjusting one or more features of the second region of pixels based on information derived from the one or more features of the first region of pixels in each of the first subset of images, the one or more features of the second region of pixels being adjusted so that the second region of pixels, when substituted in for the first region of pixels, appears to be part of the original composition of the series of sequential images;

inserting the second region of pixels represented as a two-dimensional array of pixels in each of the first subset of images to replace the first region of pixels; and

rendering at least one new image of the series of sequential images including the second region of pixels replacing the first region of pixels in the first subset of images.

23. The computer program product recited in claim 22, further comprising computer instructions for:

automatically selecting a replacement object or area as the second region of pixels using information relating to a user, the information including at least one of the geographic locations of the user, the likes and dislikes of the user, the favorite brands of the user, and the favorite foods of the user.

24. A method of identifying and replacing a region of pixels in a first image comprising:

accessing, using a processor, a database of previously identified objects or areas, the previously identified objects or areas being discrete, identifiable elements of images that are naturally occurring in the images, each of the previously identified objects or areas being represented as a two-dimensional array of pixels in the images;

searching, using the processor, within at least a portion of a first image of the series of sequential images for identifiable objects or areas by comparing with the previously identified objects or areas from the database, the identifiable objects or areas being discrete, identifiable elements of the first image that are naturally occurring in the first image;

identifying, using the processor, a first region of pixels in the first image as an identified object or area by comparing with the previously identified objects or areas from the database, the first region of pixels being identified using a two-dimensional array of pixels in the image;

identifying, using the processor, the first region of pixels and one or more features associated with the first region of pixels in the first image using a two-dimensional array of pixels in the first image;

selecting, using the processor, a second region of pixels to replace the first region of pixels, the second region of pixels being represented as a two-dimensional array of pixels in the image;

adjusting, using the processor, one or more features of the second region of pixels based on information derived from the one or more features of the first region of pixels in the first image, the one or more features of the second region of pixels being adjusted so that the second region of pixels, when substituted in for the first region of pixels, appears to be part of the original composition of the first image;

inserting, using the processor, the second region of pixels represented as a two-dimensional array of pixels in the first image to replace the first region of pixels; and

rendering, using the processor, a second image based on the first image and including the second region of pixels replacing the first region of pixels.