Virtual masking using rigid parametric modeling

• US 8,194,097 B2
• Filed: 12/12/2008
• Issued: 06/05/2012
• Est. Priority Date: 12/12/2008
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Xiao, Jing, Tan, Kar-Han, Bhattacharjya, Anoop K.
• Primary Examiner(s): Wu, Xiao M.
• Assistant Examiner(s): He, Yingchun
• Application Number:
  US12/333,623
• Publication Number:
  US 20100150406A1
• Time in Patent Office: 1,271 Days
• Field of Search: 345/619, 345629-464, 382/118, 382/154, 382/195, 382/201, 382/203, 382212-213, 382/216, 382/220, 382275-276, 382282-284
• US Class Current: 345/619
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Disclosed are systems and methods for masking at least a portion of one image with another image. Aspects of the present invention facilitate the placing of a virtual mask onto an item in an image even if that items moves in subsequent images, such as between different image frames in a video.

Claims

1. A method for masking a target item by a processing device within an image, the method comprising the steps of:

(a) defining a target image encompassing said target item within an input image;

(b) defining a mask image with which to mask said target item;

(c) scaling the size of a three-dimensional rigid parametric model to the size of the target image within said input image;

(d) determining the projection of said target image to the scaled rigid parametric model to establish a pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model;

(e) constructing a pixel mapping between said mask image and the target image to obtain a mapped mask image with a one-to-one pixel correspondence between image pixels in said target image and image pixels in said mapped mask image;

(f) determining the projection of said mapped mask image to the scaled rigid parametric model in accordance with said pixel correspondence between said target image and said mapped mask image and said pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model;

(g) replacing said target item within said input image with a composite of said target image and the mapped mask image in accordance with said pixel correspondence between said target image and said mapped mask image and said pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model, said, composite image being a weighted sum of the target image and the mapped mask image

wherein the weighted sum of the target image and the mapped mask image is defined as I_b=W*I_c+(1−

w)*I_m, where I_c is a given pixel in the target image, I_m is its corresponding pixel in the mapped mask image, and I_b is the resultant pixel in the composite image; and

w is the weight defined as

2. The method of claim 1, wherein said input image is a first in a sequence of frame images within a video sequence, and in step (a) said target image encompassing a frontal view of said target item, said method further comprising;

defining a three-dimensional tracker based on said scaled rigid parametric model to recover three-dimensional movements of said target item across said frame images; and

for each subsequent frame image following said input image;

(i) aligning said projection of said target image and said projection of said mapped mask image according to the three-dimensional orientation of said scaled rigid parametric model as determined by said three-dimensional tracker; and

(ii) replacing the target item within the current frame image with a composite of said target image and the mapped mask image in accordance with said pixel correspondence between said target image and said mapped mask image and said pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model, said composite image being a weighted sum of said target image and the mapped mask image.

3. The method of claim 1, wherein the shape of said three-dimensional rigid parametric model is independent of the shape of said target item.

4. The method of claim 1, wherein said three-dimensional rigid parametric model is a rigid geometric shape.

5. The method of claim 4, wherein said rigid geometric shape is a cylinder.

6. The method of claim 1, wherein the shape of the perimeter of said target image is defined by a two-dimensional side view of said three-dimensional rigid parametric model.

7. The method of claim 6, wherein in step (c), the three-dimensional rigid parametric model is scaled to the size of the target image within said input image by being made as large as possible while still fitting within perimeter of said target image.

8. The method of claim 6, wherein:

said three-dimensional rigid parametric model is a cylinder and said target item is not a cylinder;

the perimeter shape of said target image is a quadrilateral with four right angles having a height H and a width W; and

in step (c) the size of said three-dimensional rigid parametric model is scaled to the size of the target image by setting the height of said cylinder to H and setting the radius of said cylinder to W/(2*sin(θ

)).

9. The method of claim 8, wherein θ

equals 60 degrees.

10. The method of claim 1, wherein step (e) includes scaling said mask image to the size of said target image, and the defining of said mapped mask image includes identifying corresponding pixels between said target image and said scaled mask image based on their common relative location within their respective image.

11. The method of claim 10, wherein the pixel density of the scaled mask image is different than the pixel density of said target image, the defining of said mapped mask image further includes:

when no corresponding pixel in said scaled mask image is identifiable for a given pixel in said target image based on relative pixel location each respective image, a new pixel is defined within said mapped mask image to maintain said one-to-one correspondence by pixel interpolation of a predefined number of neighbor pixels of said scaled mask image nearest to the corresponding relative location within the scaled mask image.

12. The method of claim 2, wherein in said target item is a human face and in step (a), the defining of said target image is automated by means of a face detector.

13. The method of claim 1, wherein in step (g), the weight of a given pixel in the target image and the weight of its corresponding pixel in the mapped mask image is dependent upon the given pixel'"'"'s distance to the mask boundary, wherein the closer the given pixel is to the mask boundary the greater the weight assigned to the pixel in the target image and smaller the weight assigned to the corresponding pixel in the mapped mask image.

14. A non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform at least the steps of claim 1.

15. A system for masking a target item within an image, the system comprising:

an item detector for-defining a target image encompassing said target item within an input image;

a mask definer foe-defining a mask image with which to mask said target item;

a scaler foe-scaling the size of a three-dimensional, rigid parametric model to the size of the target image within said input image, wherein the shape of said rigid parametric model is independent of the shape of said target image;

a projector foe-determining the projection of said target image to the scaled rigid parametric model to establish a pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model; and

a mapper constructing a pixel mapping between said mask image and the target image to obtain a mapped mask image with a one-to-one pixel correspondence between image pixels in said target image and image pixels in said mapped mask image;

said projector further determining the projection of said mapped mask image to the scaled rigid parametric model in accordance with said pixel correspondence between said target image and said mapped mask image and said pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model;

said projector further replacing said target item within said input image with a composite of said target image and the mapped mask image in accordance with said pixel correspondence between said target image and said mapped mask image and said pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model, said composite image being a weighted sum of the target image and the mapped mask image;

wherein;

wherein the weighted sum of the target image and the mapped mask image is defined as I_b=W*I_c+(1−

w)*I_m, where I_c is a given pixel in the target image, I_m is its corresponding pixel in the mapped mask image, and I_b is the resultant pixel in the composite image; and

w is the weight defined as

16. The system of claim 15, wherein:

the shape of said rigid parametric model is a rigid geometric shape;

said input image is a first in a sequence of frame images within a video sequence; and

said target image encompasses a frontal view of said target item;

said system further having a data processing unit defining a three-dimensional tracker based on the rigid parametric model to recover three-dimensional movements of said target item across said frame images; and

for each subsequent frame image following said input image;

(i) aligning said projection of said target image and said projection of said mapped mask image according to the three-dimensional orientation of said scaled rigid parametric model as determined by said three-dimensional tracker; and

(ii) replacing the target item within the current frame image with a composite of said target image and the mapped mask image in accordance with said pixel correspondence between said target image and said mapped mask image and said pixel correlation between image pixel locations in said target image and corresponding projected pixel locations on said scaled rigid parametric model, said composite image being a weighted sum of said target image and the mapped mask image.

17. The system of claim 15, wherein:

said rigid geometric shape is a cylinder;

the perimeter shape of said target image is a quadrilateral with four right angles having a height H and a width W; and

said scaler scales the size of the three-dimensional rigid parametric model is scaled to the size of the target image by setting the height of said cylinder to H and setting the radius of said cylinder to W/(2*sin(θ

)), wherein θ

is fixed at a predefined angle value.

18. The system of claim 15, wherein said mapper scales said mask image to the size of said target image, the pixel density of the scaled mask image being different than the pixel density of said target image, and the defining of said mapped mask image includes:

(i) identifying corresponding pixels between said target image and said scaled mask image based on their common relative location within their respective image; and

(ii) when no corresponding pixel in said scaled mask image is identifiable for a given pixel in said target image based on relative pixel location each respective image, a new pixel is defined within said mapped mask image to maintain said one-to-one correspondence by pixel interpolation of a predefined number of neighbor pixels of said scaled mask image nearest to the corresponding relative location within the scaled mask image.