Segmenting images and simulating motion blur using an image sequence

• US 20070086675A1
• Filed: 10/13/2005
• Published: 04/19/2007
• Est. Priority Date: 10/13/2005
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Software (California), Inc.
• Inventors: Song, Yang, Leung, Thomas, Ioffe, Sergey, Chinen, Troy
• Granted Patent: US 7,702,131 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188

Abstract

A sequence of images depicts a foreground object in motion. A base image is selected, and the other images in the sequence are co-registered with the base image in order to align the images to a common coordinate system. A background image and a binary foreground mask are generated from the sequence of aligned images. By applying the foreground mask to a chosen one of the aligned images, a representation of the moving object is extracted. After blurring the background image, the extracted representation may be superimposed onto the blurred background image to produce a new image.

Claims

1. A method for extracting a moving object representation from a series of images comprising:

aligning each of the images in the series to a common coordinate system;

obtaining a background image from the aligned images; and

extracting the moving object representation from a particular image of the aligned images based on the background image.

2. The method of claim 1, further comprising:

detecting tiepoints among the images in the series,

wherein the images in the series are aligned so that the detected tiepoints coincide in the common coordinate system.

3. The method of claim 2, further comprising:

identifying keypoints in each of the images in the series,

wherein each of the detected tiepoints corresponds to matching keypoints among the images.

4. The method of claim 2, further comprising:

applying a model parameter estimation process on the detected tiepoints to reject erroneous tiepoints.

5. The method of claim 4, wherein the utilized model parameter estimation process utilizes random sample consensus (RANSAC).

6. The method of claim 1, wherein the images in the series, which include a base image, are aligned by:

calculating a transform for mapping coordinates in each of the non-base images in the series to the base image'"'"'s coordinate system; and

resampling each of the non-base images into the base image'"'"'s coordinate system using the calculated transform, thereby aligning the images in the series.

7. The method of claim 1, wherein the obtaining a background image includes determining, for each of a plurality of pixel positions in the particular image, a median value from the corresponding pixels in the aligned images.

8. The method of claim 7, wherein the obtaining a background image includes:

generating a foreground color histogram based on the detected tiepoints in the images that correspond to the moving object representation; and

calculating a background probability for each of a plurality of pixel positions in the particular image based on;

corresponding pixel values in the aligned images, and the foreground color histogram.

9. The method of claim 1, wherein the extracting the moving object representation includes:

for each of a plurality of pixel positions in the particular image, subtracting a pixel value in the particular image from a corresponding value in the background image to obtain a foreground image;

converting the foreground image to grayscale;

generating a binarized foreground mask; and

multiplying pixel values in the particular image with corresponding values in the binarized foreground mask.

10. The method of claim 9, wherein the generating a binarized foreground mask includes:

applying a threshold to pixels in the converted foreground image; and

applying morphological filtering on a signal produced by applying the threshold.

11. The method of claim 9, further comprising:

inserting the extracted moving object representation into a blurred background image.

12. The method of claim 11, wherein the blurred image is obtained by convolving the background image with a blur filter to obtain a blurred background image.

13. The method of claim 11, wherein the inserting the extracted moving object representation includes:

inverting the foreground mask;

zeroing out an area in the blurred background image by multiplying pixel values in the blurred background image with corresponding values in the inverted foreground mask; and

superimposing the extracted moving object representation onto the zeroed out area.

14. A computer program product including a computer program code physically embodied on a computer readable medium, the computer program code, when executed, causing one or more computers to:

receive a series of images;

align each of the images in the series to a common coordinate system;

obtain a background image from the aligned images; and

extract a moving object representation from a particular image of the aligned images based on the background image.

15. The computer program product of claim 14, wherein the computer program code further causes the one or more computers to:

detect tiepoints among the images in the series,

wherein the images in the series are aligned so that the detected tiepoints coincide in the common coordinate system.

16. The computer program product of claim 15, wherein the computer program code further causes the one or more computers to:

identify keypoints in each of the images in the series,

wherein each of the detected tiepoints corresponds to matching keypoints among the images in the series.

17. The computer program product of claim 14, wherein the computer program code further causes the one or more computers to:

apply a model parameter estimation process on the identified tiepoints to reject erroneous tiepoints.

18. The computer program product of claim 14, wherein, in order to align the images, which include a base image, the computer program code causes the one or more computers to:

calculate a transform for mapping coordinates in each of the non-base images in the series into the base image'"'"'s coordinate system; and

resample each of the non-base images into the base image'"'"'s coordinate system using the calculated transform thereby aligning the images in the series.

19. The computer program product of claim 18, wherein, in order to obtain the background image, the computer program code causes the one or more computers to:

determine, for each of a plurality of pixel positions in the particular image, a median value from the corresponding pixels in the aligned images.

20. The computer program product of claim 14, wherein, in order to extract the moving object representation, the computer program code causes the one or more computers to:

for each of a plurality of pixel positions in the particular image, subtract a pixel value in the particular image from a corresponding value in the background image to obtain a foreground image;

convert the foreground image to grayscale;

generate a binarized foreground mask; and

multiply pixel values in the particular image with corresponding values in the binarized foreground mask.

21. The computer program product of claim 20, wherein, in order to generate a binarized foreground mask, the computer program code causes the one or more computers to:

apply a threshold to pixels in the converted foreground image; and

apply morphological filtering on a signal produced by applying the threshold.

22. The computer program product of claim 20, wherein the computer program code further causes the one or more computers to:

insert the extracted moving object representation into a blurred background image, the blurred background image being obtained by convolving the background image with a blur filter to obtain a blurred background image.