Segmenting images and simulating motion blur using an image sequence

• US 7,702,131 B2
• Filed: 10/13/2005
• Issued: 04/20/2010
• Est. Priority Date: 10/13/2005
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Song, Yang, Leung, Thomas, Ioffe, Sergey, Chinen, Troy
• Primary Examiner(s): Seth; Manav
• Application Number:
  US11/248,175
• Publication Number:
  US 20070086675A1
• Time in Patent Office: 1,650 Days
• Field of Search: 382/100, 382/103, 382/164, 382172-180, 382190-201, 382/236, 382276-278, 382284-295, 348/14.14, 348/14.15, 348 94- 95, 348/152, 348/155, 348169-172
• US Class Current: 382/103
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188

Abstract

A sequence of images depicts a foreground object in motion. A base image is selected, and the other images in the sequence are co-registered with the base image in order to align the images to a common coordinate system. A background image and a binary foreground mask are generated from the sequence of aligned images. By applying the foreground mask to a chosen one of the aligned images, a representation of the moving object is extracted. After blurring the background image, the extracted representation may be superimposed onto the blurred background image to produce a new image.

Claims

1. A method for extracting a moving object representation from a series of images comprising:

causing one or more processors to perform the steps of;

aligning each of the images in the series to a common coordinate system;

obtaining a background image from the aligned images;

extracting the moving object representation from a particular image of the aligned images based on the background image; and

detecting tiepoints among the images in the series,

wherein the images in the series are aligned so that the detected tiepoints coincide in the common coordinate system.

2. The method of claim 1, further comprising causing the one or more processors to perform the step of:

identifying keypoints in each of the images in the series,

wherein each of the detected tiepoints corresponds to matching keypoints among the images.

3. The method of claim 1, further comprising causing the one or more processors to perform the step of:

applying a model parameter estimation process on the detected tiepoints to reject erroneous tiepoints.

4. The method of claim 3, wherein the utilized model parameter estimation process utilizes random sample consensus (RANSAC).

5. The method of claim 1, wherein the images in the series, which include a base image, are aligned by causing the one or more processors to perform the steps of:

calculating a transform for mapping coordinates in each of the non-base images in the series to the base image'"'"'s coordinate system; and

resampling each of the non-base images into the base image'"'"'s coordinate system using the calculated transform, thereby aligning the images in the series.

6. The method of claim 1, wherein the obtaining a background image includes causing the one or more processors to perform the step of determining, for each of a plurality of pixel positions in the particular image, a median value from the corresponding pixels in the aligned images.

7. The method of claim 6, wherein the obtaining a background image includes causing the one or more processors to perform the steps of:

generating a foreground color histogram based on the detected tiepoints in the images that correspond to the moving object representation; and

calculating a background probability for each of a plurality of pixel positions in the particular image based on;

corresponding pixel values in the aligned images, and the foreground color histogram.

8. The method of claim 1, wherein the extracting the moving object representation includes causing the one or more processors to perform the steps of:

for each of a plurality of pixel positions in the particular image, subtracting a pixel value in the particular image from a corresponding value in the background image to obtain a foreground image;

converting the foreground image to grayscale;

generating a binarized foreground mask; and

multiplying pixel values in the particular image with corresponding values in the binarized foreground mask.

9. The method of claim 8, wherein the generating a binarized foreground mask includes causing the one or more processors to perform the steps of:

applying a threshold to pixels in the converted foreground image; and

applying morphological filtering on a signal produced by applying the threshold.

10. The method of claim 8, further comprising causing the one or more processors to perform the step of:

inserting the extracted moving object representation into a blurred background image.

11. The method of claim 10, wherein the blurred image is obtained by causing the one or more processors to perform the step of convolving the background image with a blur filter to obtain a blurred background image.

12. The method of claim 10, wherein the inserting the extracted moving object representation includes causing the one or more processors to perform the steps of:

inverting the foreground mask;

zeroing out an area in the blurred background image by multiplying pixel values in the blurred background image with corresponding values in the inverted foreground mask; and

superimposing the extracted moving object representation onto the zeroed out area.

13. A computer program product including a computer program code physically embodied on a computer readable medium, the computer program code, when executed, causing one or more computers to:

receive a series of images;

align each of the images in the series to a common coordinate system;

obtain a background image from the aligned images;

extract a moving object representation from a particular image of the aligned images based on the background image; and

detect tiepoints among the images in the series,

wherein the images in the series are aligned so that the detected tiepoints coincide in the common coordinate system.

14. The computer program product of claim 13, wherein the computer program code further causes the one or more computers to:

identify keypoints in each of the images in the series,

wherein each of the detected tiepoints corresponds to matching keypoints among the images in the series.

15. The computer program product of claim 13, wherein the computer program code further causes the one or more computers to:

apply a model parameter estimation process on the identified tiepoints to reject erroneous tiepoints.

16. The computer program product of claim 13, wherein, in order to align the images, which include a base image, the computer program code causes the one or more computers to:

calculate a transform for mapping coordinates in each of the non-base images in the series into the base image'"'"'s coordinate system; and

resample each of the non-base images into the base image'"'"'s coordinate system using the calculated transform thereby aligning the images in the series.

17. The computer program product of claim 16, wherein, in order to obtain the background image, the computer program code causes the one or more computers to:

determine, for each of a plurality of pixel positions in the particular image, a median value from the corresponding pixels in the aligned images.

18. The computer program product of claim 13, wherein, in order to extract the moving object representation, the computer program code causes the one or more computers to:

for each of a plurality of pixel positions in the particular image, subtract a pixel value in the particular image from a corresponding value in the background image to obtain a foreground image;

convert the foreground image to grayscale;

generate a binarized foreground mask; and

multiply pixel values in the particular image with corresponding values in the binarized foreground mask.

19. The computer program product of claim 18, wherein, in order to generate a binarized foreground mask, the computer program code causes the one or more computers to:

apply a threshold to pixels in the converted foreground image; and

apply morphological filtering on a signal produced by applying the threshold.

20. The computer program product of claim 18, wherein the computer program code further causes the one or more computers to:

insert the extracted moving object representation into a blurred background image, the blurred background image being obtained by convolving the background image with a blur filter to obtain a blurred background image.