Method and apparatus for determining motion in images

• US 7,783,118 B2
• Filed: 07/13/2006
• Issued: 08/24/2010
• Est. Priority Date: 07/13/2006
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Zhou, Hui
• Primary Examiner(s): Johns; Andrew W
• Assistant Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US11/457,349
• Publication Number:
  US 20080037869A1
• Time in Patent Office: 1,503 Days
• Field of Search: 382/100, 382/107, 382/173, 382/181, 382/189, 382/190, 382/195, 382/224, 382/276, 382/286, 382/287, 382/289, 382/293, 382/294, 348/14.15, 348/439.1, 348/E17.003, 348/E5.066, 348/E5.054, 375/240.01, 375/E7.04, 375/E7.065, 375/E7.081, 375/E7.105, 375/E7.256
• US Class Current: 382/236
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20152 Watershed segmentation
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/155 involving morphological operators
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method of determining motion in a set of input images comprises registering the images in the set to identify stationary background content therein and generating difference images based on the registered images. The input images are segmented into regions and local motion within each of the regions is estimated. A determination as to whether the regions include moving content or static content is determined based on the difference images and the estimated local motion within the regions.

Claims

1. A method of determining motion in a set of input images, said method comprising:

providing a processing device to implement the following step;

receiving said set of input images, said set of input images having background motion and foreground motion;

determine a first motion direction for a first group of pixels within a reference image within said set of input images;

determine a second motion direction for a second group of pixel within said reference image;

designating the dominant group of pixels within said reference image as background pixels;

designating said background pixels as stationary content;

registering the remaining images in said set of input images to said reference image to identify background stationary content therein;

generating difference images based on the registered images;

segmenting the input images into regions;

estimating local motion within each of said regions; and

determining whether the regions include moving content or static content based on the difference images and the local estimated motion within said regions;

wherein said local motion estimating comprises generating a motion vector for each region in each of said images with the exception of the last image in said set;

wherein said region content determining for each region comprises;

determining whether the motion vector for the region signifies movement above a threshold and if so labeling the region as representing foreground moving content;

if not, determining whether the percentage of pixels in the region have a pixel value above a threshold in the corresponding difference image and if so labeling the region as representing foreground moving content; and

if not labeling the region as representing background stationary content; and

for a given region in a grayscale input image designated as background stationary content, checking the labeling of its three corresponding regions in three previous grayscale input images, and if more than seventy percent of the pixels within the three corresponding regions are designated as representing foreground movement content, then changing the designation of the given region from background stationary content to foreground movement content.

2. The method of claim 1 wherein said difference image generating comprises determining the difference between each remaining image in said set of input images and said reference image to yield absolute intensity difference images.

3. The method of claim 1 further comprising binarizing the difference images.

4. The method of claim 3 further comprising noise filtering the binarized difference images.

5. The method of claim 1 wherein said difference image generating comprises determining the difference between each remaining image in said set and said reference image to yield absolute intensity difference images.

6. The method of claim 5 wherein said noise filtering is performed by morphing said binarized images.

7. The method of claim 1 wherein said segmenting comprises applying the Watershed algorithm to said input images and compensating for over-segmentation of said input images.

8. The method of claim 7 wherein said compensating comprises merging each region of each input image that is below a threshold size with a boundary region.

9. The method of claim 8 wherein for each region, said boundary region is selected from one of a region greater than said threshold size and the largest boundary region smaller than said threshold size.

10. The method of claim 9 further comprising binarizing the difference images to identify foreground pixels based on the true/false results of the binarized difference.

11. The method of claim 1 wherein said motion vector generating for each region comprises:

for each pixel in the region, calculating the absolute difference between the pixel and a corresponding pixel in the subsequent input image in said set;

summing the calculated absolute differences;

shifting the position of the region according to a search pattern and for each position in the search pattern reperforming the calculating and summing; and

selecting the motion vector associated with the position of the region that yields the minimum absolute difference sum.

13. The method of claim 1 wherein for said last input image in the set, the local motion estimating comprises projecting motion shifted pixels from regions of the second last input image onto the last input image and wherein the region content determining comprises for each corresponding region in the last input image determining whether a threshold number of projected motion shifted pixels fall within the region.

12. A method of determining motion in a set of input images, said method comprising:

providing a processing device to implement the following step;

receiving said set of input images, said set of input images having background motion and foreground motion;

determine a first motion direction for a first group of pixels within a reference image within said set of input images;

determine a second motion direction for a second group of pixel within said reference image;

designating the dominant group of pixels within said reference image as background pixels;

designating said background pixels as stationary content;

registering the remaining images in said set of input images to said reference image to identify background stationary content therein;

generating difference images based on the registered images;

segmenting the input images into regions;

estimating local motion within each of said regions; and

determining whether the regions include moving content or static content based on the difference images and the local estimated motion within said regions;

wherein said local motion estimating comprises generating a motion vector for each region in each of said images with the exception of the last image in said set;

wherein said region content determining for each region comprises;

determining whether the motion vector for the region signifies movement above a threshold and if so labeling the region as representing foreground moving content;

if not, determining whether the percentage of pixels in the region have a pixel value above a threshold in the corresponding difference image and if so labeling the region as representing foreground moving content; and

if not labeling the region as representing background stationary content; and

for a given region within a given grayscale input image designated as foreground movement content, determining if its neighboring regions within the given grayscale input image are designated as representing background stationary content, and if so, then if the total size of neighboring regions designated as representing background stationary content is larger than the size of the given region and the given region is further smaller than a predefined size, then changing the designation of the given region from foreground movement content to background stationary content.

14. The method of claim 12 further comprising:

for a given region within a given grayscale input image designated as background stationary content, determining if its neighboring regions within the given grayscale input image are designated as representing foreground movement content, and if so, then if the total size of neighboring regions designated as representing foreground movement content is larger than the size of the given region and the given region is further smaller than a predefined size, then changing the designation of the given region from background stationary content to foreground movement content.