Method and system for segmenting moving objects from images using foreground extraction

• US 8,941,726 B2
• Filed: 12/10/2009
• Issued: 01/27/2015
• Est. Priority Date: 12/10/2009
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Marks, Tim K., Veeraraghavan, Ashok, Taguchi, Yuichi
• Primary Examiner(s): Bengzon, Greg C
• Application Number:
  US12/634,839
• Publication Number:
  US 20110141251A1
• Time in Patent Office: 1,874 Days
• Field of Search: None
• US Class Current: 348/61
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling

Abstract

A set of images is acquired of a scene by a camera. The scene includes a moving object, and a relative difference of a motion of the camera and a motion of the object is substantially zero. Statistical properties of pixels in the images are determined, and a statistical method is applied to the statistical properties to identify pixels corresponding to the object.

Claims

1. A method for extracting pixels from a set of images, comprising a processor for performing steps of the method, comprising the steps of:

acquiring the set of images of a scene with a camera having motion with respect to the scene, wherein the scene includes an object having motion with respect to the scene, wherein the motion of the object and the motion of the camera are substantially identical such that a relative difference of the motion of the camera and the motion of the object is substantially zero, in both translation location and angular orientation, while acquiring the set of images, and wherein the object is stationary with respect to the camera, in both translation location and angular orientation, while acquiring the set of images;

determining statistical properties of pixels in the set of images, wherein the statistical properties measure spatial variations of the pixels, wherein the determining further comprises classifying the pixels by thresholding the spatial variations of the pixels, and wherein the camera is mounted on a robot arm holding the object using an end effector; and

applying a statistical method to the statistical properties to identify the pixels corresponding to the object.

2. The method of claim 1, wherein a robot arm is used in a bin picking application.

3. The method of claim 1, wherein the object is held by an end effector that is mounted on a first robot arm, the camera is mounted on a second robot arm, and the camera and object move in parallel.

4. The method of claim 1, wherein the object is arranged on a moving conveyor, and the camera is moved with a velocity of the object.

5. The method of claim 1, wherein the pixels associated with a background of the scene are subject to motion blur while the pixels associated with the object appear static.

6. The method of claim 1, wherein the determining further comprises:

evaluating jointly the spatial variations of the pixels.

7. The method of claim 1, wherein each pixel has an intensity in each image, and wherein the determining further comprises:

applying a Canny edge detector to the intensities in each image.

8. The method of claim 1, wherein the statistical properties are based on a distribution of features at each pixel as the features vary across the set of images.

9. The method of claim 8, wherein the features include combinations of intensity, color, spatial gradient of intensity or color, or spatial frequency.

10. The method of claim 1, wherein S is a set of all the pixels in the set of images, F is a set of foreground pixels, and B is a set of background pixels, and g(x, y, i) represents features of the pixel (x, y) in the i^th image such that

S=F∪

B, wherein ∪

is a union operator;

F∩

B={ }, wherein ∩

is an intersection operator and { } is a null set; and

If the pixel (x, y) is in the set F, then g(x, y, i)≈

g(x, y, j) for all pairs of images {i, j} in the set of images.

11. The method of claim 1, wherein the set of images is a temporal sequence.

12. The method of claim 11, wherein the statistical properties measure variations in temporal features of the pixels.

13. The method of claim 12, wherein the temporal features include temporal gradients or temporal frequency of intensity or color.

14. The method of claim 11, wherein the statistical properties measure spatio-temporal features or a combination of spatial features and temporal features.

15. The method of claim 10, wherein the statistical method is a hypothesis test comprising:

a null hypothesis H₀ such that the features g(x, y, •

) are drawn from a foreground distribution p_f (x, y); and

an alternative hypothesis H₁ such that the features g(x, y, •

) are not drawn from the foreground distribution p_f (x, y).

16. The method of claim 15, further comprising:

evaluating the hypothesis test by thresholding a likelihood that the features are drawn from the foreground distribution p_f (x, y), and

(x,y) is in the set F if P(g(x,y,•

)|F)>

T

and

(x,y) is in the set B if P(g(x,y,•

)|F)≦

T,

where T is a threshold.

17. The method of claim 16, wherein the features are intensities I(x, y, •

) of each pixel (x, y), the foreground distribution p_f (x, y) is modeled as a normal distribution with a constant variance for all foreground pixels, a standard deviation s estimated from the intensities of a particular pixel (x, y) is

18. The method of claim 1, wherein the statistical method is Bayesian.

19. The method of claim 1, further comprising:

determining a current pose of the object using the pixels identified with the object.

20. The method of claim 19, further comprising:

positioning the object from the current pose to a desired pose.

21. The method of claim 1, wherein the statistical properties are probabilistic, and wherein the statistical method is used to estimate a confidence of a classification of each pixel as being associated with a foreground or a background.

22. The method of claim 1, wherein the camera and the object move during an exposure time of a single image.

25. The method of claim 1, wherein the spatial variations include spatial gradients or spatial frequency of intensity or color of the pixels.

23. A system for extracting pixels from a set of images, comprising:

a camera configured to acquire the set of images of a scene, wherein the camera has motion with respect to the scene, wherein the scene includes an object having motion with respect to the scene, wherein the motion of the object and the motion of the camera are substantially identical such that a relative difference of the motion of the camera and the motion of the object is substantially zero, in both translation location and angular orientation, while acquiring the set of images, and wherein the object is stationary with respect to the camera, in both translation location and angular orientation, while acquiring the set of images;

means for determining statistical properties of pixels in the set of images, wherein the statistical properties measure spatial variations of the pixels, wherein the determining further comprises classifying the pixels by thresholding the spatial variations of the pixels, and wherein the camera is mounted on a robot arm holding the object using an end effector; and

means for applying a statistical method to the statistical properties to identify the pixels corresponding to the object.

24. The system of claim 23, further comprising:

a robot arm, wherein the camera is arranged on the robot arm, and wherein the robot arm further comprises;

an end effector for holding the object.