Method and system to correct motion blur in time-of-flight sensor systems

• US 20060241371A1
• Filed: 02/06/2006
• Published: 10/26/2006
• Est. Priority Date: 02/08/2005
• Status: Abandoned Application

• Current Assignee: Canesta Incorporated (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Rafii, Abbas, Gokturk, Salih Burak
• Application Number:
  US11/349,312
• Publication Number:
  US 20060241371A1
• Time in Patent Office: Days
• US Class Current: 600/407
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating

Abstract

A method and system corrects motion blur in time-of-flight (TOF) image data in which acquired consecutive images may evidence relative motion between the TOF system and the imaged object or scene. Motion is deemed global if associated with movement of the TOF sensor system, and motion is deemed local if associated with movement in the target or scene being imaged. Acquired images are subjected to global and then to local normalization, after which coarse motion detection is applied. Correction is made to any detected global motion, and then to any detected local motion. Corrective compensation results in distance measurements that are substantially free of error due to motion-blur.

Claims

1. A method of compensating for error measurement in depth images due to relative motion between a system acquiring the images using an array of pixels and a target object being imaged, the method comprising the following steps:

(a) acquiring a sequence of images;

(b) normalizing the acquired said sequence of images relative to a referenced one of said images;

(c) detecting presence of at least one of coarse motion associated with movement of said system, and local motion associated with movement of said target object, in said acquired said sequence of images; and

(d) compensating for at least one of coarse motion and local motion in said acquired said sequence of said images;

wherein images so compensated at step (d) are substantially free of distance error due to said relative motion.

2. The method of claim 1, wherein said system is a time-of-flight system.

3. The method of claim 1, wherein step (b) includes arbitrarily selected one of said images as said reference image.

4. The method of claim 1, wherein step (b) includes normalizing to have comparable intensity levels in said images relative to said reference image.

5. The method of claim 1, wherein step (b) normalizes said images to have a mean and a standard deviation equal to a mean and a standard deviation of said reference image.

6. The method of claim 1, wherein step (b) includes at least one method selected from a group consisting of normalizing said images using edge detection, and normalizing said images using sub-image patches of said images.

7. The method of claim 1, wherein step (b) includes normalizing relative to each pixel in said pixel array.

8. The method of claim 1, wherein step (b) includes normalizing relative to each pixel in said pixel array using at least one method selected from a group consisting of normalizing image mean and standard deviation, normalizing image edges, and normalizing sub-image patches of said images.

9. The method of claim 1, wherein step (c) includes detecting motion between consecutive frames of said images.

10. The method of claim 9, wherein step (c) further includes detecting differences between normalized said images relative to a reference threshold difference.

11. The method of claim 1, wherein step (c) includes matching substantial block portions of said images relative to at least one of normalized said images and detected edges of normalized said images.

12. The method of claim 11, wherein step (c) minimizes a function given by

${\varepsilon }_{}$

i , p ⁡

( Δ

⁢

 

⁢

x , Δ

⁢

 

⁢

y ) = ∑

x ∈

w i , p ⁢

∑

y ∈

w i , p ⁢

⌊

I i N ⁡

( x + Δ

⁢

 

⁢

x , y + Δ

⁢

 

⁢

y ) - I 0 N ⁡

( x , y ) ⌋

2

where movement of said system is in an (x,y) plane, and where I_i^N is a normalized image, (Δ

x, Δ

y) is a motion vector, where energy function (E) is minimized, and a (Δ

x, Δ

y) minimizing (ε

) is selected as a global motion vector.

13. The method of claim 12, further including iterating around a first (Δ

x, Δ

y) pair obtained in minimizing energy function (ε

).

14. The method of claim 1, wherein step (c) includes detecting local motion by applying Lucas-Kanade motion detection on a per pixel basis, where optimization solves an equation:

${\mathrm{\varepsilon <br><br>}}_{i,p}<br><br>\left(\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x,\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>y\right)=\underset{x\in <br><br>w_{i,p}}{\sum <br><br>}<br><br>\underset{y\in <br><br>w_{i,p}}{\sum <br><br>}<br><br>{\lfloor <br><br>I_i^N<br><br>\left(x+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>x,y+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>y\right)-I_0^N<br><br>\left(x,y\right)\rfloor <br><br>}^2$

where optimization is applied optimization is applied on a window w_i,p around one of every pixel p and every group of pixels p of image I_i.

15. The method of claim 14, further including solving said equation using a Lucas-Kanade tracker.

16. The method of claim 1, wherein step (d) includes determining a vector [Δ

x, Δ

y] for every pixel p and for every image I_i, and compensating by constructing an image I_i0 for each image I_i;

given by I_i0^N(x,y)=I_i^N(x+Δ

x,y+Δ

y).

17. A de-blurring system to compensate for error measurement in depth images due to relative motion between a system acquiring the images using an array of pixels and a target object being imaged, the de-blurring system comprising:

a microprocessor unit;

memory storing a routine that upon execution by said microprocessor unit carries out the following steps;

(a) normalizing a sequence of images, acquired by said system, relative to a referenced one of said images;

(b) detecting presence of at least one of coarse motion associated with movement of said system, and local motion associated with movement of said target object, in said acquired said sequence of images; and

(c) compensating for at least one of coarse motion and local motion in said acquired said sequence of said images;

wherein images so compensated at step (c) are substantially free of distance error due to said relative motion.

18. The de-blurring system of claim method of claim 17, wherein said system is a time-of-flight system.

19. The de-blurring system of claim 17, wherein step (a) includes normalizing said images to have at least one characteristic selected from a group consisting of (i) said images have comparable intensity levels in said images relative to said reference image, (ii) said images have a mean and a standard deviation equal to a mean and a standard deviation of said reference image, (iii) said images are normalized using edge detection, and (iv) said images are normalized using sub-image patches of said images.

20. The de-blurring system of claim 17, wherein step (b) includes at least one of (i) detecting motion between consecutive frames of said images, (ii) detecting differences between normalized said images relative to a reference threshold difference, and (iii) matching substantial block portions of said images relative to at least one of normalized said images and detected edges of normalized said images.