Image-based tracking

• US 8,229,166 B2
• Filed: 07/07/2009
• Issued: 07/24/2012
• Est. Priority Date: 07/07/2009
• Status: Active Grant

• Current Assignee: Trimble Navigation Limited (Trimble Inc.)
• Original Assignee: Trimble Navigation Limited (Trimble Inc.)
• Inventors: Teng, Hongbo, Best, Gregory C., Wang, Sy Bor
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US12/459,843
• Publication Number:
  US 20110007939A1
• Time in Patent Office: 1,113 Days
• Field of Search: 382/103, 382/107, 382/236, 382/104, 348/169, 348/170, 348/171, 348/172, 348/94, 348/154, 348/155, 348/208.1, 348/208.2, 348/208.14, 348/352, 348/579
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of image-tracking by using an image capturing device. The method comprises: performing an image-capture of a scene by using an image capturing device; and tracking movement of the image capturing device by analyzing a set of images by using an image processing algorithm.

Claims

1. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B) generating a set of captured image data and a set of scene depth data; and

(B1) performing a rigid global transformation of said set of captured image data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

2. The method of claim 1, wherein said step (B1) further comprises:

(B1,

     1) using a mapping function of said image capturing device to map for at least one i-th frame 3D coordinates of at least one object point in the reference system attached to said image capturing device, into 2D pixel coordinates of an image point on said image capturing device sensor plane in a 2D reference coordinate system;

i being an integer.

3. The method of claim 2, wherein said step (B1, 1) further comprises:

(B1, 1,

     1) selecting said mapping function of said image capturing device from the group consisting of;

{a mapping function of regular rectilinear lenses;

a mapping function of fisheye lenses; and

a mapping function of custom-calibrated lenses}.

4. The method of claim 1, wherein said step (B1) further comprises:

(B1,

     2) obtaining a reverse of said mapping function;

wherein said reverse of said mapping function is configured to map, for at least one i-th frame, 2D pixel coordinates of at least one image point on said image capturing device sensor plane in a 2D reference coordinate system, into 3D coordinates of at least one object point in the reference system attached to said image capturing device.

5. The method of claim 1, wherein said step (B1) further comprises:

(B1,

     3) obtaining 3D depth coordinates of said at least one object point by using said pre-surveyed data of said scene.

6. The method of claim 1, wherein said step (B1) further comprises:

(B1,

     4) extracting a 6-vector describing said movement of said image capturing device from a relationship between two sequential frames;

i-th frame and k-th frame, wherein at least one object point produces at least two pixels of substantially the same intensity in said two sequential frames.

7. The method of claim 6 further comprising:

(B1, 4,

     1) optimizing said 6-vector describing the movement of said image capturing device.

8. The method of claim 6 further comprising:

(B1, 4,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

9. The method of claim 6 further comprising:

(B1, 4,

     3) optimizing a (6−

n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

10. The method of claim 1, wherein said step (B1) further comprises:

(B1,

     5) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first said pixel and a second said pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame.

11. The method of claim 10 further comprising:

(B1, 5,

     1) optimizing said 6-vector describing the movement of said image capturing device.

12. The method of claim 10 further comprising:

(B1, 5,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

13. The method of claim 10 further comprising:

(B1, 5,

     3) optimizing a (6−

n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

14. The method of claim 1, wherein said step (B1) further comprises:

(B1,

     6) extracting a 6-vector describing said movement of said image capturing device from a relationship between two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first pixel and a second pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame, and by matching at least two pixels having substantially the same intensity in said two sequential frames.

15. The method of claim 14 further comprising:

(B1, 6,

     1) optimizing said 6-vector describing the movement of said image capturing device.

16. The method of claim 14 further comprising:

(B1, 6,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

17. The method of claim 14 further comprising:

(B1, 6,

     3) optimizing a (6−

n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

18. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B1) determining said set of scene depth data in an image capturing device—

attached 3D reference system by using a K-point range measurement system attached to said image capturing device, wherein said set of scene depth data comprises an integer K-point set of 3D depth measurements of said scene; and

wherein a 3D depth coordinate of at least one object point associated with a 2D image point of said 3D object point is obtained by interpolation of said K-point set of 3D depth measurements of said scene to obtain an optimum depth measurement for at least one image point of at least one said object point; and

wherein said integer K is substantially less than number of pixels in said frame;

and

(C) tracking movement of said image capturing device by analyzing said set of images.

19. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B2) determining said depth of said object point directly for at least one image point of said object point by using an M-point range measurement system attached to said image capturing device, wherein said integer number M of depth measurements of said scene is substantially equal to the number of pixels in said frame;

and

(C) tracking movement of said image capturing device by analyzing said set of images.

20. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B3) determining said set of scene depth data in an image capturing device—

attached 3D reference system by using a feature-point range measurement system attached to said image capturing device, wherein said scene comprises a set of integer K feature object points;

wherein said feature-point range measurement system obtains a K set of 3D depth measurements of said K feature object points in said scene;

and wherein a 3D depth coordinate of at least one object point associated with a 2D image point of said 3D object point is obtained by interpolation of said K-point set of 3D depth measurements of said scene to obtain an optimum depth measurement for at least one image point of at least one said object point;

and wherein said integer K is substantially less than the number of pixels in said frame;

and

(C) tracking movement of said image capturing device by analyzing said set of images.

21. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B) obtaining a set of scene depth data by using a range measurement device selected from the group consisting of;

{a point laser beam;

a sonar;

a radar;

a laser scanner; and

a depth camera};

and

(C1) performing a rigid global transformation of said set of captured images data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

22. The method of claim 21, wherein said step (C1) further comprises:

(C1,

     1) using mapping function to map, for at least one i-th frame, 3D coordinates of at least one object point in the reference system attached to said image capturing device, into 2D pixel coordinates of an image point on said image capturing device sensor plane in a 2D reference coordinate system.

23. The method of claim 22, wherein said step (C1, 1) further comprises:

(C1, 1,

     1) selecting said mapping function from the group consisting of;

{a mapping function of regular rectilinear lenses;

a mapping function of fisheye lenses; and

a mapping function of custom-calibrated lenses}.

24. The method of claim 21, wherein said step (C1) further comprises:

(C1,

     2) obtaining a reverse of said mapping function;

said reverse of said mapping function is configured to map, for at least one i-th frame 2D pixel coordinates of at least one image point on said image capturing device sensor plane in a 2D reference coordinate system, into 3D coordinates of at least one object point in the reference system attached to said image capturing device.

25. The method of claim 21, wherein said step (C1) further comprises:

(C1,

     3) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame, wherein at least one point in said scene produces at least two pixels of substantially the same intensity in said two sequential frames.

26. The method of claim 25 further comprising:

(C1, 3,

     1) optimizing said 6-vector describing the movement of said image capturing device.

27. The method of claim 25 further comprising:

(C1, 3,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

28. The method of claim 25 further comprising:

(C1, 3,

     3) optimizing a (6−

n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

29. The method of claim 21, wherein said step (C1) further comprises:

(C1,

     4) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first said pixel and a second said pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame.

30. The method of claim 29 further comprising:

(C1, 4,

     1) optimizing said 6-vector describing the movement of said image capturing device.

31. The method of claim 30 further comprising:

(C1, 4,

     3) optimizing a (6−

n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

32. The method of claim 29 further comprising:

(C1, 4,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

33. The method of claim 21, wherein said step (C1) further comprises:

(C1,

     5) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first pixel and a second pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame, and by matching at least two pixels having substantially the same intensity in said two sequential frames.

34. The method of claim 33 further comprising:

(C1, 5,

     1) optimizing said 6-vector describing the movement of said image capturing device.

35. The method of claim 33 further comprising:

(C1, 5,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

36. The method of claim 33 further comprising:

(C1, 5,

     3) optimizing a (6−

n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

37. An apparatus for image-tracking comprising:

(A) an image capturing device configured to perform an image-capture of a scene;

and

(B1) a means for performing a rigid global transformation of said set of captured images and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.