Image-based tracking

• US 20110007939A1
• Filed: 07/07/2009
• Published: 01/13/2011
• Est. Priority Date: 07/07/2009
• Status: Active Grant

• Current Assignee: Trimble Inc.
• Original Assignee: Trimble Navigation Limited (Trimble Inc.)
• Inventors: Wang, Sy Bor, Best, Gregory C., Teng, Hongbo
• Granted Patent: US 8,229,166 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of image-tracking by using an image capturing device. The method comprises: performing an image-capture of a scene by using an image capturing device; and tracking movement of the image capturing device by analyzing a set of images by using an image processing algorithm.

Claims

1. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

and

(B) tracking movement of said image capturing device by analyzing a set of images.

2. The method of claim 1, wherein said step (A) further comprises:

(A1) performing said image-capture of said scene in real time by using said image capturing device;

wherein said image capturing device is selected from the group consisting of;

{a digital camera;

a digital video camera;

a digital camcorder;

a stereo digital camera;

a stereo video camera;

a motion picture camera; and

a television camera}.

3. The method of claim 1, wherein said step (A) further comprises:

(A2) performing image-recording of said scene by using said image capturing device.

4. The method of claim 1, wherein said step (A) further comprises:

(A3) obtaining a set of depth data of said scene by pre-surveying said scene.

5. The method of claim 4, wherein said step (A3) further comprises:

(A3,

     1) obtaining said set of depth data of said scene by using a surveying instrument selected from the group consisting of;

{a Global Navigation Satellite System (GNSS) surveying system;

a laser plane system; and

a theodolite}.

6. The method of claim 1, wherein said step (B) further comprises:

(B1) performing a rigid global transformation of said set of captured image data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

7. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     1) using a mapping function of said image capturing device to map for at least one i-th frame 3D coordinates of at least one object point in the reference system attached to said image capturing device, into 2D pixel coordinates of an image point on said image capturing device sensor plane in a 2D reference coordinate system;

i being an integer.

8. The method of claim 7, wherein said step (B1, 1) further comprises:

(B1, 1,

     1) selecting said mapping function of said image capturing device from the group consisting of;

{a mapping function of regular rectilinear lenses;

a mapping function of fisheye lenses; and

a mapping function of custom-calibrated lenses}.

9. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     2) obtaining a reverse of said mapping function;

wherein said reverse of said mapping function is configured to map, for at least one i-th frame, 2D pixel coordinates of at least one image point on said image capturing device sensor plane in a 2D reference coordinate system, into 3D coordinates of at least one object point in the reference system attached to said image capturing device.

10. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     3) obtaining 3D depth coordinates of said at least one object point by using said pre-surveyed data of said scene.

11. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     4) extracting a 6-vector describing said movement of said image capturing device from a relationship between two sequential frames;

i-th frame and k-th frame, wherein at least one object point produces at least two pixels of substantially the same intensity in said two sequential frames.

12. The method of claim 11 further comprising:

(B1, 4,

     1) optimizing said 6-vector describing the movement of said image capturing device.

13. The method of claim 11 further comprising:

(B1, 4,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

14. The method of claim 11 further comprising:

(B1, 4,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

15. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     5) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first said pixel and a second said pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame.

16. The method of claim 15 further comprising:

(B1, 5,

     1) optimizing said 6-vector describing the movement of said image capturing device.

17. The method of claim 15 further comprising:

(B1, 5,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

18. The method of claim 15 further comprising:

(B1, 5,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

19. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     6) extracting a 6-vector describing said movement of said image capturing device from a relationship between two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first pixel and a second pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame, and by matching at least two pixels having substantially the same intensity in said two sequential frames.

20. The method of claim 19 further comprising:

(B1, 6,

     1) optimizing said 6-vector describing the movement of said image capturing device.

21. The method of claim 19 further comprising:

(B1, 6,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

22. The method of claim 19 further comprising:

(B1, 6,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

23. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B) obtaining a set of scene depth data by using a range measurement device selected from the group consisting of;

{a point laser beam;

a sonar;

a radar;

a laser scanner; and

a depth camera};

and

(C) tracking movement of said image capturing device by analyzing said set of images.

24. The method of claim 23, wherein said step (A) further comprises:

(A1) performing said image-capture of said scene in real time by using said image capturing device.

25. The method of claim 23, wherein said step (A) further comprises:

(A2) performing an image-recording of said scene by using said image capturing device.

26. The method of claim 23, wherein said step (B) further comprises:

(B1) determining said set of scene depth data in a image capturing device-attached 3D reference system by using a K-point range measurement system attached to said image capturing device, wherein said set of scene depth data comprises an integer K-point set of 3D depth measurements of said scene; and

wherein a 3D depth coordinate of at least one object point associated with a 2D image point of said 3D object point is obtained by interpolation of said K-point set of 3D depth measurements of said scene to obtain an optimum depth measurement for at least one image point of at least one said object point; and

wherein said integer K is substantially less than number of pixels in said frame.

27. The method of claim 23, wherein said step (B) further comprises:

(B2) determining said depth of said object point directly for at least one image point of said object point by using an M-point range measurement system attached to said image capturing device, wherein said integer number M of depth measurements of said scene is substantially equal to the number of pixels in said frame.

28. The method of claim 23, wherein said step (B) further comprises:

(B3) determining said set of scene depth data in a image capturing device-attached 3D reference system by using a feature-point range measurement system attached to said image capturing device, wherein said scene comprises a set of integer K feature object points;

wherein said feature-point range measurement system obtains a K set of 3D depth measurements of said K feature object points in said scene;

and wherein a 3D depth coordinate of at least one object point associated with a 2D image point of said 3D object point is obtained by interpolation of said K-point set of 3D depth measurements of said scene to obtain an optimum depth measurement for at least one image point of at least one said object point;

and wherein said integer K is substantially less than the number of pixels in said frame.

29. The method of claim 23, wherein said step (C) further comprises:

(C1) performing a rigid global transformation of said set of captured images data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

30. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     1) using mapping function to map, for at least one i-th frame, 3D coordinates of at least one object point in the reference system attached to said image capturing device, into 2D pixel coordinates of an image point on said image capturing device sensor plane in a 2D reference coordinate system.

31. The method of claim 30, wherein said step (C1, 1) further comprises:

(C1, 1,

     1) selecting said mapping function from the group consisting of;

{a mapping function of regular rectilinear lenses;

a mapping function of fisheye lenses; and

a mapping function of custom-calibrated lenses}.

32. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     2) obtaining a reverse of said mapping function;

said reverse of said mapping function is configured to map, for at least one i-th frame 2D pixel coordinates of at least one image point on said image capturing device sensor plane in a 2D reference coordinate system, into 3D coordinates of at least one object point in the reference system attached to said image capturing device.

33. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     3) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame, wherein at least one point in said scene produces at least two pixels of substantially the same intensity in said two sequential frames.

34. The method of claim 33 further comprising:

(C1, 3,

     1) optimizing said 6-vector describing the movement of said image capturing device.

35. The method of claim 33 further comprising:

(C1, 3,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

36. The method of claim 33 further comprising:

(C1, 3,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

37. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     4) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first said pixel and a second said pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame.

38. The method of claim 37 further comprising:

(C1, 4,

     1) optimizing said 6-vector describing the movement of said image capturing device.

39. The method of claim 37 further comprising:

(C1, 4,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

40. The method of claim 38 further comprising:

(C1, 4,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

41. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     5) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first pixel and a second pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame, and by matching at least two pixels having substantially the same intensity in said two sequential frames.

42. The method of claim 41 further comprising:

(C1, 5,

     1) optimizing said 6-vector describing the movement of said image capturing device.

43. The method of claim 41 further comprising:

(C1, 5,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

44. The method of claim 41 further comprising:

(C1, 5,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

45. An apparatus for image-tracking comprising:

(A) an image capturing device configured to perform an image-capture of a scene;

and

(B) a means for tracking movement of said image capturing device by analyzing a set of images.

46. The apparatus of claim 45 further comprising:

(C) a means for obtaining a set of depth data of said scene.

47. The apparatus of claim 45, wherein said means (B) further comprises:

(B1) a means for performing a rigid global transformation of said set of captured images and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

48. The apparatus of claim 46;

wherein said means (C) further comprises;

(C1) a surveying instrument selected from the group consisting of;

{a Global Navigation Satellite System (GNSS) surveying system;

a laser plane system; and

a theodolite}.

49. The apparatus of claim 46;

wherein said means (C) further comprises;

(C2) a range measurement device selected from the group consisting of;

{a point laser beam;

a sonar;

a radar;

a laser scanner; and

a depth camera}.

50. An article comprising:

a storage medium having stored thereon instructions that, when executed, result in a computing platform having the capability to perform an image-tracking of an image capturing device by executing an algorithm further comprising;

(A) storing a set of image-captured data of a scene obtained by said image capturing device;

and

(B) tracking movement of said image capturing device by analyzing said set of image-captured data.