IMAGE-BASED SURFACE TRACKING

• US 20120195466A1
• Filed: 07/06/2010
• Published: 08/02/2012
• Est. Priority Date: 07/07/2009
• Status: Active Grant

• Current Assignee: Trimble Navigation Limited (Trimble Inc.)
• Original Assignee: Trimble Navigation Limited (Trimble Inc.)
• Inventors: Teng, Hongbo, Best, Gregory C., Wang, Sy Bor
• Granted Patent: US 9,224,208 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of image-tracking by using an image capturing device (12). The method comprises: performing an image-capture of a scene (54) by using an image capturing device; and tracking movement (62) of the image capturing device (12) by analyzing a set of images by using an image processing algorithm (64).

Claims

1. A method of image-based positioning comprising:

(A) performing an image-capture of a scene by using an image capturing device;

and

(B) determining the position coordinates of said image capturing device by analyzing a set of images.

2. The method of claim 1, wherein said step (A) further comprises:

(A1) performing said image-capture of said scene in real time by using said image capturing device;

wherein said image capturing device is selected from the group consisting of;

{a digital camera;

a digital video camera;

a digital camcorder;

a stereo digital camera;

a stereo video camera;

a motion picture camera; and

a television camera}.

3. The method of claim 1, wherein said step (A) further comprises:

(A2) performing image-recording of said scene by using said image capturing device.

4. The method of claim 1, wherein said step (A) further comprises:

(A3) obtaining a set of depth data of said scene by pre-surveying said scene.

5. The method of claim 4, wherein said step (A3) further comprises:

(A3,

     1) obtaining said set of depth data of said scene by using a surveying instrument selected from the group consisting of;

{a Global Navigation Satellite System (GNSS) surveying system;

a laser plane system; and

a theodolite}.

6. The method of claim 1, wherein said step (B) further comprises:

(B1) performing a rigid global transformation of said set of captured image data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

7. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     1) using a mapping function of said image capturing device to map for at least one i-th frame 3D coordinates of at least one object point in the reference system attached to said image capturing device, into 2D pixel coordinates of an image point on said image capturing device sensor plane in a 2D reference coordinate system;

i being an integer.

8. The method of claim 7, wherein said step (B1, 1) further comprises:

(B1, 1,

     1) selecting said mapping function of said image capturing device from the group consisting of;

{a mapping function of regular rectilinear lenses;

a mapping function of fisheye lenses; and

a mapping function of custom-calibrated lenses}.

9. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     2) obtaining a reverse of said mapping function;

wherein said reverse of said mapping function is configured to map, for at least one i-th frame, 2D pixel coordinates of at least one image point on said image capturing device sensor plane in a 2D reference coordinate system, into 3D coordinates of at least one object point in the reference system attached to said image capturing device.

10. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     3) obtaining 3D depth coordinates of said at least one object point by using said pre-surveyed data of said scene.

11. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     4) extracting a 6-vector describing said movement of said image capturing device from a relationship between two sequential frames;

i-th frame and k-th frame, wherein at least one object point produces at least two pixels of substantially the same intensity in said two sequential frames.

12. The method of claim 11 further comprising:

(B1, 4,

     1) optimizing said 6-vector describing the movement of said image capturing device.

13. The method of claim 11 further comprising:

(B1, 4,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

14. The method of claim 11 further comprising:

(B1, 4,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

15. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     5) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first said pixel and a second said pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame.

16. The method of claim 15 further comprising:

(B1, 5,

     1) optimizing said 6-vector describing the movement of said image capturing device.

17. The method of claim 15 further comprising:

(B1, 5,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

18. The method of claim 15 further comprising:

(B1, 5,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

19. The method of claim 6, wherein said step (B1) further comprises:

(B1,

     6) extracting a 6-vector describing said movement of said image capturing device from a relationship between two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first pixel and a second pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame, and by matching at least two pixels having substantially the same intensity in said two sequential frames.

20. The method of claim 19 further comprising:

(B1, 6,

     1) optimizing said 6-vector describing the movement of said image capturing device.

21. The method of claim 19 further comprising:

(B1, 6,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

22. The method of claim 19 further comprising:

(B1, 6,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

23. A method of image-tracking comprising:

(A) performing an image-capture of a scene by using an image capturing device;

(B) obtaining a set of scene depth data by using a range measurement device selected from the group consisting of;

{a point laser beam;

a sonar;

a radar;

a laser scanner; and

a depth camera};

and

(C) tracking movement of said image capturing device by analyzing said set of images.

24. The method of claim 23, wherein said step (A) further comprises:

(A1) performing said image-capture of said scene in real time by using said image capturing device.

25. The method of claim 23, wherein said step (A) further comprises:

(A2) performing an image-recording of said scene by using said image capturing device.

26. The method of claim 23, wherein said step (B) further comprises:

(B1) determining said set of scene depth data in a image capturing device—

attached 3D reference system by using a K-point range measurement system attached to said image capturing device, wherein said set of scene depth data comprises an integer K-point set of 3D depth measurements of said scene; and

wherein a 3D depth coordinate of at least one object point associated with a 2D image point of said 3D object point is obtained by interpolation of said K-point set of 3D depth measurements of said scene to obtain an optimum depth measurement for at least one image point of at least one said object point;

and wherein said integer K is substantially less than number of pixels in said frame.

27. The method of claim 23, wherein said step (B) further comprises:

(B2) determining said depth of said object point directly for at least one image point of said object point by using an M-point range measurement system attached to said image capturing device, wherein said integer number M of depth measurements of said scene is substantially equal to the number of pixels in said frame.

28. The method of claim 23, wherein said step (B) further comprises:

(B3) determining said set of scene depth data in a image capturing device—

attached 3D reference system by using a feature-point range measurement system attached to said image capturing device, wherein said scene comprises a set of integer K feature object points;

wherein said feature-point range measurement system obtains a K set of 3D depth measurements of said K feature object points in said scene;

and wherein a 3D depth coordinate of at least one object point associated with a 2D image point of said 3D object point is obtained by interpolation of said K-point set of 3D depth measurements of said scene to obtain an optimum depth measurement for at least one image point of at least one said object point;

and wherein said integer K is substantially less than the number of pixels in said frame.

29. The method of claim 23, wherein said step (C) further comprises:

(C1) performing a rigid global transformation of said set of captured images data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

30. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     1) using mapping function to map, for at least one i-th frame, 3D coordinates of at least one object point in the reference system attached to said image capturing device, into 2D pixel coordinates of an image point on said image capturing device sensor plane in a 2D reference coordinate system.

31. The method of claim 30, wherein said step (C1, 1) further comprises:

(C1, 1,

     1) selecting said mapping function from the group consisting of;

{a mapping function of regular rectilinear lenses;

a mapping function of fisheye lenses; and

a mapping function of custom-calibrated lenses}.

32. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     2) obtaining a reverse of said mapping function;

said reverse of said mapping function is configured to map, for at least one i-th frame 2D pixel coordinates of at least one image point on said image capturing device sensor plane in a 2D reference coordinate system, into 3D coordinates of at least one object point in the reference system attached to said image capturing device.

33. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     3) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame, wherein at least one point in said scene produces at least two pixels of substantially the same intensity in said two sequential frames.

34. The method of claim 33 further comprising:

(C1, 3,

     1) optimizing said 6-vector describing the movement of said image capturing device.

35. The method of claim 33 further comprising:

(C1, 3,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

36. The method of claim 33 further comprising:

(C1, 3,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

37. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     4) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first said pixel and a second said pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame.

38. The method of claim 37 further comprising:

(C1, 4,

     1) optimizing said 6-vector describing the movement of said image capturing device.

39. The method of claim 37 further comprising:

(C1, 4,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

40. The method of claim 38 further comprising:

(C1, 4,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

41. The method of claim 29, wherein said step (C1) further comprises:

(C1,

     5) extracting a 6-vector describing said movement of said image capturing device from a relationship between said two sequential frames;

i-th frame and k-th frame by matching the depth data for at least two pixels;

a first pixel and a second pixel;

wherein said first pixel is located in said i-th frame, and wherein said second pixel is located in said k-th frame, and by matching at least two pixels having substantially the same intensity in said two sequential frames.

42. The method of claim 41 further comprising:

(C1, 5,

     1) optimizing said 6-vector describing the movement of said image capturing device.

43. The method of claim 41 further comprising:

(C1, 5,

     2) optimizing said 6-vector describing the movement of said image capturing device by solving a nonlinear least-squares problem for said relationship between said two sequential frames.

44. The method of claim 41 further comprising:

(C1, 5,

     3) optimizing a (6-n)-vector describing an n-dimensionally restricted movement of said image capturing device by optimizing said relationship between said two sequential frames;

wherein integer n is less than 6.

45. An apparatus for image-tracking comprising:

(A) an image capturing device configured to perform an image-capture of a scene;

and

(B) a means for tracking movement of said image capturing device by analyzing a set of images.

46. The apparatus of claim 45 further comprising:

(C) a means for obtaining a set of depth data of said scene.

47. The apparatus of claim 45, wherein said means (B) further comprises:

(B1) a means for performing a rigid global transformation of said set of captured images and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device.

48. The apparatus of claim 46;

wherein said means (C) further comprises;

(C1) a surveying instrument selected from the group consisting of;

{a Global Navigation Satellite System (GNSS) surveying system;

a laser plane system; and

a theodolite}.

49. The apparatus of claim 46;

wherein said means (C) further comprises;

(C2) a range measurement device selected from the group consisting of {a point laser beam;

a sonar;

a radar;

a laser scanner; and

a depth camera}.

50. An article comprising:

a storage medium having stored thereon instructions that, when executed, result in a computing platform having the capability to perform an image-tracking of an image capturing device by executing an algorithm further comprising;

(A) storing a set of image-captured data of a scene obtained by said image capturing device;

and

(B) tracking movement of said image capturing device by analyzing said set of image-captured data.

51. A method of image-based positioning comprising:

(A) performing image-recording of a scene by using an image capturing device.

(B) performing an initial image analysis to determine suitability of recorded images

(C) obtaining a set of depth data of said scene by pre-surveying said scene;

(D) using scene understanding techniques to discard unsuitable regions of

images of broadly defined features for tracking, wherein said broadly defined features are selected from the group consisting of;

points;

representations of regions; and

representations of contours;

and

(E) tracking movement of said image capturing device by analyzing a set of suitably recorded images.

52. The method of claim 51, wherein said step (B) further comprises:

(B1) performing an initial image assessment analysis to find out whether a set of recorded images is suitable for surface tracking.

53. The method of claim 52, wherein said step (B1) further comprises:

(B1,1) extracting of at least three attributes from at least one recorded image.

54. The method of claim 52, wherein said step (B1) further comprises:

(B1,2) checking saturation quality of at least one recorded image to check if an image consists substantially of a single Red-Green-Blue value.

55. The method of claim 52, wherein said step (B1) further comprises:

(B1,3) checking the texture quality of at least one recorded image.

56. The method of claim 52, wherein said step (B1) further comprises:

(B1,4) checking an image contrast of at least one recorded image.

57. The method of claim 51, wherein said step (D) further comprises:

(D1) performing object recognition.

58. The method of claim 57, wherein said step (D1) further comprises:

(D1,1) performing extraction of sparse features from a recorded image.

59. The method of claim 57, wherein said step (D1) further comprises:

(D1,2) performing clustering neighboring features together.

60. The method of claim 57, wherein said step (D1) further comprises:

(D1,3) inferring an object category from at least one cluster of neighboring features.

61. The method of claim 60, wherein said step (D1,3) further comprises:

(D1,3,1), determining the suitability of the inferred object category for surface tracking.

62. The method of claim 51, wherein said step (E) further comprises:

(E1) performing a rigid global transformation of said set of captured suitable image data and said set of scene depth data into a set of 6-coordinate data;

wherein said set of 6-coordinate data represents movement of said image capturing device; and

wherein said set of 6-coordinate data represents a surface tracking solution.

63. The method of claim 62, wherein said step (E1) further comprises:

(E1,1) using a set of broadly defined-features to initialize said surface tracking solution.

64. The method of claim 63, wherein said step (E1,1) further comprises:

(E1,1,1) detecting a set of broadly defined features;

(E1,1,2) establishing correspondences between said set of broadly defined features and at least two selected frames;

(E1,1,3) estimating homography between said at least selected two frames using parameters of said RANSAC mathematical model;

(E1,1,4) deriving an initial pose of said image capturing device from said homography between said at least selected two frames;

wherein said pose of said image-capturing device comprises position coordinates of said image-capturing device and a set of angular coordinates of said image-capturing device;

and

(E1, 1,

     5) using said derived initial pose of said image capturing device as an initial solution to said surface tracking solution.

65. The method of claim 62, wherein said step (E1) further comprises:

(E1,2) using a set of broadly defined-features on at least one tracking surface for a strict two-dimensional (strict_—

2D) surface tracking solution.

66. The method of claim 65, wherein said step (E1,2) further comprises:

(E1,2,1) detecting a set of broadly defined features on a single tracking surface;

(E1,2,2) selecting a set of coplanar broadly defined features by using parameters of said RANSAC mathematical model;

(E1,2,3) establishing correspondences between said set of coplanar broadly defined features and at least two selected frames;

(E1,2,4) deriving an initial pose of said image capturing device from said homography between said at least selected two frames;

(E1, 2,

     5) using said derived initial pose of said image capturing device as an initial solution to said strict two-dimensional (strict_—

2D) surface tracking solution;

and

(E1,2,6) grouping said set of coplanar features and using an area around said group of coplanar features as an input to said strict_—

2D surface tracking solution.

67. The method of claim 62, wherein said step (E1) further comprises:

(E1,3) using a set of coplanar broadly defined-features on a plurality of two-dimensional (2D) tracking surfaces for a substantially two-dimensional (sub_—

2D) surface tracking.

68. The method of claim 67, wherein said step (E1, 3) further comprises:

(E1,3,1) detecting a set of broadly defined features on said plurality of tracking surfaces;

(E1,3,2) selecting a set of coplanar broadly defined features by using parameters of said RANSAC mathematical model;

(E1,3,3) establishing correspondences between said set of coplanar broadly defined features and at least two selected frames;

(E1,3,4) deriving an initial pose of said image capturing device from said homography between said at least selected two frames;

(E1, 3,

     5) using said derived initial pose of said image capturing device as an initial solution to a substantially two-dimensional (sub_—

2D) surface tracking solution;

and

(E1,3,6) selecting a local area around each said selected coplanar broadly defined feature, grouping a plurality of said selected local areas into a two-dimensional (2D) global area and using said 2D global area as an input to said sub_—

2D surface tracking solution.

69. The method of claim 62, wherein said step (E1) further comprises:

(E1,4) using a set of broadly defined-features extracted from a three dimensional (3D) for a substantially three-dimensional (sub_—

3D) surface tracking.

70. The method of claim 69, wherein said step (E1, 4) further comprises:

(E1,4,1) detecting a set of broadly defined features on said 3D surface;

(E1,4,2) establishing correspondences between said set of broadly defined features and at least two selected frames;

(E1,4,2) estimating homography between said at least two selected frames;

(E1,4,4) deriving an initial pose of said image capturing device from said homography between said at least selected two frames;

(E1,4,5) using said derived initial pose of said image capturing device as an initial solution to said substantially three-dimensional (sub_—

3D) surface tracking solution;

and

(E1,4,6) selecting a local area around each said selected broadly defined feature, grouping a plurality of said selected local areas into a three-dimensional (3D) global area and using said 3D global area as an input to said sub_—

3D surface tracking solution.