A 3D Scene Scanner and a Position and Orientation System

• US 20090323121A1
• Filed: 09/08/2006
• Published: 12/31/2009
• Est. Priority Date: 09/09/2005
• Status: Active Grant

• Current Assignee: Callaghan Innovation
• Original Assignee: Callaghan Innovation
• Inventors: Valkenburg, Robert Jan, Penman, David William, Alwesh, Nawar Sami, Palmer, George Terry, Schoonees, Johann August
• Granted Patent: US 8,625,854 B2
• Time in Patent Office: Days
• US Class Current: 358/1.18
• CPC Class Codes: G01B 11/002 for measuring two or more coordinates
• CPC Class Codes: G01B 11/303 using photoelectric detection means
• CPC Class Codes: G01C 15/002 Active optical surveying means optical plumbing G01C15/105
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/875 for determining attitude
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

A hand-held mobile 3D scanner (10) for scanning a scene. The scanner (10) comprises a range sensor (11) that is arranged to sense the location of surface points in the scene relative to the scanner (10) and generate representative location information, a texture sensor (12) that is arranged to sense the texture of each surface point in the scan of the scene and generate representative texture information, and a position and orientation sensor (13) that is arranged to sense the position and orientation of the scanner (10) during the scan of the scene and generate representative position and orientation information. A control system (14) is also provided that is arranged to receive the information from each of the sensors and generate data representing the scan of the scene.

Claims

1. A hand-held mobile 3D scanner for scanning a scene comprising:

a range sensor that is arranged to sense the location of surface points in the scene relative to the scanner and generate representative location information;

a texture sensor that is arranged to sense the texture of each surface point in the scan of the scene and generate representative texture information;

a position and orientation sensor that is arranged to sense the position and orientation of the scanner during the scan of the scene by interacting with multiple reference targets located about the scene and generate representative position and orientation information; and

a control system that is arranged to receive the information from each of the sensors and generate data representing the scan of the scene.

2. A hand-held mobile 3D scanner according to claim 1 wherein the data generated relates to scanned surface points in the scene and comprises information on the 3D positions of those surface points in space and texture information in relation to those surface points.

3. A hand-held mobile 3D scanner according to claim 2 wherein the data further comprises viewpoint information in relation to the viewpoint from which the surface points were scanned by the scanner.

4. A hand-held mobile 3D scanner according to claim 2 wherein the control system is arranged to generate the texture information for the data based on texture values sensed by the texture sensor from multiple viewpoints during the scan.

5. A hand-held mobile 3D scanner according to claim 1 wherein the control system is arranged to generate a texture model representing the scan of the scene.

6. A hand-held mobile 3D scanner according to claim 1 wherein the data generated by the control system is in the form of rich-3D data.

7. A hand-held mobile 3D scanner according to claim 1 wherein the control system is arranged to generate a 3D substantially photo-realistic representation of the scene from the data.

8. A hand-held mobile 3D scanner according to claim 1 wherein the control system comprises a user interface that is operable by a user to control scanning parameters.

9. A hand-held mobile 3D scanner according to claim 1 wherein the control system comprises an output display that is arranged to generate a progressive representation of the scene as it is being scanned.

10. A hand-held mobile 3D scanner according to claim 1 wherein the control system is arranged to filter out data associated with scanned surface points in the scene that fall outside scanning zones that are selected by the user.

11. A hand-held mobile 3D scanner according to claim 1 wherein the control system is arranged to increase or decrease the resolution of the range and texture sensors for particular scanning zones that are selected by the user.

12. A hand-held mobile 3D scanner according to claim 1 wherein the range sensor comprises any one of the following:

a light detection and ranging (LIDAR) device, triangulation-based device, or a non-scanning time-of-flight camera device.

13. A hand-held mobile 3D scanner according to claim 1 wherein the texture sensor comprises a colour camera that is arranged to capture digital images of the scene, each digital image comprising an array of pixels and each pixel or group of pixels corresponding to a surface point in the scan of the scene from which texture information can be extracted.

14. A hand-held mobile 3D scanner according to claim 1 wherein the texture sensor comprises a multi-spectral laser imager that is arranged to sense texture information relating to the scanned surface points of the scene.

15. A hand-held mobile 3D scanner according to claim 1 wherein the position and orientation sensor comprises an optical tracking device that senses the position and orientation of the scanner by tracking visible reference targets located about the scene.

16. A hand-held mobile 3D scanner according to claim 15 wherein the optical tracking device comprises one or more direction sensors that are arranged to detect visible reference targets and generate direction information relating to the direction of the visible reference targets relative to the scanner, the optical tracking device processing the direction information to determine the position and orientation of the scanner.

17. A hand-held mobile 3D scanner according to claim 16 wherein the direction sensors are optical sensors that are each arranged to view outwardly relative to the scanner to provide direction information relating to any visible reference targets.

18. A hand-held mobile 3D scanner according to claim 15 wherein the position and orientation sensor additionally comprises an inertial sensor that is arranged to sense the position and orientation of the scanner and provide representative position and orientation information if the optical tracking device experiences target dropout.

19. A portable 3D scanning system for scanning a scene comprising:

a hand-held mobile scanner comprising;

a range sensor that is arranged to sense the location of surface points in the scene relative to the scanner and generate representative location information;

a texture sensor that is arranged to sense the texture of each surface point in the scan of the scene and generate representative texture information; and

a position and orientation sensor that is arranged to sense the position and orientation of the scanner during the scan of the scene and generate representative position and orientation information;

multiple reference targets for placing randomly about the scene, the position and orientation sensor interacting with detectable reference targets to sense the position and orientation of the scanner; and

a control system that is arranged to control the scanner and its sensors and the reference targets, receive the information from each of the sensors, and generate data representing the scan of the scene.

20. A portable 3D scanning system according to claim 19 wherein the data generated relates to scanned surface points in the scene and comprises information on the 3D positions of those surface points in space and texture information in relation to those surface points.

21. A portable 3D scanning system according to claim 20 wherein the data further comprises viewpoint information in relation to the viewpoint from which the surface points were scanned by the scanner.

22. A portable 3D scanning system according to claim 20 wherein the control system is arranged to generate the texture information for the data based on texture values sensed by the texture sensor from multiple viewpoints during the scan.

23. A portable 3D scanning system according to claim 19 wherein the control system is arranged to generate a texture model representing the scan of the scene.

24. A portable 3D scanning system according to claim 19 wherein the data generated by the control system is in the form of rich-3D data.

25. A portable 3D scanning system according to claim 19 wherein the control system is arranged to generate a 3D substantially photo-realistic representation of the scene from the data.

26. A portable 3D scanning system according to claim 19 wherein the control system comprises a user interface that is operable by a user to control scanning parameters.

27. A portable 3D scanning system according to claim 19 wherein the control system comprises an associated output display that is arranged to generate a progressive representation of the scene as it is being scanned.

28. A portable 3D scanning system according to claim 19 wherein the control system is arranged to filter out data associated with scanned surface points in the scene that fall outside scanning zones that are selected by the user.

29. A portable 3D scanning system according to claim 19 wherein the control system is arranged to increase or decrease the resolution of the range and texture sensors for particular scanning zones that are selected by the user.

30. A portable 3D scanning system according to claim 19 wherein the range sensor comprises any one of the following:

a light detection and ranging LIDAR) device, triangulation-based device, or a non-scanning time-of-flight camera device.

31. A portable 3D scanning system according to claim 19 wherein the texture sensor comprises a colour camera that is arranged to capture digital images of the scene, each digital image comprising an array of pixels and each pixel or group of pixels corresponding to a surface point in the scan of the scene from which texture information can be extracted.

32. A portable 3D scanning system according to claim 19 wherein the texture sensor comprises a multi-spectral laser imager that is arranged to sense texture information relating to the scanned surface points of the scene.

33. A portable 3D scanning system according to claim 19 wherein the position and orientation sensor comprises an optical tracking device that senses the position and orientation of the scanner by tracking visible reference targets located about the scene.

34. A portable 3D scanning system according to claim 33 wherein the optical tracking device comprises one or more direction sensors that are arranged to detect visible reference targets and generate direction information relating to the direction of the visible reference targets relative to the scanner, the optical tracking device processing the direction information to determine the position and orientation of the scanner.

35. A portable 3D scanning system according to claim 34 wherein the direction sensors are optical sensors that are each arranged to view outwardly relative to the scanner to provide direction information relating to any visible reference targets.

36. A portable 3D scanning system according to claim 33 wherein the position and orientation sensor additionally comprises an inertial sensor that is arranged to sense the position and orientation of the scanner and provide representative position and orientation information if the optical tracking device experiences target dropout.

37. A method of scanning a scene comprising the steps of:

operating a hand-held mobile scanner to scan the scene, the scanner comprising;

a range sensor that is arranged to sense the shape of the object(s) in the scene on a surface point-by-point basis and generate representative shape information;

a texture sensor that is arranged to sense the texture of the object(s) in the scene on a surface point-by-point basis and generate representative texture information; and

a position and orientation sensor that is arranged to sense the position and orientation of the scanner in a local reference frame by interacting with multiple reference targets located about the scene and generate representative position and orientation information;

obtaining the shape, texture, and position and orientation information from the sensors;

processing the shape, texture, and position and orientation information; and

generating data representing the scan of the scene.

38. A method of scanning a scene according to claim 37 wherein the step of processing the shape, texture, and position and orientation information comprises extracting information about each surface point of the surfaces and objects in the scan on a point-by-point basis by computing the 3D position of each surface point in the local reference frame from the shape information and the position and orientation information;

generating the texture information around the region of each surface point from the texture information from the texture sensor and the position and orientation information; and

extracting the viewpoint from which the surface point was scanned by the scanner from the position and orientation information.

39. A method of scanning a scene according to claim 37 wherein the step of generating data representing the scene comprises constructing rich-3D data.

40. A method of scanning a scene according to claim 37 wherein the method further comprises the step of processing the data to generate a 3D substantially photo-realistic representation of the scene for display.

41. A method of scanning a scene according to claim 37 wherein the method further comprises the step of placing reference targets about the scene and operating the position and orientation sensor of the scanner to interact with detectable reference targets to sense the position and orientation of the scanner in the local reference frame.

42. A method of scanning a scene according to claim 37 wherein the step of operating the hand-held mobile scanner to scan the scene comprises scanning the surfaces and objects of the scene from multiple viewpoints.

43. A method of scanning a scene according to claim 37 wherein the step of operating the hand-held mobile scanner to scan the scene comprises first initially setting scanning zones within the scene such that processing step discards any information relating to surface points of objects in the scene that fall outside the scanning zones.

44-48. -48. (canceled)

49. A position and orientation system for sensing the position and orientation of a mobile object that is moveable in an environment comprising:

multiple reference targets locatable in random positions within the environment to define a local reference frame;

an optical tracking device mounted to the mobile object comprising one or more direction sensors that are arranged to detect visible reference targets and generate direction information relating to the direction of the visible reference targets relative to the optical tracking device; and

a control system arranged to operate the optical tracking device, receive the direction information, and process the direction information to initially determine the 3D positions of the reference targets and then generate position and orientation information relating to the position and orientation of the mobile object in the local reference frame during movement of the mobile object.

50. A position and orientation system according to claim 49 wherein the direction sensors of the optical tracking device are optical sensors that are each arranged to view outwardly relative to the scanner to provide direction information relating to any visible reference targets.

51. A position and orientation system according to claim 50 wherein the optical sensors comprise an arrangement of cameras that view outwardly relative to the mobile object to provide direction information relating to any visible reference targets.

52. A position and orientation system according to claim 49 wherein the one or more direction sensors of the optical tracking device are arranged to form an omnidirectional direction sensor.

53. A position and orientation system according to claim 49 wherein the reference targets each comprise a switchable light source that is arranged to emit light for sensing by the direction sensors of the optical tracking device.

54. A position and orientation system according to claim 49 wherein the control system is arranged to auto-calibrate in operation by automatically determining the 3D position of visible reference targets in the environment by processing direction information from the optical tracking device.

55. A position and orientation system according to claim 54 wherein the control system is arranged to auto-calibrate at start-up and continue to periodically auto-calibrate during operation to register the movement, removal, and addition of reference targets to the environment.

56. A position and orientation system according to claim 54 wherein the control system is arranged to provide the user with feedback on the quality of the distribution of the reference targets within the environment after auto-calibration has taken place, the distribution of the reference targets affecting the accuracy of the position and orientation information generated.

57. A position and orientation system according to claim 49 wherein the control system comprises a user interface that is operable by a user to control the system and an associated output display for presenting the position and orientation information.

58. A position and orientation system according to claim 49 further comprising an inertial sensor that is arranged to sense the position and orientation of the mobile object and provide representative position and orientation information if the optical tracking device experiences target dropout.

59. A method of sensing the position and orientation of a mobile object that is moveable in an environment comprising the steps of:

placing multiple reference targets at random positions within the environment to define a local reference frame;

mounting an optical tracking device to the mobile object, the optical tracking device comprising one or more direction sensors that are arranged to detect visible reference targets and generate direction information relating to the direction of the visible reference targets relative to the optical tracking device;

operating the optical tracking device to track and sense visible reference targets as it moves with the mobile object in the environment and generate direction information; and

processing the direction information to initially determine the 3D positions of the reference targets and then generate position and orientation information relating to the position and orientation of the mobile object in the local reference frame during movement of the mobile object.

60. A method of sensing the position and orientation of a mobile object according to claim 59 wherein the step operating the optical tracking device to track and sense visible reference targets further comprises the step of operating the optical tracking device to initially determine the 3D position of visible reference targets in the environment by auto-calibrating from the direction information.

61. A method of sensing the position and orientation of a mobile object according to claim 60 wherein the step of auto-calibrating comprises:

moving the mobile object into N locations in the environment and sensing direction information for visible reference targets at each location;

calculating initial estimates of the position and orientation of the mobile object at the N locations;

calculating accurate estimates of the position and orientation of the mobile object; and

reconstructing the reference target 3D positions by triangulation using the direction information and the accurate estimates of the position and orientation of the mobile object.

62. A method of sensing the position and orientation of a mobile object according to claim 61 wherein the step of calculating initial estimates comprises executing a closed form algorithm.

63. A method of sensing the position and orientation of a mobile object according to claim 61 wherein the step of calculating accurate estimates comprises executing a non-linear minimisation algorithm.

64. A method of sensing the position and orientation of a mobile object according to claim 60 wherein the step of auto-calibrating occurs periodically to register the movement, removal, and addition of reference targets.

65. A method of sensing the position and orientation of a mobile object according to claim 60 further comprising the step of feeding back information on the quality of the distribution of the reference targets within the environment after auto-calibration step has taken place, the distribution of the reference targets affecting the accuracy of the position and orientation information generated.

66. A method of sensing the position and orientation of a mobile object according to claim 59 wherein the step of processing the direction information to generate position and orientation information comprises:

calculating an initial estimate of the position and orientation of the mobile object using a boot-strapping process;

predicting the current position and orientation of the mobile object based on previous position and orientation estimate;

associating the sensed direction information with specific individual reference target 3D positions; and

updating the current position and orientation prediction using the individual reference target 3D positions and direction information.

67. A method of sensing the position and orientation of a mobile object according to claim 66 wherein the step of predicting the current position and orientation of the mobile object comprises extrapolating from the previous position and orientation estimate.

68. A method of sensing the position and orientation of a mobile object according to claim 66 wherein the step of updating the current position and orientation predication comprising executing a non-linear algorithm.

69. A method of sensing the position and orientation of a mobile object according to claim 59 further comprising mounting an inertial sensor to the mobile object and operating the inertial sensor to sense the position and orientation of the mobile object and generate representative position and orientation information if the optical tracking device experiences target dropout.