MOTION COMPENSATION IN A THREE DIMENSIONAL SCAN

• US 20140120493A1
• Filed: 11/01/2012
• Published: 05/01/2014
• Est. Priority Date: 11/01/2012
• Status: Active Grant

• Current Assignee: Align Technology Incorporated
• Original Assignee: Align Technology Incorporated
• Inventors: Levin, Adi
• Granted Patent: US 8,948,482 B2
• Time in Patent Office: Days
• US Class Current: 433/29
• CPC Class Codes: A61B 1/24 for the mouth, i.e. stomatoscopes, e.g. with tongue depressors tongue depressors per se A61B13/00; Instruments for opening or keeping open the mouth combined with saliva removers A61C17/00; mouth openers for animals A61D15/00
• CPC Class Codes: A61B 6/512 Intraoral means
• CPC Class Codes: A61C 9/0053 Optical means or methods, e.g. scanning the teeth by a laser or light beam
• CPC Class Codes: A61C 9/0066 Depth determination through adaptive focusing
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20048 Transform domain processing
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/30036 Dental; Teeth
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/38 Registration of image sequences

Abstract

The present disclosure provides computing device implemented methods, computing device readable media, and systems for motion compensation in a three dimensional scan. Motion compensation can include receiving three-dimensional (3D) scans of a dentition, estimating a motion trajectory from one scan to another, and calculating a corrected scan by compensating for the motion trajectory. Estimating the motion trajectory can include one or more of: registering a scan to another scan and determining whether an amount of movement between the scans is within a registration threshold; determining an optical flow based on local motion between consecutive two-dimensional (2D) images taken during the scan, estimating and improving a motion trajectory of a point in the scan using the optical flow; and estimating an amount of motion of a 3D scanner during the scan as a rigid body transformation based on input from a position tracking device.

Claims

1. A system, comprising:

a scanning module including an intraoral three dimensional (3D) scanner; and

a processing module coupled to the scanning module to;

receive a plurality of three-dimensional (3D) scans of a dentition;

estimate a motion trajectory from a particular one of the plurality of scans to a subsequent scan, wherein the estimation includes one or more of;

registering the particular scan to at least one of a previous scan and the subsequent scan, determining whether an amount of movement between the previous scan, the particular scan, and the subsequent scan is within a registration threshold, and estimating the motion trajectory based on the registration;

using data from a plurality of two-dimensional (2D) images of the dentition taken during the particular scan of the dentition, determining an optical flow based on local motion between consecutive images of the plurality of 2D images, estimating a motion trajectory of a point in the particular scan, improving the estimated motion trajectory using the optical flow to estimate a correctness of the estimated motion trajectory; and

estimating an amount of motion of a 3D scanner during the particular scan as a rigid body transformation based on input from at least one position tracking device that is connected to the 3D scanner; and

calculate a corrected scan by compensating for the motion trajectory.

2. The system of claim 1, where the processing module is coupled to the scanning module to:

align the previous scan with respect to the particular scan;

determine an overlapping portion of the previous scan with respect to the particular scan based on the alignment; and

determine an amount of movement between the previous scan and the particular scan based on the overlapping portion and/or a non-overlapping portion to register the particular scan to the at least one of the previous scan and the subsequent scan.

3. The system of claim 1, where the system includes a 2D imager coupled to the 3D scanner, and where the processing module is coupled to the scanning module to estimate a velocity of the 3D scanner based on input from the at least one position tracking device and to reset the estimated velocity to zero in response to input from the 2D imager indicating that the 3D scanner is not moving.

4. The system of claim 3, where the processing module is coupled to the scanning module to reset the estimated velocity to zero in six degrees of freedom in response to the input from the 2D imager comprising the plurality of 2D images indicating that the 3D scanner is not moving.

5. The system of claim 1, where the 3D scanner is in a first reference frame and where the scanning module includes a 2D imager in a second reference frame that is fixed with respect to the first reference frame;

and

where the processing module is coupled to the scanning module to;

receive the data from the plurality of two-dimensional (2D) images of the dentition taken, with the 2D imager, during a particular scan of the dentition, where each of the plurality of 2D images is associated with a respective time in the particular scan; and

compensate for motion during the scan using the improved estimated motion trajectory.

6. The system of claim 5, where the processing module is coupled to the scanning module to:

map the point from a coordinate system of the scan to a coordinate system of the image to result in a 2D trajectory for the point;

differentiate the resulting 2D trajectory to determine a number of velocity elements of the point;

apply the 2D trajectory and the number of velocity elements to the optical flow; and

sum squares of the optical flow with the applied 2D trajectory and the number of velocity elements between a reference time in the scan and a recorded time of the point in the scan to estimate the correctness of the estimated motion trajectory.

7. The system of claim 6, where the processing module is coupled to the scanning module to:

minimize an objective function that penalizes inaccurate trajectories at each point in the scan based on the optical flow with the applied 2D trajectory and the number of velocity elements to improve the estimated motion trajectory; and

regularize the 2D trajectory to penalize inaccurate trajectories.

14. The medium of claim 6, where the instructions to register the distorted scan to the at least one of the previous scan and the subsequent scan comprise instructions to:

associate a first point on a time axis in a middle of a scanning time of the previous scan with a rigid body transformation of the previous scan that aligns the previous scan with the distorted scan; and

associate a second point on the time axis in a middle of a scanning time of the subsequent scan with a rigid body transformation of the subsequent scan that aligns the subsequent scan with the distorted scan; and

where the instructions to estimate the first motion trajectory comprise instructions to interpolate between the first point, a point on the time axis in a middle of the scanning time of the distorted scan, and the second point.

15. The medium of claim 14, where the instructions to estimate the second motion trajectory comprise instructions to trim the first motion trajectory to only include motion trajectory during the scanning time of the distorted scan.

8. A computing device readable physical medium having instructions that can be executed by a processor to cause a computing device to:

receive a plurality of three-dimensional (3D) scans of a dentition;

register a distorted one of the plurality of scans to at least one of a previous scan and a subsequent scan;

determine whether an amount of movement between the previous scan, the distorted scan, and the subsequent scan is within a registration threshold;

estimate a first motion trajectory from the previous scan to the subsequent scan based on the registration;

estimate a second motion trajectory during a scanning time of the distorted scan; and

calculate a corrected scan for the distorted scan by compensating for the motion trajectory during the scanning time of the distorted scan.

9. The medium of claim 8, where the instructions can be executed by the processor to estimate the first and the second motion trajectories and calculate the corrected scan in response to the amount of movement being within the registration threshold.

10. The medium of claim 9, where the instructions can be executed by the processor to reject the distorted scan in response to the amount of movement being outside of the registration threshold.

11. The medium of claim 8, where the instructions to register the distorted scan to the at least one of the previous scan and the subsequent scan comprise instructions to:

align the previous scan with respect to the distorted scan;

determine an overlapping portion of the previous scan with respect to the distorted scan based on the alignment; and

determine an amount of movement between the previous scan and the distorted scan based on the overlapping portion and/or a non-overlapping portion.

12. The medium of claim 11, where the determined amount of movement between the previous scan, the distorted scan, and the subsequent scan includes the determined amount of movement between the previous scan and the distorted scan.

13. The medium of claim 11, where the instructions to align the previous scan with respect to the distorted scan comprise instructions to perform a rigid body transformation on the previous scan to align the previous scan with the distorted scan.

16. The medium of claim 8, where the first motion trajectory comprises a snapshot of an overall motion trajectory across the plurality of scans, and where the instructions can be executed by the processor to compute the overall motion trajectory based on a plurality of snapshots each associated with a respective registered one of the plurality of scans.

17. The medium of claim 8, where the plurality of scans comprise a plurality of 3D intraoral scans of a patient'"'"'s dentition.

18. The medium of claim 17, where the plurality of scans have a scanning frequency of at least ten scans per second.

19. The medium of claim 8, where instructions can be executed by the processor to iteratively repeat the registration of the distorted scan, estimation of the first motion trajectory, estimation of the second motion trajectory, and calculation of the corrected scan to improve an accuracy of the corrected scan.

20. The medium of claim 8, where the instructions to register the distorted one of the plurality of scans comprise instructions to register the distorted one of the plurality of scans to a plurality of previous scans and to a plurality of subsequent scans.

21. The medium of claim 8, where the previous scan comprises an immediately previous scan and where the subsequent scan comprises an immediately subsequent scan.

22. A computer implemented method, comprising:

receiving first data from a three-dimensional (3D) scan of a dentition taken with a 3D scanner in a first reference frame;

receiving second data from a plurality of two-dimensional (2D) images of the dentition taken during the scan of the dentition with a 2D imager in a second reference frame that is fixed with respect to the first reference frame, where each of the plurality of 2D images is associated with a respective time in the scan;

determining an optical flow based on local motion between consecutive images of the plurality of 2D images;

estimating a motion trajectory of a point in the scan;

improving the estimated motion trajectory using the optical flow to estimate a correctness of the estimated motion trajectory; and

compensating for motion during the scan using the improved estimated motion trajectory;

where the method is performed by the computer.

23. The method of claim 22, where determining the optical flow includes calculating the local motion between two consecutive images based on a change in position of a point in the two consecutive images and based on a change in time between the two consecutive images.

24. The method of claim 23, where calculating the local motion comprises calculating a 2D motion vector based on the change in position of the point.

25. The method of claim 22, where estimating the motion trajectory of the point in the scan includes using a rigid body transformation of the point dependent upon time in the scan.

26. The method of claim 25, where the method includes defining the rigid body transformation such that applying the rigid body transformation to the point maps the point from a coordinate system at a reference time in the scan to a coordinate system at a recorded time of the point in the scan.

27. The method of claim 26, where the compensating for motion during the scan comprises applying an inverse of the rigid body transformation to the point in the scan to map the point from the recorded time of the point in the scan to the reference time in the scan.

28. The method of claim 27, where the reference time is a middle time of the scan.

29. The method of claim 22, where estimating the correctness of the estimated motion trajectory comprises:

mapping the point from a coordinate system of the scan to a coordinate system of the 2D scan to result in a 2D trajectory for the point;

differentiating the resulting 2D trajectory to determine a number of velocity elements of the point;

applying the 2D trajectory and the number of velocity elements to the optical flow; and

summing squares of the optical flow with the applied 2D trajectory and the number of velocity elements between a reference time in the scan and a recorded time of the point in the scan.

30. The method of claim 29, where improving the estimated motion trajectory comprises minimizing an objective function that penalizes inaccurate trajectories at each point in the scan based on the optical flow with the applied 2D trajectory and the number of velocity elements.

31. The method of claim 29, where the method includes regularizing the 2D trajectory to penalize inaccurate trajectories.

32. The method of claim 22, where estimating the motion trajectory comprises:

estimating a global velocity vector based only on the optical flow; and

mapping the global velocity vector to a lateral motion trajectory comprising the estimated motion trajectory.

33. The method of claim 22, where the method includes using a heuristic to determine which points in the scan are visible in a particular one of the plurality of 2D images prior to determining the optical flow.

34. A system, comprising:

a scanning module including;

an intraoral three dimensional (3D) scanner; and

at least one position tracking device coupled to the 3D scanner; and

a processing module coupled to the scanning module to;

estimate an amount of motion of the 3D scanner during a scan as a rigid body transformation based on input from the at least one position tracking device; and

compensate for the amount of motion by applying an inverse of the rigid body transformation to a plurality of points comprising the scan.

35. The system of claim 34, where the at least one position tracking device includes a gyroscope to measure angular velocity of the 3D scanner.

36. The system of claim 34, where the at least one position tracking device includes an accelerometer to measure acceleration of the 3D scanner.

37. The system of claim 34, where:

the scanning module includes a two-dimensional (2D) imager that is coupled to the 3D scanner; and

the processing module is coupled to the scanning module to track motion of the 3D scanner based only on input from the at least one position tracking device and input form the 2D imager.

38. The system of claim 34, where:

the scanning module includes a two-dimensional (2D) imager that is coupled to the 3D scanner and fixed with respect to the 3D scanner; and

the processing module is coupled to the scanning module to estimate a velocity of the 3D scanner based on input from the at least one position tracking device and to reset the estimated velocity to zero in response to input from the 2D imager indicating that the 3D scanner is not moving.

39. The system of claim 38, where the processing module is coupled to the scanning module to reset the estimated velocity to zero in six degrees of freedom in response to the input from the 2D imager comprising a plurality of 2D images indicating that the 3D scanner is not moving.