Photograph-based assessment of dental treatments and procedures

• US 10,248,883 B2
• Filed: 08/20/2015
• Issued: 04/02/2019
• Est. Priority Date: 08/20/2015
• Status: Active Grant

• Current Assignee: Align Technology Incorporated
• Original Assignee: Align Technology Incorporated
• Inventors: Borovinskih, Artem, Derakhshan, Mitra, Koppers, Carina, Meyer, Eric, Tolstaya, Ekaterina, Brailov, Yury
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Sullivan, Tyler W.
• Application Number:
  US14/831,548
• Publication Number:
  US 20170049311A1
• Time in Patent Office: 1,321 Days
• CPC Class Codes: A61C 2007/004 Automatic construction of a set of axes for a tooth or a plurality of teeth
• CPC Class Codes: A61C 7/002 Orthodontic computer assisted systems
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30036 Dental; Teeth
• CPC Class Codes: G06T 3/06 Topological mapping of higher dimensional structures onto lower dimensional surfaces
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06V 30/1831 using gradient analysis

Abstract

The current document is directed to methods and systems for monitoring a dental patient'"'"'s progress during a course of treatment. A three-dimensional model of the expected positions of the patient'"'"'s teeth can be projected, in time, from a three-dimensional model of the patient'"'"'s teeth prepared prior to beginning the treatment. A digital camera is used to take one or more two-dimensional photographs of the patient'"'"'s teeth, which are input to a monitoring system. The monitoring system determines virtual-camera parameters for each two-dimensional input image with respect to the time-projected three-dimensional model, uses the determined virtual-camera parameters to generate two-dimensional images from the three-dimensional model, and then compares each input photograph to the corresponding generated two-dimensional image in order to determine how closely the three-dimensional arrangement of the patient'"'"'s teeth corresponds to the time-projected three-dimensional arrangement.

Claims

1. A dental-treatment monitoring system comprising:

one or more processors;

one or more electronic memories that store instructions and data;

one or more mass-storage devices that store encoded images and patient information;

a communications subsystem through which the dental-treatment monitoring system receives images and information from remote computer systems; and

computer instructions, encoded in one or more of the one or more electronic memories, that control the dental-treatment monitoring system to store, in one or more data-storage devices selected from among the one or more electronic memories and one or more mass-storage devices, a three-dimensional model of a patient'"'"'s teeth, receive, through the communications subsystem, one or more two-dimensional digital photographs of the patient'"'"'s teeth taken at a time t during the course of a dental treatment, project, in time, the three-dimensional model of the patient'"'"'s teeth to the time t to generate and store a time-projected three-dimensional model of the patient'"'"'s teeth representing an expected configuration of the patient'"'"'s teeth, compare one or more of the one or more two-dimensional digital photographs to corresponding two-dimensional digital images generated from the time-projected three-dimensional model of the patient'"'"'s teeth to generate one or more comparison values and generate, from the received one or more two-dimensional digital photographs, a set of one or more processed images with associated metadata for analysis, determine, for each image in the set of one or more processed images, a set of virtual-camera parameters that describe the position, and orientation for a virtual camera that produces a generated image from the time-projected three-dimensional model of the patient'"'"'s teeth equivalent to the image in the set of one or more processed images, use, for each image in the set of one or more processed images, the standard type of view and additional metadata to generate an initial set of virtual-camera parameters for the image in the set of one or more processed images, use the standard type of view and additional metadata to determine a value for an iteration variable;

carry out a number of optimization iterations equal to the value of the iteration variable; and

finally refine the virtual-camera parameters for the image in the set of one or more processed images, generate, for each image in the set of one or more processed images, a generated image corresponding to the image from the time-projected three-dimensional model of the patient'"'"'s teeth and the virtual-camera parameters determined for the image in the set of one or more processed images, and compare each image in the set of one or more processed images with the corresponding generated image to generate the one or more comparison values for the image in the set of one or more processed images, determine, from the one or more comparison values, whether or not a configuration of the patient'"'"'s teeth is within a threshold level of correspondence to the expected configuration of the patient'"'"'s teeth, and store an indication of the determination in one of the one or more electronic memories.

2. The dental-treatment monitoring system of claim 1 wherein the received two-dimensional digital photographs are associated with a patient identifier;

and wherein each of the received two-dimensional digital photographs are associated with metadata that includes one or more of;

one or more text labels for the two-dimensional image, including an indication of a standard type of view represented by the image, and characterizations and parameters for the digitally encoded image, including an image size, date and time information, an indication of a camera model and make, an indication of a camera orientation, an indication of an aperture, an indication of a shutter speed, an indication of a focal length, an indication of a metering mode, and an indication of a speed.

3. The dental-treatment monitoring system of claim 2 further comprises verifying and filtering, by the dental-treatment monitoring system, the patient ID and the received two-dimensional digital photographs and associated metadata to produce a set of one or more processed images for analysis by:

accessing stored information to verify that the patient ID corresponds to a patient record;

for each input image, determining that the image corresponds to the standard type of view indicated by the one or more text labels, applying filters and contrast adjustment to enhance the image for subsequent processing, when the image is the first image of a standard type of view, adding the image and associated metadata to the set of one or more processed images, and when the image has the same standard type of view as an image already added to the set of one or more processed images, replacing the image already added to the set of one or more processed images with the image and associated metadata; and

determining whether the set of one or more processed images can be used for dental-treatment-progress analysis.

4. The dental-treatment monitoring system of claim 1 wherein each optimization iteration includes:

a first adjustment of the current virtual-camera parameters for the image in the set of one or more processed images;

a second adjustment of the current virtual-camera parameters for the image;

a first refinement of the current virtual-camera parameters for the image; and

a second refinement of the current virtual-camera parameters for the image.

5. The dental-treatment monitoring system of claim 4, further comprising, for each optimization iteration, thresholding the image to generate a first teeth mask for the image;

thresholding a next corresponding image generated from the time-projected three-dimensional model of the patient'"'"'s teeth using the current virtual-camera parameters for the image to generate a next second corresponding teeth mask, generating a distance transform of the first teeth mask, searching over scalings, rotations, and translations of the next second corresponding teeth mask to identify a next minimum-cost overlay of the next second corresponding teeth mask over the distance transform of the first teeth mask, and adjusting the virtual-camera parameters for the image corresponding to the next minimum-cost overlay; and

selecting, as the adjusted virtual-camera parameters for the image, the virtual-camera parameters associated with lowest-cost overlay of any of the generated next second corresponding teeth mask over the distance transform of the first teeth mask.

6. The dental-treatment monitoring system of claim 5 wherein the cost of an overlay of a next second corresponding teeth mask over the distance transform of the first teeth mask is the sum of the distances associated with pixels in the distance transform of the first teeth underlying a tooth region within the next second corresponding teeth mask.

7. The dental-treatment monitoring system of claim 5 wherein adjusting the virtual-camera parameters for the image corresponding to the next minimum-cost overlay includes:

adjusting the virtual-camera parameters to move the position of the center of the next corresponding image generated from the time-projected three-dimensional model of the patient'"'"'s teeth;

adjusting the virtual-camera parameters to rotate the next corresponding image generated from the time-projected three-dimensional model of the patient'"'"'s teeth in the image plane;

adjusting the virtual-camera parameters to rotate the next corresponding image generated from the time-projected three-dimensional model of the patient'"'"'s teeth about an axis parallel to the image plane;

adjusting the virtual-camera parameters to rescale the next corresponding image generated from the time-projected three-dimensional model of the patient'"'"'s teeth.

8. The dental-treatment monitoring system of claim 4 wherein the second adjustment of the current virtual-camera parameters for the image comprises:

for each of the patient'"'"'s two jaws, masking out the other of the two jaws and associated teeth from the image, and optimizing the virtual-camera parameters for the image with respect to a pixel-associated cost for the image and a next corresponding image.

9. The dental-treatment monitoring system of claim 8 wherein the cost for the image and a next corresponding image is computed as a linear combination of the mutual information for the image and the next corresponding image and the sum of pixel-associated costs for the image and the next corresponding image based on a dot-product result for gradient vectors for the image and the next corresponding image computed for each pixel.

10. The dental-treatment monitoring system of claim 8 wherein optimizing the virtual-camera parameters for the image comprises:

for each optimization iteration, generating a next corresponding image generated from the time-projected three-dimensional model of the patient'"'"'s teeth using the current virtual-camera parameters for the image, computing a cost for the image and next corresponding image, and perturbing the virtual-camera parameters for the image in a direction that minimizes the cost for the image and next corresponding image.

11. The dental-treatment monitoring system of claim 8 wherein optimizing the virtual-camera parameters for the image comprises application of a Nelder-Mead downhill simplex optimization method, with seven dimensions, including three rotations, three translations, and the virtual-camera view angle.

12. The dental-treatment monitoring system of claim 4 wherein the first refinement of the current virtual-camera parameters for the image comprises:

for each of the patient'"'"'s two jaws, for each tooth group of the currently considered jaw, masking the image to leave the currently considered tooth group, and optimizing the virtual-camera parameters for the image with respect to a gradient-vector-based cost for the image and a next corresponding image.

13. The dental-treatment monitoring system of claim 4 wherein the second refinement of the current virtual-camera parameters for the image comprises:

for each of the patient'"'"'s two jaws, for each tooth group of the currently considered jaw, masking the image to leave the currently considered tooth group, and optimizing the virtual-camera parameters for the image with respect to a Laplacian-vector-based cost for the image and a next corresponding image.

14. The dental-treatment monitoring system of claim 1 wherein comparing each image in the set of one or more processed images with the corresponding generated image to generate a comparison value for the image further comprises:

for each image, preparing a teeth contour from the corresponding generated image;

overlaying the teeth contour onto the image; and

generating a fitness metric for the image, corresponding image, and overlaid contour.

15. The dental-treatment monitoring system of claim 14 wherein the fitness metric is based on a per-pixel cost equal to the normalized dot product of gradient vectors for the image and corresponding image computed for a pixel.

16. The dental-treatment monitoring system of claim 15 wherein the fitness metric is the sum of the per-pixel costs for pixels overlying the contour divided by the length of the contour.

17. The dental-treatment monitoring system of claim 15 wherein the fitness metric is the sum of the per-pixel costs for pixels overlying the contour divided by the sum of maximum costs of pixels within a compact neighborhood that includes each pixel overlying the contour.

18. A method carried out within a dental-treatment monitoring system having one or more processors, one or more electronic memories that store instructions and data, one or more mass-storage devices that store encoded images and patient information, and a communications subsystem through which the dental-treatment monitoring system receives images and information from remote computer systems, the method comprising:

storing, in one or more data-storage devices selected from among the one or more electronic memories and one or more mass-storage devices, a three-dimensional model of a patient'"'"'s teeth;

receiving, through the communications subsystem, one or more two-dimensional digital photographs of the patient'"'"'s teeth taken at a time t during the course of a dental treatment, projecting, in time, the three-dimensional model of the patient'"'"'s teeth to the time t to generate and store a time-projected three-dimensional model of the patient'"'"'s teeth representing an expected configuration of the patient'"'"'s teeth;

comparing one or more of the one or more two-dimensional digital photographs to corresponding two-dimensional digital images generated from the time-projected three-dimensional model of the patient'"'"'s teeth to generate one or more comparison values;

generating, from the received one or more two-dimensional digital photographs, a set of one or more processed images with associated metadata for analysis, determining, for each image in the set of one or more processed images, a set of virtual-camera parameters that describe the position and orientation for a virtual camera that produces a generated image from the time-projected three-dimensional model of the patient'"'"'s teeth equivalent to the image in the set of one or more processed images, using, for each image in the set of one or more processed images, the standard type of view and additional metadata to generate an initial set of virtual-camera parameters for the image, using the standard type of view and additional metadata to determine a value for an iteration variable;

carrying out a number of optimization iterations equal to the value of the iteration variable; and

finally refining the virtual-camera parameters for the image, generating, for each image in the set of one or more processed images, a generated image corresponding to the image from the time-projected three-dimensional model of the patient'"'"'s teeth and the virtual-camera parameters determined for the image, and comparing each image in the set of one or more processed images with the corresponding generated image to generate a comparison value for the image;

determining, using the one or more comparison values, whether or not a configuration of the patient'"'"'s teeth is within a threshold level of correspondence to the expected configuration of the patient'"'"'s teeth; and

storing an indication of the determination in one of the one or more electronic memories.

19. A dental-treatment monitoring system comprising:

one or more processors;

one or more electronic memories that store instructions and data;

one or more mass-storage devices that store encoded images and patient information;

a communications subsystem through which the dental-treatment monitoring system receives images and information from remote computer systems; and

computer instructions, encoded in one or more of the one or more electronic memories, that control the dental-treatment monitoring system to store, in one or more data-storage devices selected from among the one or more electronic memories and one or more mass-storage devices, a three-dimensional model of a patient'"'"'s teeth, receive, through the communications subsystem, one or more two-dimensional digital photographs of the patient'"'"'s teeth taken at a time t during the course of a dental treatment, project, in time, the three-dimensional model of the patients teeth to the time t to generate and store a time-projected three-dimensional model of the patient'"'"'s teeth representing an expected configuration of the patients teeth, compare one or more of the one or more two-dimensional digital photographs to corresponding two-dimensional digital images generated from the time-projected three-dimensional model of the patients teeth to generate one or more comparison values and generate, from the received one or more two-dimensional digital photographs, a set of one or more processed images with associated metadata for analysis, determine, for each image in the set of one or more processed images, a set of virtual-camera parameters that describe the position and orientation for a virtual camera that produces a generated image from the time-projected three-dimensional model of the patient'"'"'s teeth equivalent to the image in the set of one or more processed images, generate, for each image in the set of one or more processed images, a generated image corresponding to the image from the time-projected three-dimensional model of the patients teeth and the virtual-camera parameters determined for the image, and compare each image in the set of one or more processed images with the corresponding generated image to generate a comparison value for the image and preparing, for each image, a teeth contour from the corresponding generated image;

overlaying the teeth contour onto the image, generating a fitness metric for the image, corresponding image, and overlaid contour, wherein the fitness metric is based on a per-pixel cost equal to the normalized dot product of gradient vectors for the image and corresponding image computed for each pixel, determine, from the one or more comparison values, whether or not a configuration of the patient'"'"'s teeth is within a threshold level of correspondence to the expected configuration of the patients teeth, and store an indication of the determination in one of the one or more electronic memories.