Automated pharmaceutical pill identification

• US 10,229,321 B2
• Filed: 10/24/2017
• Issued: 03/12/2019
• Est. Priority Date: 01/23/2012
• Status: Active Grant

• Current Assignee: Interlink AI Incorporated
• Original Assignee: PerceptiMed, Inc.
• Inventors: Gershtein, Eugene, Jacobs, Alan Jeffrey, Gebala, Alexander
• Primary Examiner(s): Yang, Wei Wen
• Application Number:
  US15/792,723
• Publication Number:
  US 20180046862A1
• Time in Patent Office: 504 Days
• Field of Search: None
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 30/2528 Combination of methods, e.g. classifiers, working on the same input data
• CPC Class Codes: G06V 30/2552 Combination of methods, e.g. classifiers, working on different input data, e.g. sensor fusion

Abstract

A pill identification system identifies a pill type for a pharmaceutical composition from images of the pharmaceutical composition. The system extracts features from images taken of the pill. The features extracted from the pill image include color, size, shape, and surface features of the pill. In particular, the features include rotation-independent surface features of the pill that enable the pill to be identified from a variety of orientations when the images are taken. The feature vectors are applied to a classifier that determines a pill identification for each image. The pill identification for each image is scored to determine identification for the pharmaceutical composition.

Claims

1. A computer-implemented method for identifying a pharmaceutical composition, the method comprising:

accessing a set of images of the pharmaceutical composition, the set of images including;

a first image of the pharmaceutical composition from a first point of view, and

a second image of the pharmaceutical composition from a second point of view different from the first point of view;

extracting features from each image of the pharmaceutical composition in the set of images;

generating a feature vector associated with each image of the pharmaceutical composition that includes the features extracted for that image and one or more features appended from another image in the set of images;

applying a plurality of classifiers to each of the feature vectors to determine a set of classifications associated with each image, wherein each classification is associated with a score; and

determining a pill identification by accumulating the score associated with each classification for each image in the set of images.

2. The computer-implemented method of claim 1, further comprising:

receiving a pill verification request specifying a desired pill type;

determining whether the pill identification matches the desired pill type; and

reporting the determination of whether the pill identification matches the desired pill type.

3. The computer-implemented method of claim 2, the method further comprising directing the pharmaceutical composition to an accepted bin responsive to the pill identification matching the desired pill type and directing the pharmaceutical composition to a rejected bin responsive to a failure to match.

4. The computer-implemented method of claim 1, wherein the extracted features further include color, shape, size, and surface features.

5. The computer-implemented method of claim 4, wherein the feature extraction of each image includes performing the steps of:

identifying a pill outline of the image;

applying an erosion filter to the pill outline of the image;

replicating the pill outline to the exterior of the image after the erosion filter is applied; and

applying an anisotropic filter to the image after the pill outline is replicated.

6. The computer-implemented method of claim 1, wherein the features include a measure of surface features of the pharmaceutical composition in the image, wherein the measure of surface features is determined for each image by performing the steps of:

converting the image to a polar coordinate image;

sub sampling the polar coordinate image; and

extracting the measure of surface features from the subsampled polar coordinate image.

7. The computer-implemented method of claim 1, wherein the appended one or more features in the feature vector associated with each image of the pharmaceutical composition includes at least one of shape, color, and size-related features extracted from the other image.

8. The computer-implemented method of claim 7, wherein the appended features extracted from the other image are sorted with features from the image of the pharmaceutical composition prior to appending the sorted features to the feature vector.

9. The computer-implemented method of claim 1, wherein the plurality of classifiers comprises a set of hierarchical classifiers comprising at least one top-level classifier and at least one bottom-level classifier.

10. The computer-implemented method of claim 9, wherein the feature vectors include color features and the top-level classifier selects a lower-level classifier based on the color features in the feature vector.

11. The computer-implemented method of claim 9, wherein the feature vectors include size and shape features and the set of hierarchical classifiers further comprise at least one intermediate classifier that selects a lower-level classifier based on the size and shape features in the feature vector.

12. The computer-implemented method of claim 9, wherein the at least one bottom-level classifier determines a classification associated with a score based on surface features.

13. The computer-implemented method of claim 1, wherein determining the pill identification by accumulating the score associated with each classification for each image comprises determining a weighting for each image classification based on a confidence level of the respective classifications.

14. The computer-implemented method of claim 1, wherein the feature vector associated with each image includes a surface complexity feature, and wherein determining the pill identification by accumulating the score associated with each classification for each image comprises weighting the set of classifications for each image based on a relative surface complexity feature of the feature vector associated with the image.

15. A computer-implemented method for identifying a pharmaceutical composition, the method comprising:

accessing a set of images of the pharmaceutical composition, the set of images including;

a first image of the pharmaceutical composition from a first point of view, and

a second image of the pharmaceutical composition from a second point of view different from the first point of view;

developing a set of features for each image, wherein the features are developed for each image by;

identifying, in the image, an outline of the pharmaceutical composition;

calculating, from the image, a characterization of the image based at least in part on the identified outline;

reducing a resolution of the characterization of the image;

extracting, from the reduced resolution characterization of the image, features indicative of surface features of the image which comprise at least a portion of the features developed for the image;

appending to the set of features one or more features from another image in the set of images;

providing the set of features for each image to a classification module;

training a set of hierarchical classifiers based at least in part on the set of features for each image;

determining a pharmaceutical composition identification by;

applying the set of hierarchical classifiers to the set of features for each image to determine a set of classifications associated with each image, wherein each classification is associated with a score; and

accumulating the score associated with each classification for each image in the set of images.

16. The computer-implemented method of claim 15, wherein the extracting step further extracts color, size, and shape features from the image.

17. The computer-implemented method of claim 15, wherein the outline of the pharmaceutical composition is identified using an edge-detection algorithm.

18. The computer-implemented method of claim 15, wherein the outline of the pharmaceutical composition is identified using a grayscale segmentation algorithm.