Detection of surgical instruments on surgical tray

• US 10,357,325 B2
• Filed: 03/02/2017
• Issued: 07/23/2019
• Est. Priority Date: 01/16/2017
• Status: Active Grant

• Current Assignee: The Aga Khan University
• Original Assignee: The Aga Khan University
• Inventors: Sayani, Saleem, Muqeet, Muhammad Abdul, Mateen, Haris, Ahmed, Hafiz Imtiaz
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US15/448,201
• Publication Number:
  US 20180204323A1
• Time in Patent Office: 873 Days
• CPC Class Codes: A61B 50/33 Trays
• CPC Class Codes: A61B 90/92 coded with colour
• CPC Class Codes: A61B 90/96 using barcodes
• CPC Class Codes: A61B 90/98 using electromagnetic means, e.g. transponders
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/0008 checking presence/absence

Abstract

The subject matter described herein generally relates to determining whether any surgical instruments are missing on a surgical tray, and, if any instrument is missing, identifying the missing instrument(s). In one aspect, a server can receive, from a camera device, a matrix code and a bitmap of an image of a tray having a first plurality of instruments. The server can detect the tray based on the matrix code. The server can receive, from a database, a list of a second plurality of instruments configured to be stored in the detected tray. The server can detect the first plurality of instruments stored in the tray by using the bitmap. The server can compare the preset list of the second plurality of instruments with the detected first plurality of instruments to generate an output characterizing whether any instruments is missing from the tray and an identification of the missing instrument(s).

Claims

1. A method comprising:

receiving, by at least one computing server and from a camera device, a matrix code and a bitmap of an image of a tray having a first plurality of instruments;

detecting, by the at least one computing server, the tray based on the matrix code;

retrieving, by the at least one computing server and from a database associated with the at least one computing server, a preset list of a second plurality of instruments configured to be stored in the detected tray;

detecting, by the at least one computing server, the first plurality of instruments stored in the tray by using the bitmap;

comparing, by the at least one computing server, the preset list of the second plurality of instruments with the detected first plurality of instruments to generate an output characterizing whether one or more instruments are missing from the tray and an identification of the one or more instruments when the one or more instruments are missing from the tray; and

transmitting, by the at least one computing server, the output to a computing device.

2. The method of claim 1, wherein:

the matrix barcode is unique for the tray; and

the matrix barcode is a quick response code.

3. The method of claim 1, wherein the camera device is a single camera configured to rotate around at least one of yaw, pitch, and roll axes.

4. The method of claim 3, wherein the camera device is affixed to a ceiling.

5. The method of claim 1, wherein:

the at least one computing server identifies features of the first plurality of instruments and represents the identified features as a compact feature vector, the representation as the compact feature vector being performed prior to the comparing; and

the compact feature vector is prepared using one or more of dimensionality reduction techniques comprising independent component analysis, kernel principal component analysis, latent semantic analysis, partial least squares, principal component analysis, multifactor dimensionality reduction, nonlinear dimensionality reduction, multilinear principal component analysis, multilinear subspace learning, semidefinite embedding, autoencoder, and deep feature synthesis.

6. The method of claim 1, wherein the detecting of the first plurality of instruments comprises:

retrieving the bitmap from a framebuffer within the at least one computing server; and

identifying features of the first plurality of instruments from the bitmap.

7. The method of claim 6, wherein the detecting of the first plurality of instruments further comprises transforming the bitmap with identified features to a hue, saturation and value (HSV) representation of the bitmap, the at least one computing server using the HSV representation to expedite the identifying of the features.

8. The method of claim 6, wherein the detecting of the first plurality of instruments further comprises performing morphological operations for each of the first plurality of instruments, the morphological operations comprising erosion and dilation, the at least one computing server applying the erosion to remove noise from a binary image to shrink the binary image, the at least one computing server applying the dilation to grow back pixels in the shrunk binary image, the removal of the noise expediting the identifying of the features.

9. The method of claim 6, wherein the detecting of the first plurality of instruments further comprises identifying a first plurality of color markers on respective instruments of the first plurality of instruments, the first plurality of color markers expediting the identifying of the features.

10. The method of claim 9, further comprising enhancing one or more colors of respective one or more of the first plurality of color markers to further expedite the identifying of the features.

20. The method of claim 1, wherein the database is operably coupled to the at least one computing server via a communication network.

11. A non-transitory computer program product storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

receiving, from a camera device, a matrix code and a bitmap of an image of a tray having a first plurality of instruments;

detecting the tray based on the matrix code;

retrieving, from a database associated with the at least one programmable processor, a preset list of a second plurality of instruments configured to be stored in the detected tray;

detecting the first plurality of instruments stored in the tray by using the bitmap;

comparing the preset list of the second plurality of instruments with the detected first plurality of instruments to generate an output characterizing whether one or more instruments are missing from the tray and an identification of the one or more instruments when the one or more instruments are missing from the tray; and

transmitting the output to a computing device.

12. The non-transitory computer program product of claim 11, wherein the detecting of the first plurality of instruments comprises:

retrieving the bitmap from a framebuffer within the at least one programmable processor; and

identifying features of the first plurality of instruments from the bitmap.

13. The non-transitory computer program product of claim 12, wherein the detecting of the first plurality of instruments further comprises transforming the bitmap with identified features to a hue, saturation and value (HSV) representation of the bitmap, the at least one programmable processor using the HSV representation to expedite the identifying of the features.

14. The non-transitory computer program product of claim 12, wherein the detecting of the first plurality of instruments further comprises performing morphological operations for each of the first plurality of instruments, the morphological operations expediting the identifying of the features.

15. The non-transitory computer program product of claim 12, wherein the detecting of the first plurality of instruments further comprises identifying a first plurality of color markers on respective instruments of the first plurality of instruments, the first plurality of color markers expediting the identifying of the features.

16. The non-transitory computer program product of claim 15, wherein the operations further comprise enhancing one or more colors of respective one or more of the first plurality of color markers to further expedite the identifying of the features.

17. A system comprising:

a camera device to take an image of a tray having a first plurality of instruments, the camera device configured to store a matrix barcode associated with the image, the matrix code being unique for the tray, the camera device further configured to store a bitmap of an image of the tray; and

at least one computing server operably coupled to the camera device, the at least one computing server configured to;

receive, from the camera device, the matrix code and the bitmap of the image of the tray;

detect the tray based on the matrix code;

retrieve, from a database associated with the at least one computing server, a preset list of a second plurality of instruments configured to be stored in the detected tray;

detect the first plurality of instruments stored in the tray by using the bitmap;

compare the preset list of the second plurality of instruments with the detected first plurality of instruments to generate an output characterizing whether one or more instruments are missing from the tray and an identification of the one or more instruments when the one or more instruments are missing from the tray.

18. The system of claim 17, wherein the at least one computing server is remote to and operably coupled to the camera device, the at least one computing server being operably coupled to the camera device via internet.

19. The system of claim 17, wherein the matrix barcode is a quick response code.