Global visual vocabulary, systems and methods

• US 10,049,300 B2
• Filed: 02/27/2018
• Issued: 08/14/2018
• Est. Priority Date: 02/13/2014
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Song, Bing, McKinnon, David
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): Vanchy, Jr., Michael
• Application Number:
  US15/907,130
• Publication Number:
  US 20180189608A1
• Time in Patent Office: 168 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

Systems and methods of generating a compact visual vocabulary are provided. Descriptor sets related to digital representations of objects are obtained, clustered and partitioned into cells of a descriptor space, and a representative descriptor and index are associated with each cell. Generated visual vocabularies could be stored in client-side devices and used to obtain content information related to objects of interest that are captured.

Claims

1. A system comprising:

a recognition engine configured to;

extract a feature descriptor set for each of a plurality of medical images using a feature detection algorithm, each feature descriptor set existing within a descriptor space; and

a vocabulary generation engine coupled with the recognition engine and configured to;

obtain the plurality of descriptor sets;

partition the descriptor space into a plurality of cells based on one or more clustered regions of descriptors within the descriptor space;

assign an index to each cell of the plurality of cells based on a representative descriptor in each cell of the plurality of cells, wherein each representative descriptor is derived from a selected actual descriptor from the plurality of descriptor sets that is closest to an average of all descriptors in a corresponding cell of the descriptor space, and wherein each of the assigned indices is of a number of bytes selected based on an amount of cells comprising the plurality of cells; and

install a global vocabulary module comprising at least some of the representative descriptor and index pairs on at least one device, wherein the global vocabulary module is configured to generate a set of content indices that reference corresponding cells in the descriptor space based on an input set of descriptors corresponding to a person.

2. The system of claim 1, further comprising a communication interface to communicate with a content identification engine, the content identification engine being configured to:

obtain the set of content indices from the global vocabulary module;

construct a query of a content database based on the set of content indices; and

configure the at least one device to obtain a search result in response to the query of the content database, wherein the search result comprises at least one of a medical record or genomic data.

3. The system of claim 2, wherein constructing the query comprises using one or more of a tf-idf function, a term frequency ranking function, a constant ranking function, an Okapi BM25 ranking function, and a machine-learned ranking function.

4. The system of claim 2, wherein the search result is generated based on at least one threshold match between at least one of the assigned indices and at least one of the content indices of the set of content indices.

5. The system of claim 1, wherein the set of content indices correspond to an object that is identified by the medical device using the global vocabulary module.

6. The system of claim 1, wherein the global vocabulary module is integrated with one or more applications installed on the medical device.

7. The system of claim 1, wherein the plurality of medical images comprises one or more medical records.

8. The system of claim 1, wherein the recognition module is further configured to obtain the plurality of medical images.

9. The system of claim 1, wherein the recognition module is further configured to obtain at least one of object information or metadata regarding one or more of the plurality of medical images.

10. The system of claim 1, wherein the at least one device comprises a mobile or client-side device.

11. The system of claim 10, wherein the mobile or client-side device comprises a medical device.

12. The system of claim 1, wherein the at least one device comprises the recognition engine.

13. The system of claim 1, wherein the plurality of descriptor sets comprises image descriptors.

14. The system of claim 1, wherein the plurality of descriptor sets comprises multi-modal descriptors.

15. The system of claim 14, wherein the multi-modal descriptors comprise one or more key points.

16. The system of claim 1, wherein the feature detection algorithm comprises at least one of a SIFT, SURF, FAST, FREAK, BRISK, and DAISY algorithm.

17. The system of claim 1, wherein at least some of the plurality of descriptor sets exist within a common descriptor space.

18. The system of claim 17, wherein the common descriptor space is one of a SIFT, SURF, FAST, FREAK, BRISK, or DAISY descriptor space.

19. The system of claim 1, wherein each descriptor set in the plurality of descriptor sets has its own descriptor space.

20. The system of claim 1, wherein the vocabulary generation engine is further configured to cluster the plurality of descriptor sets into regions within the descriptor space using at least one of hierarchal k-mean, approximate k-mean, k-means clustering, and histogram binning.

21. The system of claim 1, wherein the vocabulary generation engine is further programmed to perform the step of partitioning the descriptor space based on Voronoi decomposition.

22. The system of claim 1, wherein each of the assigned indices is no more than three bytes.

23. The system of claim 1, wherein the input set of descriptors comprises image descriptors.

24. The system of claim 1, wherein the input set of descriptors comprises multi-modal descriptors.

25. A method comprising:

extracting a feature descriptor set for each of a plurality of medical images using a feature detection algorithm, each feature descriptor set existing within a descriptor space;

partitioning the descriptor space into a plurality of cells based on one or more clustered regions of descriptors within the descriptor space;

assigning an index to each cell of the plurality of cells based on a representative descriptor in each cell of the plurality of cells, wherein each representative descriptor is derived from a selected actual descriptor from the plurality of descriptor sets that is closest to an average of all descriptors in a corresponding cell of the descriptor space, and wherein each of the assigned indices is of a number of bytes selected based on an amount of cells comprising the plurality of cells; and

installing a global vocabulary module comprising at least some of the representative descriptor and index pairs on at least one device, wherein the global vocabulary module is configured to generate a set of content indices that reference corresponding cells in the descriptor space based on an input set of descriptors corresponding to a person.

26. The method of claim 25, further comprising:

configuring the at least one device to obtain a search result in response to a query of a content database based on the set of content indices, wherein the search result comprises at least one of a medical record or genomic data.

27. A non-transitory computer-readable medium having computer instructions stored thereon, which, when executed by a processor, cause the processor to perform one or more steps comprising:

extracting a feature descriptor set for each of a plurality of medical images using a feature detection algorithm, each feature descriptor set existing within a descriptor space;

partitioning the descriptor space into a plurality of cells based on one or more clustered regions of descriptors within the descriptor space;

assigning an index to each cell of the plurality of cells based on a representative descriptor in each cell of the plurality of cells, wherein each representative descriptor is derived from a selected actual descriptor from the plurality of descriptor sets that is closest to an average of all descriptors in a corresponding cell of the descriptor space, and wherein each of the assigned indices is of a number of bytes selected based on an amount of cells comprising the plurality of cells; and

installing a global vocabulary module comprising at least some of the representative descriptor and index pairs on at least one device, wherein the global vocabulary module is configured to generate a set of content indices that reference corresponding cells in the descriptor space based on an input set of descriptors corresponding to a person.

28. A computing device comprising:

a tangible, non-transitory, computer-readable memory having a global vocabulary module stored thereon, the global vocabulary module comprising representative descriptor and index pairs determined by (1) partitioning a descriptor space into a plurality of cells based on one or more clustered regions of descriptors within the descriptor space, and (2) assigning an index to each cell of the plurality of cells based on a representative descriptor in each cell of the plurality of cells, wherein each representative descriptor is derived from a selected actual descriptor from a plurality of descriptor sets that is closest to an average of all descriptors in a corresponding cell of the descriptor space, and wherein each of the assigned indices is of a number of bytes selected based on an amount of cells comprising the plurality of cells;

a recognition engine configured to extract an input set of descriptors corresponding to a person using a feature detection algorithm, the input set of descriptors existing within the descriptor space; and

at least one processor communicatively coupled with the tangible, non-transitory computer-readable memory which, upon execution of software instructions, is configured to;

generate a set of content indices that reference corresponding cells in the descriptor space based on the input set of descriptors using the global vocabulary module;

communicate, via a communication interface, the set of content indices to a content identification engine configured to construct a query of a content database based on the set of content indices;

obtain a search result in response to the query of the content database, wherein the search result comprises at least one of a medical record or genomic data; and

facilitate a display of the search result to a user.