Image processing system and method

• US 10,007,864 B1
• Filed: 02/08/2018
• Issued: 06/26/2018
• Est. Priority Date: 10/14/2016
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Gorelick, Micha, Mason, Hilary, Custer, Grant
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/891,969
• Publication Number:
  US 20180181837A1
• Time in Patent Office: 138 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns

Abstract

An image processing system involves a camera, at least one processor associated with the camera, non-transitory storage, a lexical database of terms and image classification software. The image processing system uses the image classification software to assign hyponyms and associated probabilities to an image and then builds a subset hierarchical tree of hypernyms from the lexical database of terms. The processor then scores the hypernyms and identifies at least one hypernym for the image that has a score that is calculated to have a value that is greater than one of: a pre-specified threshold score, or all other calculated level scores within the subset hierarchical tree. The associated methods are also disclosed.

Claims

1. A method for processing digital images, the method comprising:

processing, by a computer system, a digital image using a visual database to generate an image classification, the image classification including a first term and an associated probability value indicative of a probability that the first term accurately describes the digital image;

generating, by the computer system, a subset hierarchical tree based on a lexical database, the subset hierarchical tree including the first term and a plurality of other terms from the lexical database, the first term and plurality of other terms sharing a common parent and arranged in a hierarchy based on specificity;

calculating, by the computer system, probability values for one or more of the plurality of other terms in the subset hierarchical tree based on the probability value associated with the first term;

identifying, by the computer system, based on the calculating, a second term from the plurality of other terms that is broader than the first term and that has a calculated probability value that satisfies a specified criterion; and

generating, by the computer system, an output to a user based on the identified second term.

2. The method of claim 1, wherein the second term is a hypernym indicative of a category that includes the first term as a hyponym.

3. The method of claim 1, wherein the lexical database includes a plurality of nodes representative of terms, the plurality of nodes arranged in a hierarchy based on specificity of the associated terms, the plurality of nodes including:

a single root node;

a plurality of hyponym nodes; and

a plurality of hypernym nodes, each of the plurality of hypernym nodes arranged in the hierarchy between the root node and at least one of the plurality of hyponym nodes.

4. The method of claim 3, wherein:

the root node is associated with a root term of greatest generality in the hierarchy;

each of the hyponym nodes is associated with a hyponym term of greatest specificity in the hierarchy; and

each of the hypernym nodes is associated with a hypernym term that is more specific than terms associated with each of its one or more parent nodes and less specific than terms associated with each of its one or more child nodes.

5. The method of claim 3, wherein the subset hierarchical tree is a subset of the lexical database including a hyponym node associated with the first term, a subset of the plurality hyponym nodes associated with other hyponym terms that share the common parent with the first term, and a subset of the plurality of hypernym nodes associated with hypernym terms within the hierarchy between the subset of the plurality of hyponym nodes and the root node.

6. The method of claim 5, wherein calculating probability values for the plurality of other terms in the subset hierarchical tree includes:

calculating, by the computer system, on a per-node basis, a probability value for a given node in the subset hierarchical tree based on probability values of immediate child nodes to the given node using a specified scoring function.

7. The method of claim 6, wherein the specified scoring function defines the probability value for the given node as based on any of:

a sum of the probability values of the immediate child nodes;

a log sum of the probability values of the immediate child nodes;

an average of the probability values of the immediate child nodes; and

/or

a distance, in the subset hierarchical tree, from the given node to a hyponym node that is a child to the given node.

8. The method of claim 1, wherein the calculated probability value for the second term of the plurality of other terms satisfies the specified criterion if the calculated probability value is greater than:

a specified threshold probability value; and

/or

all other calculated probability values for terms in the subset hierarchical tree.

9. The method of claim 1, wherein the generated output is in response to a query by the user for a digital image that is associated with the identified second term.

10. A system for processing digital images, the system comprising:

one or more processors; and

memory having stored thereon;

a lexical database; and

instructions, which when executed by the one or more processors, cause the system to;

receive an image classification associated with a digital image, the image classification including a first term and an associated probability value indicative of a probability that the first term accurately describes the digital image;

generate a subset hierarchical tree using the lexical database, the subset hierarchical tree including the first term and a plurality of other terms from the lexical database, the first term and plurality of other terms sharing a common parent and arranged in a hierarchy based on specificity;

calculate probability values for one or more of the plurality of other terms in the subset hierarchical tree based on the probability value associated with the first term;

identify, based on the calculating, a second term from the plurality of other terms that is broader than the first term and that has a calculated probability value that satisfies a specified criterion; and

generating an output based on the identified second term.

11. The system of claim 10, wherein the lexical database includes a plurality of nodes representative of terms, the plurality of nodes arranged in a hierarchy based on specificity of the associated terms, the plurality of nodes including:

a single root node;

a plurality of hyponym nodes; and

a plurality of hypernym nodes, each of the plurality of hypernym nodes arranged in the hierarchy between the root node and at least one of the plurality of hyponym nodes.

12. The system of claim 11, wherein:

the root node is associated with a root term of greatest generality in the hierarchy;

each of the hyponym nodes is associated with a hyponym term of greatest specificity in the hierarchy; and

each of the hypernym nodes is associated with a hypernym term that is more specific than terms associated with each of its one or more parent nodes and less specific than terms associated with each of its one or more child nodes.

13. The system of claim 11, wherein the subset hierarchical tree is a subset of the lexical database including a hyponym node associated with the first term, a subset of the plurality hyponym nodes associated with other hyponym terms that share the common parent with the first term, and a subset of the plurality of hypernym nodes associated with hypernym terms within the hierarchy between the subset of the plurality of hyponym nodes and the root node.

14. The system of claim 13, wherein calculating probability values for the plurality of other terms in the subset hierarchical tree includes:

calculating, by the computer system, on a per-node basis, a probability value for a given node in the subset hierarchical tree based on probability values of immediate child nodes to the given node using a specified scoring function.

15. The system of claim 14, wherein the specified scoring function defines the probability value for the given node as based on any of:

a sum of the probability values of the immediate child nodes;

a log sum of the probability values of the immediate child nodes;

an average of the probability values of the immediate child nodes; and

/or

a distance, in the subset hierarchical tree, from the given node to a hyponym node that is a child to the given node.

16. The system of claim 10, wherein the calculated probability value for the second term of the plurality of other terms satisfies the specified criterion if the calculated probability value is greater than:

a specified threshold probability value; and

/or

all other calculated probability values for terms in the subset hierarchical tree.

17. The system of claim 10, further comprising:

a camera;

wherein the digital image is captured by the camera; and

wherein the memory has further instructions stored thereon, which when executed by the one or more processors, cause the system to further;

process the digital image using an image classification algorithm to generate the image classification.

18. The system of claim 10, wherein the one or more processors and memory are housed within an autonomous vehicle.

19. A method for searching a set of digital images, the method comprising:

receiving, by a computer system, a user query;

accessing, by the computer system, a plurality of digital images based on the user query, each of the plurality of digital images including a hyponym probability value indicative of a probability that a given hyponym term accurately describes a given digital image;

generating, by the computer system, a subset hierarchical tree based on a lexical database, the subset hierarchical tree including;

a plurality of hyponym nodes associated with a plurality of hyponym terms of the plurality of digital images; and

a plurality of hypernym nodes associated with hypernym terms that are related to the plurality of hyponym terms, each of the plurality of hypernym nodes arranged in the hierarchy between a root node and at least one of the plurality of hyponym nodes;

calculating, by the computer system, on a per node basis, probability values for each hypernym node in the subset hierarchical tree, based on probability values of child nodes to the hypernym node using a specified scoring function;

identifying, by the computer system, based on the calculating, a set of hypernym terms associated with hypernym nodes with probability values that satisfy a specified criterion;

clustering, by the computer system, the set of hypernym terms based on a similarity algorithm and the user search query; and

returning, by the computer system, a set of digital images from the accessed plurality of digital images based on a result of the clustering.

20. The method of claim 19, further comprising:

combining the hyponym probability values of each of the accessed plurality of digital images before generating the subset hierarchical tree.

21. The method of claim 20, wherein combining the hyponym probability values of each of the accessed plurality of digital images includes calculating a normalized log sum of the hyponym probability values for each hyponym term occurring in the accessed plurality of digital images.

22. The method of claim 20, wherein the specified scoring function defines the probability value for a given node in the subset hierarchical tree as based on any of:

a sum of the probability values of immediate child nodes to the given node;

a log sum of the probability values of the immediate child nodes to the given node;

an average of the probability values of the immediate child nodes to the given node; and

/or

a distance, in the subset hierarchical tree, from the given node to a hyponym node that is a child to the given node.

23. The method of claim 20, wherein the probability value of a given hypernym node satisfies the specified criterion if the probability value is greater than:

a specified threshold probability value; and

/or

probability values of all of a specified percentage of other hypernym nodes in the subset hierarchical tree.