Image processing system and method

• US 9,946,958 B1
• Filed: 10/14/2016
• Issued: 04/17/2018
• Est. Priority Date: 10/14/2016
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Gorelick, Micha, Mason, Hilary, Custer, Grant
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/293,679
• Publication Number:
  US 20180107899A1
• Time in Patent Office: 550 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns

Abstract

An image processing system involves a camera, at least one processor associated with the camera, non-transitory storage, a lexical database of terms and image classification software. The image processing system uses the image classification software to assign hyponyms and associated probabilities to an image and then builds a subset hierarchical tree of hypernyms from the lexical database of terms. The processor then scores the hypernyms and identifies at least one hypernym for the image that has a score that is calculated to have a value that is greater than one of: a pre-specified threshold score, or all other calculated level scores within the subset hierarchical tree. The associated methods are also disclosed.

Claims

1. An image processing system comprising:

a camera;

at least one processor associated with the camera;

non-transitory storage, associated with the camera and accessible to the processor, the non-transitory storage containing therein

an image obtained via the camera, and

programming executable by the at least one processor;

a lexical database of terms, accessible to the processor,

wherein the lexical database of terms is arranged in a hierarchy including a single root node, multiple leaf nodes, and multiple hypernym nodes, located between the root node and at least one leaf node and arranged such that an individual hypernym node will have either

the root node or another hypernym node as a parent node, and

at least one leaf node or hypernym node as a child node,

wherein the root node is associated with a term in the hierarchy of greatest generality,

wherein each of the leaf nodes has an associated hyponym term, the hyponym terms representing terms in the hierarchy of greatest specificity, and

wherein each of the hypernym nodes is associated with a term that is more specific than its parent node and less specific than each of its child nodes, and

wherein each of the hyponym terms has an assigned value between 0 and 1;

image classification software which, when executed by the at least one processor will

classify the image, using contents of a visual database having multiple images therein with each of the images having at least one hyponym from the lexical database of terms associated with it so as to assign to the image at least two hyponym terms and a specific probability value representing a probability that each of the at least two hyponym terms accurately describes content of the image;

wherein, when the programming is executed by the at least one processor, the processor will, for the image

i) build a subset hierarchical tree, from the lexical database of terms, containing the assigned at least two hyponym terms, all hyponym terms sharing a common parent with the assigned at least two hyponym terms, and all hypernyms within the hierarchy between the at least two hyponyms and the root node,

ii) calculate a first level score for each hypernym in the subset hierarchical tree, that is a first level hypernym because it is directly connected to a leaf node, using a specified scoring function and the assigned specific probability value for each hyponym;

iii) calculate a second level score for each hypernym that is a second level hypernym, because it is directly connected to at least one first level hypernym, using first level scores of its child nodes in the specified scoring function,

iv) calculate additional level scores for each additional level of parent hypernyms that are above the second level hypernyms using calculated scores for all immediate child hypernyms of each parent hypernym in the specified scoring function, and

v) identify at least one hypernym for the image, the at least one hypernym being a hypernym associated with a node, other than the root node, that has a specific level score that is calculated to have a value that is greater than one of;

a pre-specified threshold score, or

all other calculated level scores within the subset hierarchical tree.

2. The image processing system of claim 1, wherein the image is a classified image that is one of multiple classified images and wherein, when the programming is executed by the at least one processor following classification but prior to “

i)”

, the processor will, for the multiple classified images

combine their individual leaf node probabilities, on an individual hypernym basis, as a normalized log sum of the individual probabilities; and

treat the multiple classified images as the image for purposes of “

i)”

through “

v)”

.

3. The image processing system of claim 2, further comprising:

a user interface via which a user can input a query, wherein, upon receipt of the query, the processor will apply a clustering algorithm to a set of the multiple classified images, selected based upon hyponym scores, to determine at least one similarity among the set.

4. The image processing system of claim 3, wherein the clustering algorithm is one of a Leacock-Chodorow similarity algorithm, a Wu-Palmer similarity algorithm, a Resnik similarity algorithm, a Jiang &

Conrath similarity algorithm, a Lin similarity algorithm, or a Nguyen and Al-Mubaid similarity algorithm.

5. The image processing system of claim 1, wherein the lexical database of terms is one or more of:

WordNet, eXtended WordNet, FrameNet, BabelNet, Malayalam WordNet, Chinese Wordnet, WordNet Libre du Franç

ais, Just Another WordNet Subset (“

JAWS”

), IndoWordNet, MultiWordNet, EuroWordNet, Global Wordnet, BalkaNet, Russian WordNet, FinnWordNet, GermaNet, OpenWN-PT, plWordNet, PolNet, or BulNet.

6. The image processing system of claim 5, wherein the visual database is ImageNet.

7. The image processing system of claim 6, wherein the image classification software is constructed according to an architecture corresponding to one of:

GoogLeNet or AlexNet.

8. The image processing system of claim 1, wherein the visual database is ImageNet.

9. The image processing system of claim 8, wherein the image classification software is constructed according to an architecture corresponding to one of:

GoogLeNet or AlexNet.

10. The image processing system of claim 1, wherein the camera, at least one processor, and non-transitory storage are housed within one of:

an autonomous undersea vehicle, an autonomous land vehicle, an autonomous flying vehicle, a game/trail camera unit, or a smartphone.

11. The image processing system of claim 1, wherein the specified scoring function is calculated, on a per node basis, as:

12. The image processing system of claim 1, wherein the specified scoring function is calculated, on a per node basis, as:

13. The image processing system of claim 1, wherein the specified scoring function is calculated, on a per node basis, as:

Score(node)=[g(node)×

log₁₀(sum of g(node) for all images in visual database)]

where g(node)=average probability of all directly connected child nodes.

14. The image processing system of claim 1, wherein the specified scoring function is calculated, on a per node basis, as: