Visual semantic complex network and method for forming network

• US 10,296,531 B2
• Filed: 11/30/2013
• Issued: 05/21/2019
• Est. Priority Date: 11/30/2013
• Status: Active Grant

• Current Assignee: Beijing SenseTime Technology Development Co., Ltd. (SenseTime group Limited)
• Original Assignee: Beijing SenseTime Technology Development Co., Ltd. (SenseTime group Limited)
• Inventors: Tang, Xiaoou, Qiu, Shi, Wang, Xiaogang
• Primary Examiner(s): Spieler, William
• Application Number:
  US15/100,152
• Publication Number:
  US 20170046427A1
• Time in Patent Office: 1,998 Days
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

A visual semantic complex network system and a method for generating the system have been disclosed. The system may comprise a collection device configured to retrieve a plurality of images and a plurality of texts associated with the images in accordance with given query keywords; a semantic concept determination device configured to determine semantic concepts of the retrieved images and retrieved texts for the retrieved images, respectively; a descriptor generation device configured to, from the retrieved images and texts, generate text descriptors and visual descriptors for the determined semantic concepts; and a semantic correlation device configured to determine semantic correlations and visual correlations from the generated text and visual descriptor, respectively, and to combine the determined semantic correlations and the determined visual correlations to generate the visual semantic complex network system.

Claims

1. A visual semantic complex network system, comprising:

a memory configured for storing executable units; and

a processor electronically communicated with the memory and executing the executable units to;

retrieve a plurality of images and a plurality of texts associated with the plurality of images based on one or more query-keywords;

determine semantic concepts and representative images corresponding to the retrieved texts and the retrieved images, respectively;

generate text descriptors and visual descriptors based on the determined semantic concepts and representative images, wherein the text descriptors are generated as one or more vectors comprising a frequency of the semantic concepts within the plurality of retrieved texts and wherein the visual descriptors are generated as one or more binary vectors comprising a hashed representation of each of the plurality of retrieved images;

determine semantic correlations and visual correlations associated with the semantic concepts based on the generated text and visual descriptors, respectively;

generate the visual semantic complex network system as a navigable graph based on combining the determined semantic correlations and the determined visual correlations, wherein the graph includes a plurality of nodes corresponding to the determined semantic concepts;

enter a query keyword to the visual semantic complex network system;

present a query space being a first subgraph of the visual semantic complex network system, the query space containing concepts each comprising the entered query keyword;

present a local concept space being a second subgraph of the visual semantic complex network system in response to selecting the concept in the query space as a centric concept, the local concept space comprising the centric concept and neighbor concepts excluding the entered keyword; and

present the query space in response to selecting the centric concept in the local concept space.

2. The system according to claim 1, wherein the visual semantic complex network system comprises a K-nearest-neighbor graph network system, K is an integer.

3. The system according to claim 2, wherein a semantic correlation and a visual correlation are generated between each two of the concepts, respectively.

4. The system according to claim 3, wherein the processor is configured to,

collect a plurality of text snippets corresponding to the semantic concepts,

determine a term frequency vector of the collected snippets;

truncate the determined vector such that a plurality of terms in the vector with a highest term frequency is maintained; and

normalize the truncated vectors to generate said text descriptors, such that the semantic correlation between each two of the semantic concepts are generated from the normalized vector.

5. The system according to claim 3, wherein the processor is configured to,

encode each of the representative images by a hashing function to generate a binary vector for each retrieved image,

accumulate the generated binary vectors;

and

quantize the accumulated vector back to a binary vector as said visual descriptor, such that the visual correlation between each two of the concepts are generated from the binary vector.

6. A method for forming a visual semantic complex network system for Web images, comprising:

retrieving a plurality of images and a plurality of texts associated with the plurality of images based on one or more query keywords;

determining semantic concepts and representative images of the retrieved texts and the retrieved images, respectively;

generating text descriptors and visual descriptors based on the semantic concepts and the representative images, wherein the text descriptors are generated as one or more vectors comprising a frequency of the semantic concepts within the plurality of retrieved texts and wherein the visual descriptors are generated as one or more binary vectors comprising a hashed representation of each of the plurality of retrieved images;

determining semantic correlations and visual correlations associated with of the semantic concepts based on the generated text descriptor and the generated visual descriptor, respectively;

generating the visual semantic complex network system as a navigable graph based on combining the semantic correlations and the visual correlations, wherein the graph includes a plurality of nodes corresponding to the determined semantic concepts;

entering a query keyword to the visual semantic complex network system;

presenting a query space being a first subgraph of the visual semantic complex network system, the query space containing concepts each comprising the entered query keyword;

presenting a local concept space being a second subgraph of the visual semantic complex network system in response to selecting the concept in the query space as a centric concept, the local concept space comprising the centric concept and neighbor concepts excluding the entered keyword; and

presenting the query space in response to selecting the centric concept in the local concept space.

7. The method according to claim 6, wherein the step of determining semantic correlations and visual correlations from the generated text descriptor and the generated visual descriptor comprises:

generating a semantic correlation and a visual correlation between each two of the semantic concepts.

8. The method according to claim 7, wherein the step of generating a semantic correlation between each two of the semantic concepts comprises:

collecting a plurality of text snippets corresponding to the semantic concepts,

determining a term frequency vector of the collected snippets;

truncating the determined-vector such that a plurality of terms in the vector with a highest term frequency is maintained; and

normalizing the truncated vectors to generate said text descriptors, the semantic correlation between each two of the semantic concepts are generated from the normalized vectors.

9. The method according to claim 7, wherein the step of determining the visual correlations comprises:

encoding each of the representative images by a hashing function so as to generate a binary vector for each retrieved image,

accumulating the generated binary vectors;

quantizing the accumulated vector back to a binary vector as said visual descriptor; and

determining the visual correlation between each two of the concepts from the binary vectors.

10. A method for searching images with a visual semantic complex network system, comprising:

entering a query keyword to the visual semantic complex network system;

presenting a query space being a first subgraph of the visual semantic complex network system, the query space containing concepts each comprising the entered query keyword;

presenting a local concept space being a second subgraph of the visual semantic complex network system in response to selecting the concept in the query space as a centric concept, the local concept space comprising the centric concept and neighbor concepts excluding the entered keyword;

presenting the query space in response to selecting the centric concept in the local concept space;

obtaining a list of images according to the query keyword;

determining a group of related concept communities from the obtained list;

determining a group of related semantic concepts from the determined communities;

generating visual descriptors based on the related semantic concepts and the determined communities, wherein the visual descriptors are generated as one or more binary vectors comprising a hashed representation of each image in the obtained list of images;

determining images of a top plurality of concepts based on the determined related semantic concepts and the visual descriptors;

forming a re-ranking pool of the determined images, which are matched with the query keyword.

11. A method for browsing images with semantic concepts, comprising:

entering a query keyword into a display system;

generating a plurality of semantic concepts each comprising the entered query keyword based on one or more queries matching the entered keyword;

generating visual descriptors based on the semantic concepts and the one or more queries, wherein the visual descriptors are generated as one or more binary vectors comprising a hashed representation of each image to be browsed;

visualizing the generated semantic concepts in a query space of the display system, based on the visual descriptors; and

switching the query space to a local concept space of the display unit in response to selecting a semantic concept selected from the query space as a centric concept, wherein on the local concept space the selected concept together with its neighbor concepts excluding the entered query keyword are shown;

presenting the query apace in response to selecting the centric concept in the local concept space.

12. The method according to claim 11, further comprising:

selecting another concept in the local concept space; and

switching to another local concept space where said another concept and its neighbor concepts are shown in response to selecting the another local concept.