System for collecting interest graph by relevance search incorporating image recognition system

• US 9,600,499 B2
• Filed: 06/23/2011
• Issued: 03/21/2017
• Est. Priority Date: 06/23/2011
• Status: Active Grant

• Current Assignee: Cyber AI Entertainment Inc.
• Original Assignee: Cyber AI Entertainment Inc.
• Inventors: Kutaragi, Ken, Usuki, Takashi, Yokote, Yasuhiko
• Primary Examiner(s): OBISESAN, AUGUSTINE KUNLE
• Application Number:
  US14/129,005
• Publication Number:
  US 20140149376A1
• Time in Patent Office: 2,098 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors

Abstract

A system for comprehensively and effectively acquiring, as an interest graph, targets and regions of interest unique to a user is provided. A system according to the invention is a search system using as input means image information containing various objects and subjects, in which by querying an image recognition engine on the server side via a network about a selected image, the image recognition engine extracts and recognizes in real time various generic objects and the like contained in the specified image, and notifies a relevance search engine on the server side of recognized image components contained in the input image, and the relevance search engine extracts related elements for the individual image components, and visually presents a relevance graph with the extracted related elements as nodes together with the depths of relationships between the nodes.

Claims

1. An interest graph collection system that is a search system using image information containing at least one of various objects and subjects as input, the system comprising:

one or more central processing units operable to execute one or more programs;

memory that stores the one or more programs to be executed by the one or more central processing units;

the one or more programs comprising instructions for;

selecting from among a large number of images existing on the Internet or on a dedicated network or images uploaded on the Internet by a number of users via a number of network terminals, an entire image or a specific region of an image in which the users have an interest;

extracting and recognizing in real time various objects, the objects comprising at least one of a generic object, a specific object, a person, a face, a scene, characters, a sign, an illustration, a logo, and a favicon contained in the selected entire image or the specific image region by an image recognition engine on a server side, when queried by the users about the selected image via the network; and

determining other related elements directly or indirectly related to one or more individual image components to a certain degree or more by a relevance search engine on the server side, when notified of image components contained in a recognized input image via the image recognition engine,

extracting the related elements based on multidimensional feature vectors describing direct relationships between elements held on a relevance knowledge database in the relevance search engine in a learnable manner, and

visually representing the image components recognized by the image recognition engine and the related elements extracted by the relevance search engine as nodes in a relevance graph together with one or more depths of relationships between the nodes on the respective network terminal of the users as a two-dimensional image, or a three-dimensional image with depth, or a four-dimensional spatiotemporal image to which a time axis variable as observation time of the relevance graph is added,

wherein, in a respective relevance search operation, by respective ones of the users tapping or touching on a touch screen an arbitrary node on the relevance graph displayed on the network terminal for selection, or moving a cursor of a pointer onto an arbitrary node for selection, or

by the respective user flicking on the touch screen toward an arbitrary region on the relevance graph, or moving a cursor of a pointer to an arbitrary region on the relevance graph and dragging and scrolling an entire image, or using a cursor key or the like for a similar operation, or using an input operation having a similar effect using a gesture, a line of sight, voice, or a brain wave,

the relevance search engine additionally transmits a new relevance graph centered on the selected node or the region after the move, including a course thereto, so that the respective user can visually recognize broad relationships between a plurality of nodes on the relevance graph, seamlessly tracing a node or a region of interest to the respective user,

wherein, in the relevance search operation, a more detailed relevance graph centered on a specific image component selected by the respective user from among a plurality of image components presented by the image recognition engine, or a specific node on the relevance graph displayed on the network terminal selected by the respective user double tapping or pinching out the node on the touchscreen, or operating a pointer or the like, enlarging a region centered on the node, or by the respective user using an input operation having a similar effect using a gesture, a line of sight, voice, or a brainwave, can be visually represented on the network terminal of the user;

the series of operations being considered to show a certain degree of interest of the respective user to the node, a feature vector value representing the depth of interest of the respective user in the node is adaptively increased on a multidimensional feature vector describing direct relationships between elements with the respective user as the center node to allow an interest graph corresponding to an individual user of the one of the one or more users, with the individual user as the center node to be acquired; and

the interest graph can be expanded to a wide range of the one or more users for acquisition to be collected as a statistical broad interest graph over a specific user cluster or all of the users,

wherein, in the relevance search operation, allowing reflection of a subjective evaluation of the individual user is enabled with respect to a multidimensional feature vector that describes direct relationships between elements held as the relevance knowledge database in the relevance search engine; and

wherein, based on the modified multidimensional feature vector, causes the respective network terminal of each of at least two of the users to represent the same one or two nodes and visually represents the mutual relationships between the nodes and the depths of the relationships as a relevance graph reflecting a factor of a difference in perception wherein each of said at least two of the users has the difference in the perception.

2. The interest graph collection system according to claim 1, wherein, in the relevance search operation, by the user, not tracing a selected node of interest on the relevance graph, querying again the image recognition engine on the server side about the images of the node via the network, new image components related to the node are acquired with the assistance of the image recognition engine, and new related elements starting from the image components are transmitted from the relevance search engine to the network terminal, so that the user can visually recognize new relationships to the node together with the depths of the mutual relationships on a relevance graph;

and

the relevance search engine presumes that the user recognizes and uses the existence of relationships between a series of nodes leading to the node from the image component as the starting point to the node in the last similar operation, and adaptively increases feature vector values representing the depths of direct relationships between nodes constituting the series of relationships on the multidimensional feature vector describing direct relationships between elements to allow the relevance knowledge database in the relevance search engine to learn additionally.

3. The interest graph collection system according to claim 1, wherein, in the relevance search operation, for the image components that can be recognized by the image recognition engine and the related elements associated with the image components, reduced image thumbnails generated from a photograph, an illustration, characters, a sign, a logo, a favicon, and the like representing the image components and the related elements are transmitted to the network terminal in place of the original image by the relevance search engine, so that nodes on a relevance graph can be displayed and selected in units of image thumbnails.

4. The interest graph collection system according to claim 1, wherein, in the relevance search operation, it is made possible to query the image recognition engine on the server side about a plurality of nodes;

and

as input condition selection functions included in an image recognition process, logical operators (AND and OR) are introduced, so that when AND is selected, a node commonly and directly related to the nodes, and when OR is selected, a node directly related to at least one of the nodes can be visually represented on the network terminal together with the depth of the mutual relationship.

5. The interest graph collection system according to claim 1, wherein, in the relevance search operation, it is made possible to query the image recognition engine on the server side about a plurality of nodes;

as an input condition selection function included in an image recognition process, a connection search operator (Connection Search) is introduced, so that a connection between the plurality of nodes that seems to have no connection at all is searched for as a series of connections via other nodes directly and indirectly related to their respective input nodes, to detect an indirect relationship between the node across different layers (classes); and

the nodes can be displayed on the network terminal in a relevance graph including the shortest path between the nodes, and at the same time, in the connection search process, the detected indirect relationship between the plurality of nodes is learned and acquired in the relevance knowledge database in the relevance search engine to be prepared for the same or a similar connection search request afterward.

6. The interest graph collection system according to claim 1, wherein, in the relevance search operation, a connection operator (LIKE) for connecting a node indirectly related to the user or a node regarded as having no relevance to the user and the user as a direct relationship, and a disconnection operator (DISLIKE) for cutting the direct relationship between the node already connected and the user are introduced, so that a value representing the depth of interest of the user in the node is increased, reduced, or erased on a multidimensional feature vector describing the direct relationships between the elements with the user as the center node to update the interest graph corresponding to the individual user with the user as the center node.

7. The interest graph collection system according to claim 1, wherein, in the relevance search operation, for the possibility of existence and non-existence of a new direct relationship for a node other than the user, a reference operator (REFERENCE) for presenting that the nodes should be directly connected, and an unreference operator (UNREFERENCE) for presenting non-existence of a direct relationship as the existence of the direct relationship of the node already directly connected is doubtful are introduced to allow the relevance search engine to draw attention of the user to the possibility of existence or non-existence of the new direct relationship between the nodes;

and

the relevance search engine can update the value of a feature vector on a relationship between nodes judged to be related or unrelated by a supervisor having specific authority or more than a fixed number of users; and

the update can be reflected as an updated relevance graph for the nodes on the network terminal, and all users can be notified of update information on the existence or non-existence of the new direct relationship.

8. The interest graph collection system according to claim 1, wherein,

the network terminal of each of at least two of the users to represent at least the same two nodes and reflecting a factor of difference in perception that the individual user has on the network terminal of the user in a relevance graph includes rendering a visual effect for a first one of the two nodes for a first individual user and rendering a visual effect for a second one of the two nodes for a second individual user, the second one of the two nodes different from the first one of the two nodes.

9. The interest graph collection system according to claim 1, wherein, in the relevance search operation, an environmental filter comprising at least one of the time zone, the date, the season, the period, and the location at which the user is going to conduct search can be applied to the multidimensional feature vector describing direct relationships between elements held as the relevance knowledge database in the relevance search engine;

and

based on the modified multidimensional feature vector, the mutual relationships between the nodes and the depths of the relationships can be represented as a relevance graph reflecting a spatiotemporal factor comprising at least one of observation time and location characteristics on the network terminal of the user.

10. The interest graph collection system according to claim 1, wherein, in the relevance search operation, when metadata is attached to the image itself corresponding to the node selected by the user, in order to verify the suitability of the metadata to the image or the correctness of the metadata itself, the image recognition engine compares various features extractable from the image and their feature quantities with specific features that an object or a subject represented by metadata has and their feature quantities on a priority basis, so that processing time for image recognition by the image recognition engine can be shortened, and metadata that is unrelated to or does not show relevance to a certain degree or more to the image can be excluded.

11. The interest graph collection system according to claim 1, wherein, through the relevance search process by the user, when a visual interest or a transition of interest over a specific user cluster or a group of all users to a node other than a person changes to a certain degree or more, a significant change in the degree of interest in the node is made into statistical information, and a user, a user cluster, all users, or a specific third person directly related to the nodes can be promptly notified of the information on the relevance graph.

12. The interest graph collection system according to claim 1, wherein, as for the display of the relevance graph, the display of an interest graph with the user himself or herself as the center node is limited to the user in terms of privacy protection.

13. The interest graph collection system according to claim 1, wherein, utilizing the interest graph, to a specific user or a specific user group who shows an interest to a certain degree or more in a product or service represented by a specific node, direct presentation of an advertisement or a recommendation arousing consumer demand for the product or service, and linking to a third party supplying the product or service are enabled, and at the same time, utilizing the relevance graph to which the attribute of the user or a factor of the space and time to which the user belongs is added, direct presentation of an additional advertisement or a recommendation for other products or services directly or indirectly related to the attribute or the space and time, and linking to third parties supplying the other products or services are enabled.

14. The interest graph collection system according to claim 1, wherein visual information showing an advertisement, a service, or a recommendation and/or link information that can be presented by utilizing the interest graph can be presented on the relevance graph displayed on a network terminal of a user as a target in a manner that display/non-display is selectable.

15. The interest graph collection system according to claim 1, wherein utilizing an interest graph acquired through a visual relevance search process incorporating the image recognition engine, communications between a wide range of user having similar interests are aroused through the relevance search, so that, in addition to the interest graph, a broad dynamic social graph including relationships between people and people between people and objects other than people can be acquired via the network.