Information processing system, information processing method, communications terminals and control method and control program thereof

• US 10,115,127 B2
• Filed: 12/12/2012
• Issued: 10/30/2018
• Est. Priority Date: 12/16/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nomura, Toshiyuki, Yamada, Akio, Iwamoto, Kota, Mase, Ryota
• Primary Examiner(s): Macasiano, Marilyn G
• Application Number:
  US14/365,955
• Publication Number:
  US 20140310097A1
• Time in Patent Office: 2,148 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0252 based on events or environment, e.g. weather or festivals
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

Advertisement information relating to an object is provided in real time, while capturing images of the object. m first local features which are respectively feature vectors from one dimension to i dimensions are stored in association with an object, n feature points are extracted from a video picture, n second local features which are respectively feature vectors from one dimension to j dimensions are generated, the smaller number of dimensions is selected, of the number of dimensions i and the number of dimensions j, and an object is recognized to be present in the video picture and advertisement information relating to that object is provided when determination is made that at least a prescribed ratio of the m first local features of the selected number of dimensions corresponds to the n second local features of the selected number of dimensions.

Claims

1. An information processing system, comprising:

a communications terminal, the communications terminal comprising

a processor comprising a second local feature generation device which extracts n feature points from a video picture and generates n second local features which are respectively feature vectors from one dimension to j dimensions, in respect of n local regions containing each of the n feature points, and

a terminal communications control unit for sending the n second local features; and

an advertisement delivery server in communication with the communications terminal, the advertisement delivery server comprising

a server communications control unit for receiving the n second local features,

a first local feature storage which stores, in association with an object, m first local features that are generated in respect of each of m local regions containing each of m feature points in an image of the object and that are respectively feature vectors from one dimension to i dimensions, and

a processor which comprises

a recognition device which selects a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local features and the number of dimensions j of the feature vectors of the second local features, and recognizes that the object is present in the video picture when determination is made that at least a prescribed ratio of the m first local features which are feature vectors up to the selected number of dimensions corresponds to the n second local features which are feature vectors up to the selected number of dimensions, and

an advertisement information providing device which provides advertisement information relating to the object recognized by the recognition device, wherein the server communications control unit sends the advertisement information to the terminal communications control unit of the communications terminal,

wherein the first local features and the second local features are generated by dividing into a plurality of sub-regions the local regions containing feature points extracted from an image or video picture and by generating feature vectors of a plurality of dimensions which are histograms of gradient directions in the plurality of sub-regions,

wherein the first local features and the second local features are generated by selecting the dimensions having a greater correlation between adjacent sub-regions among the generated feature vectors of a plurality of dimensions.

2. The information processing system according to claim 1, wherein

the first local feature storage device further stores advertisement information relating to the object after associating the information with the object;

and

the advertisement information providing device refers to the first local feature storage device and displays advertisement information relating to the object recognized by the recognition device.

3. The information processing system according to claim 1, further comprising an advertisement information addition device which adds related advertisement information to the image of the object in the video picture, when the recognition device has recognized that the object is present in the video picture.

4. The information processing system according to claim 1, wherein the advertisement information providing device displays, as an accessible link, a link to a site for purchasing a product that is the object in the video picture, with the link serving as the advertisement information.

5. The information processing system according to claim 1, wherein

the object is a storage medium storing content including at least one of music and a video picture, and

the advertisement information providing device displays, in audible and/or viewable fashion, a portion of content including at least one of the music and video picture as the advertisement information.

6. The information processing system according to claim 1, wherein the plurality of dimensions of the feature vectors are arranged in a cycle in the local region, for each prescribed number of dimensions, in such a manner that dimensions are selected sequentially from the dimensions contributing to the feature features of the feature points, and in such a manner that dimensions are selected sequentially from the first dimension, in accordance with increase in accuracy required in the local features.

7. The information processing system according to claim 6, wherein the second local feature generation device generates the second local features having a greater number of dimensions, in respect of an object having a higher correlation than another object, in accordance with the level of correlation between objects.

8. The information processing system according to claim 6, wherein the first local feature storage device stores the first local features having a greater number of dimensions, in respect of an object having a higher correlation than another object.

9. An information processing method, comprising:

a second local feature generation step of extracting by a processor of a communications terminal n feature points from a video picture and generating n second local features which are respectively feature vectors from one dimension to j dimensions, in respect of n local regions containing each of the n feature points;

a first communicating step of sending the n second local features from the communications terminal to an advertisement delivery server;

a reading step of reading out, from a first local feature storage device of the advertisement delivery server, m first local features which are respectively feature vectors from one dimension to i dimensions, with these quantities being stored in the first local feature storage device and generated previously in respect of each of m local regions containing each of m feature points in an image of an object;

a recognition step of selecting by a processor of the advertisement delivery server a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local features and the number of dimensions j of the feature vectors of the second local features, and recognizing that the object is present in the video picture when determination is made that at least a prescribed ratio of the m first local features which are feature vectors up to the selected number of dimensions corresponds to the n second local features which are feature vectors up to the selected number of dimensions;

an advertisement information providing step of providing by the processor of the advertisement delivery server advertisement information relating to the object recognized in the recognition step; and

a second communicating step of sending the advertisement information from the advertisement delivery server to the communications terminal,

wherein the first local features and the second local features are generated by dividing into a plurality of sub-regions the local regions containing feature points extracted from an image or video picture and by generating feature vectors of a plurality of dimensions which are histograms of gradient directions in the plurality of sub-regions,

wherein the first local features and the second local features are generated by selecting the dimensions having a greater correlation between adjacent sub-regions among the generated feature vectors of a plurality of dimensions.