Information retrieving system, information retrieving method, information retrieving apparatus, information retrieving program, image recognizing apparatus image recognizing method image recognizing program and sales

• US 8,458,038 B2
• Filed: 01/27/2005
• Issued: 06/04/2013
• Est. Priority Date: 01/29/2004
• Status: Active Grant

• Current Assignee: ZETA Bridge Corporation
• Original Assignee: ZETA Bridge Corporation
• Inventors: Ando, Naotaka
• Primary Examiner(s): Danneman, Paul
• Application Number:
  US10/597,168
• Publication Number:
  US 20080279481A1
• Time in Patent Office: 3,050 Days
• Field of Search: None
• US Class Current: 705/26.1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 2201/09 Recognition of logos

Abstract

Steps after a user finds an attractive product until he or she buys it are successively performed so that he or she can buy it stressfreely. The user photographs an attractive product with a portable camera telephone terminal 1 and transmits image data 7 to a server 3 through Internet 2. The server 3 has a DB 4 in which images such as various types of products, persons, and logo marks are registered and a DB 5 in which related information of image data registered in the DB 4 is registered. The server 3 performs an image recognizing process for the image data 7, retrieves data that match the image data 7 with a high match rate from the image data registered in the DB 4, and extracts corresponding related information from the DB 5. The related information is provided to the user through the Internet 2. By only transmitting the image data 7, the user can obtain related information of an attractive product. When the user accesses a site 6 for the product corresponding to the related information, he or she can obtain detailed information and buy the product.

Claims

1. An information retrieving system which retrieves information according to image data and moving image data, the information retrieving system comprising:

a terminal unit having;

image data obtaining means for obtaining image data and moving image data, and

transmitting means for transmitting the image data and the moving image data obtained by the image data obtaining means to a communication network, and

a server unit having;

a first database in which retrievable recognizing image data is registered,

a second database in which meta data corresponding to the recognizing image data registered in the first database is registered,

an automatic extracting system that automatically detects and extracts information broadcast on television broadcasts,

a registering section that collects the information broadcast on the television broadcasts and information published on web sites and registers the information as the retrievable recognizing image data to the first database, and collects meta data corresponding to the recognizing image data registered in the first database and registers the meta data to the second database,

receiving means for receiving the image data and the moving image data transmitted from the terminal unit through the network,

a feature extracting section which extracts a feature of image data of which the image data and the moving image data received by the receiving means are broken down frame by frame,

a feature comparing section which extracts a feature of the recognizing image data registered in the first database and compares the feature of the image data of which the image data and the moving image data are broken down frame by frame with a feature of the recognizing image data,

retrieving means for retrieving the recognizing image data having a high match rate to an image element of the image data from the first database according to a compared result by the feature comparing section, and

meta data extracting means for extracting the meta data from the second database according to a retrieved result by the retrieving means,

wherein the server unit allows the terminal unit to browse the meta data extracted by the meta data extracting means through the communication network.

2. The information retrieving system as set forth in claim 1,

wherein the retrieving means calculates a score of the match rate to the recognizing image data according to the feature data of the image data transmitted from the terminal unit and retrieves the recognizing image data having the score which is a predetermined value or greater as image data having the high match rate, and

wherein the retrieving means retrieves the recognizing image data included in individual pieces of the image data of which the image data and the moving image data are broken down frame by frame at a predetermined rate or greater as image data having the high match rate.

3. The information retrieving system as set forth in claim 2,

wherein the server unit side obtains the feature data.

4. The information retrieving system as set forth in claim 2,

wherein the terminal unit pre-obtains the feature data of each piece of the image data of which the image data and the moving image data obtained by the image data obtaining means are broken down frame by frame and causes the transmitting means to transmit the feature data as the image data.

5. The information retrieving system as set forth in claim 1, wherein the automatic extracting system automatically detects and extracts commercials from the television broadcasts.

6. The information retrieving system as set forth in claim 1, wherein the registering section collects the meta data from a different source than where the corresponding recognizing image data is collected.

7. The information retrieving system as set forth in claim 1, wherein the feature comparing section calculates a positional relationship of a plurality of feature points.

8. The information retrieving system as set forth in claim 1,

wherein the retrieving means retrieves the recognizing image data according to data including either gray-scaled data of each piece of the image data of which the image data and the moving image data obtained by the image data obtaining means are broken down frame by frame or color information of each piece of the image data of which the image data and the moving image data obtained by the image data obtaining means are broken down frame by frame.

9. The information retrieving system as set forth in claim 1,

wherein the retrieving means outputs retrieved results corresponding to a plurality of image elements included in each piece of the image data of which the image data and the moving image data received by the receiving means are broken down frame by frame, and

wherein the meta data is extracted from the second database according to a retrieved result of a part of a plurality of image elements and other image elements of the plurality of image elements are retrieved from the first database according to the extracted meta data.

10. The information retrieving system as set forth in claim 9,

wherein the retrieving means has a first recognizing means for recognizing a person, a second recognizing means for recognizing characters, and a third recognizing means for recognizing a shape other than the person and the characters, and

wherein the first, second, and third recognizing means perform recognizing processes for each piece of the image data of which the image data and the moving image data are broken down frame by frame and retrieve the person, the characters, and the shape other than the person and the characters included in individual pieces of the image data of which the image data and the moving image data are broken down frame by frame as image elements from the recognizing image data.

11. The information retrieving system as set forth in claim 1,

wherein the server unit transmits an address on the communication network for the meta data extracted by the meta data extracting means to the terminal unit and provides the meta data to the terminal unit on the communication network.

12. The information retrieving system as set forth in claim 1,

wherein the server unit transmits the meta data extracted by the meta data extracting means to the terminal unit through the communication network.

13. The information retrieving system as set forth in claim 1,

wherein when the meta data is browsed on the terminal unit, an address on the communication network relating to the meta data is allowed to be accessed from the terminal unit.

14. The information retrieving system as set forth in claim 1,

wherein the terminal unit is a portable telephone terminal having a camera function.

15. An information retrieving method of a retrieving unit which retrieves information according to image data and moving image data which a terminal unit obtains, the information retrieving method comprising steps of:

causing the terminal unit to obtain the image data and the moving image data;

causing the terminal unit to transmit the image data and the moving image data obtained at the image data obtaining step from the terminal unit to a communication network;

causing the retrieving unit to receive the image data and the moving image data transmitted at the transmitting step through the network;

causing the retrieving unit to automatically detect and extract information broadcast on television broadcasts,

causing the retrieving unit to collect the information broadcast on the television broadcasts and information published on web sites and register the information as retrievable recognizing image data to a first database, and collect meta data corresponding to the recognizing image data registered in the first database and register the meta data to a second database;

causing the retrieving unit to extract a feature of image data of which the image data and the moving image data received at the receiving step are broken down frame by frame, extract a feature of the recognizing image data registered in the first database, and retrieve the recognizing image data having a high match rate to an image element of the image data from the first database in which retrievable recognizing image data is registered according to a compared result of a feature of each piece of the image data of which the image data and the moving image data are broken down frame by frame with a feature of the recognizing image data; and

causing the retrieving unit to extract the meta data from the second database in which the meta data corresponding to the recognizing image data registered in the first database according to a retrieved result at the retrieving step,

wherein the retrieving unit allows the terminal unit to browse the meta data extracted at the meta data extracting step through the communication network.

16. An information retrieving unit which retrieves information according to image data, the information retrieving unit comprising:

a first database in which retrievable recognizing image data is registered;

a second database in which meta data corresponding to the recognizing image data registered to the first database is registered;

an automatic extracting system that automatically detects and extracts information broadcast on television broadcasts,

a registering section that collects the information broadcast on the television broadcasts and information published on web sites and registers the information as the retrievable recognizing image data to the first database, and collects meta data corresponding to the recognizing image data registered in the first database and registers the meta data to the second database;

receiving means for receiving image data and moving image data transmitted from a terminal unit through a communication network;

a feature extracting section which extracts a feature of each piece of image data of which the image data and the moving image data received by the receiving means are broken down frame by frame;

a feature comparing section which extracts a feature of the recognizing image data registered in the first database and compares the feature of the image data of which the image data and the moving image data are broken down frame by frame with the feature of the recognizing image data;

retrieving means for retrieving the recognizing image data having a high match rate to an image element of the image data from the first database according to a compared result by the feature comparing section; and

meta data extracting means for extracting the meta data from the second database according to a retrieved result by the retrieving means,

wherein the terminal unit is allowed to browse the meta data extracted by the meta data extracting means through the communication network.

17. An information retrieving method of a retrieving unit which retrieves information according to image data, the information retrieving method comprising the steps of:

causing the retrieving unit to automatically detect and extract information broadcast on television broadcasts,

causing the retrieving unit to collect the information broadcast on the television broadcasts and information published on web sites and register the information as retrievable recognizing image data to a first database, and collect meta data corresponding to the recognizing image data registered in the first database and registers the meta data to a second database;

causing the retrieving unit to receive image data and moving image data transmitted from a terminal unit through a communication network;

causing the retrieving unit to extract a feature of image data of which the image data and the moving image data received at the receiving step are broken down frame by frame, extract a feature of the recognizing image data registered in the first database, and retrieve the recognizing image data having a high match rate to an image element of the image data from the first database in which the retrievable recognizing image data is registered according to a compared result of the feature of the image data of which the image data and the moving image data are broken down frame by frame with the feature of the recognizing image data; and

causing the retrieving unit to extract the meta data from the second database in which the meta data corresponding to the recognizing image data registered in the first database according to a retrieved result at the retrieving step,

wherein the retrieving unit allows the terminal unit to browse the meta data extracted at the meta data extracting step through the communication network.

18. A non-transitory recording medium storing an information retrieving program which causes a computer unit to execute an information retrieving method of retrieving information according to image data, the information retrieving method comprising the steps of:

causing the retrieving unit to automatically detect and extract information broadcast on television broadcasts,

causing the retrieving unit to collect the information broadcast on the television broadcasts and information published on web sites and register the information as retrievable recognizing image data to a first database, and collect meta data corresponding to the recognizing image data registered in the first database and registers the meta data to a second database;

causing the retrieving unit to receive image data and moving image data transmitted from a terminal unit through a communication network;

causing the retrieving unit to extract a feature of image data of which the image data and the moving image data received at the receiving step are broken down frame by frame, extract a feature of the recognizing image data registered in the first database, and retrieve the recognizing image data having a high match rate to an image element of the image data from the first database in which the retrievable recognizing image data is registered according to a compared result of the feature of the image data of which the image data and the moving image data are broken down frame by frame with the feature of the recognizing image data; and

causing the retrieving unit to extract the meta data from the second database in which the meta data corresponding to the recognizing image data registered in the first database according to a retrieved result at the retrieving step,

wherein the retrieving unit allows the terminal unit to browse the meta data extracted at the meta data extracting step through the communication network.

19. An image recognizing unit which recognizes a predetermined shape from image data, comprising:

a first database in which retrievable recognizing image data is registered;

a second database in which meta data of the recognizing image data registered in the first database is registered;

an automatic extracting system that automatically detects and extracts information broadcast on television broadcasts,

a registering section that collects the information broadcast on the television broadcasts and information published on web sites and registers the information as the retrievable recognizing image data to the first database, and collects the meta data corresponding to the recognizing image data registered in the first database and registers the meta data to the second database;

a feature extracting section which extracts a feature of image data of which input image data and input moving image data are broken down frame by frame;

a feature comparing section which extracts a feature of the recognizing image data registered in the first database and compares the feature of the image data of which the image data and the moving image data are broken down frame by frame with the feature of the recognizing image data;

retrieving means for retrieving the recognizing image data having a high match rate to an image element of the image data from the first database according to a compared result by the feature comparing section; and

meta data extracting means for extracting the meta data from the second database according to a retrieved result by the retrieving means.

20. The image recognizing unit as set forth in claim 19,

wherein the retrieving means calculates a score of a match rate to the recognizing image data according to feature data of the input image data and retrieves the recognizing image data having the score which is a predetermined value or greater as the image data having the high match rate, and

wherein the recognizing image data included in each piece of the image data of which the moving image data are broken down frame by frame at a predetermined rate or greater are the retrieved result corresponding to the moving image data.

21. The image recognizing unit as set forth in claim 19,

wherein the retrieving means retrieves the recognizing image data according to data including either gray-scaled data of each piece of the image data of which the input image data and the moving image data are broken down frame by frame or color information of each piece of the image data of which the input image data and the moving image data are broken down frame by frame.

22. The image recognizing unit as set forth in claim 19,

wherein the retrieving means outputs retrieved results corresponding to a plurality of image elements included in individual pieces of the image data of which the image data and the moving image data are broken down frame by frame, and

wherein the meta data is extracted from the second database according to a retrieved result corresponding to a part of the plurality of image elements and other image elements of the plurality of image elements are retrieved from the first database according to the extracted meta data.

23. The image recognizing unit as set forth in claim 22,

wherein the retrieving means has a first recognizing means for recognizing a person, a second recognizing means for recognizing characters, and a third recognizing means for recognizing a shape other than the person and the characters, and

wherein the first, second, and third recognizing means perform recognizing processes for each piece of the image data of which the image data and the moving image data are broken down frame by frame and retrieve the person, the characters, and the shape other than the person and the characters included in individual pieces of the image data of which the image data and the moving image data are broken down frame by frame as the image elements from the recognizing image data.

24. The image recognizing unit as set forth in claim 22,

wherein the meta data is extracted from the second database according to the retrieved result corresponding to the part of the plurality of image elements included in individual pieces of the image data of which the image data and the moving image data are broken down frame by frame and the other image elements of the plurality of image elements are retrieved from the first database according to the extracted meta data.

25. An image recognizing method of recognizing a predetermined shape from image data, the image recognizing method comprising the steps of:

causing a retrieving unit to automatically detect and extract information broadcast on television broadcasts,

causing the retrieving unit to collect the information broadcast on the television broadcasts and information published on web sites and register the information as retrievable recognizing image data to a first database, and collect meta data corresponding to the recognizing image data registered in the first database and register the meta data to a second database;

causing the retrieving unit to extract a feature of image data of which input image data and input moving image data are broken down frame by frame, extract a feature of the recognizing image data registered in the first database, and retrieve the recognizing image data having a high match rate to an image element of the image data from the first database in which the retrievable recognizing image data is registered according to a compared result of the feature of the image data of which the image data and the moving image data are broken down frame by frame with the feature of the recognizing image data; and

causing the retrieving unit to extract the meta data from the second database in which the meta data of the recognizing image data registered in the first database is registered according to a retrieved result at the retrieving step.

26. A non-transitory recording medium storing an image recognizing program which causes a computer unit to execute an image recognizing method of recognizing a predetermined shape from image data, the image recognizing method comprising the steps of:

causing a retrieving unit to automatically detect and extract information broadcast on television broadcasts,

causing the retrieving unit to collect the information broadcast on the television broadcasts and information published on web sites and register the information as retrievable recognizing image data to a first database, and collect meta data corresponding to the recognizing image data registered in the first database and register the meta data to a second database;

causing the retrieving unit to extract a feature of image data of which input image data and input moving image data are broken down frame by frame, extract a feature of the recognizing image data registered in the first database, and retrieve the recognizing image data having a high match rate to an image element of the image data from the first database in which the retrievable recognizing image data is registered according to a compared result of the feature of the image data of which the image data and the moving image data are broken down frame by frame with the feature of the recognizing image data; and

causing the retrieving unit to extract the meta data from the second database in which the meta data of the recognizing image data registered in the first database is registered according to a retrieved result at the retrieving step.