SYSTEM AND METHOD FOR PROVIDING ADDITIONAL INFORMATION USING IMAGE MATCHING

• US 20150286894A1
• Filed: 03/04/2013
• Published: 10/08/2015
• Est. Priority Date: 11/16/2012
• Status: Active Grant

• Current Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Original Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Inventors: Cho, Hoon-young, Cho, Min-hyung, Lee, Jaehyung
• Granted Patent: US 9,536,175 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04N 21/278 Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content

Abstract

The present invention relates to a system and method for providing additional information using image matching. The present invention provides a system for providing additional information using image matching, including: a image input unit configured to receive a video or an image input from a user terminal; a query image determination unit configured to determine a query image, i.e., a query target, from the video or image received from the image input unit; a feature point extraction unit configured to extract feature points of the query image determined by the query image determination unit, and to generate feature point information of the extracted feature points; a reference meta-information database configured to store a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information to be provided to the user terminal in association with the structure image ID; an image matching unit configured to determine a matching structure image ID by comparing the feature point information of the query image generated by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database; and an additional information provision unit configured to provide the additional information stored in association with the structure image ID determined by the image matching unit to the user terminal, and a method using the same.

Claims

1. A system for providing additional information using image matching, comprising:

a image input unit configured to receive a video or an image input from a user terminal;

a query image determination unit configured to determine a query image, i.e., a query target, from the video or image received from the image input unit;

a feature point extraction unit configured to extract feature points of the query image determined by the query image determination unit, and to generate feature point information of the extracted feature points;

a reference meta-information database configured to store a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information to be provided to the user terminal in association with the structure image ID;

an image matching unit configured to determine a matching structure image ID by comparing the feature point information of the query image generated by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database; and

an additional information provision unit configured to provide the additional information stored in association with the structure image ID determined by the image matching unit to the user terminal.

2. The system of claim 1, wherein the query image determination unit comprises:

a frame extraction unit configured to extract the input video as a plurality of frame images when the video is input from the user terminal; and

a query image selection unit configured to determine the query image, i.e., the query target, from among the frame images extracted by the frame extraction unit.

3. The system of claim 2, wherein the query image selection unit determines the query image based on gradient values of the frame images.

4. The system of claim 3, wherein the query image is determined based on scores that are calculated by the following equation:

5. The system of claim 1, wherein:

the reference meta-information database further comprises an original image ID of the original image and original image feature point information of the original image; and

the structure image feature point information comprises feature point information inside an original image region and feature point information outside the original image region.

6. The system of claim 5, wherein the feature point information outside the original image region is formed by feature point information that does not match the original image feature point information when matching between the original image feature point information and the structure image feature point information is performed.

7. The system of claim 6, wherein the image matching unit determines N (where N is a natural number) structure image IDs of highest matching scores by comparing the feature point information of the query image extracted by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database, and finally determines a structure image ID by comparing the feature point information outside the original image region corresponding to the determined structure image IDs with the feature point information of the query image.

8. The system of claim 6, wherein:

the image input unit receives location information of the user terminal from the user terminal along with the video or image;

the reference meta-information database comprises structure location information stored in association with the structure image ID of the structure on which the original image has been installed; and

the image matching unit determines N (where N is a natural number) structure image IDs of highest matching scores by comparing the feature point information of the query image extracted by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database, and finally determines a structure image ID by comparing the feature point information outside the original image region corresponding to the determined structure image IDs with the feature point information of the query image.

9. The system of claim 1, further comprising a direction calculation unit configured to determine first directional angle information of the user terminal directed toward the structure image.

10. The system of claim 9, wherein:

the image input unit additionally receives a video or an image input from the user terminal within a specific time interval;

the query image determination unit, the feature point extraction unit and the image matching unit determine a query image, i.e., a query target, from the additionally received video or image, extract feature points, and then determine a matching structure image ID;

the direction calculation unit determines second directional angle information of the user terminal directed toward the structure image, and determines a direction of movement of the user terminal based on the first directional angle information and the second directional angle information; and

the additional information provision unit provides the stored additional information to the user terminal based on the structure image ID determined by the image matching unit and the direction of movement of the determined user terminal.

11. A method of providing additional information using image matching, comprising:

a first step of receiving a video or an image input from a user terminal;

a second step of determining a query image, i.e., a query target, from the received video or image;

a third step of extracting feature points of the determined query image and generating feature point information of the extracted feature points;

a fourth step of determining a matching structure image ID by comparing the feature point information of the feature points of the extracted query image with structure image feature point information; and

a fifth step of providing additional information, stored in association with the determined structure image ID, to the user terminal.