System and method for providing additional information using image matching

• US 9,754,183 B2
• Filed: 11/21/2016
• Issued: 09/05/2017
• Est. Priority Date: 11/16/2012
• Status: Active Grant

• Current Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Original Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Inventors: Cho, Hoon-young, Cho, Min-hyung, Lee, Jaehyung
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US15/358,117
• Publication Number:
  US 20170083786A1
• Time in Patent Office: 288 Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04N 21/278 Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content

Abstract

The present invention relates to a system and method for providing additional information using image matching. The present invention provides a system for providing additional information using image matching, including: a image input unit configured to receive a video or an image input from a user terminal; a query image determination unit configured to determine a query image, i.e., a query target, from the video or image received from the image input unit; a feature point extraction unit configured to extract feature points of the query image determined by the query image determination unit, and to generate feature point information of the extracted feature points; a reference meta-information database configured to store a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information to be provided to the user terminal in association with the structure image ID; an image matching unit configured to determine a matching structure image ID by comparing the feature point information of the query image generated by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database; and an additional information provision unit configured to provide the additional information stored in association with the structure image ID determined by the image matching unit to the user terminal, and a method using the same.

Claims

1. A server, comprising:

a reference meta-information database configured to store at least a structure image identifier of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information associated with the structure image ID, and

wherein the server configured to perform functions comprising;

receiving a query image;

extracting one or more feature points from the query image;

generating feature point information based on the one or more feature points;

determining a matching structure image ID by comparing the feature point information with the structure image feature point information stored in the reference meta-information database;

providing additional information associated with the matching structure image ID.

2. The system of claim 1, wherein the query image is related to an input video, and wherein receiving the query image comprises:

receiving the input video;

extracting a plurality of frame images from the input video; and

determining the query image from the plurality of frame images.

3. The system of claim 2, wherein determining the query image from the plurality of frame images comprises determining the query image based on one or more gradient values of the plurality of frame images.

4. The system of claim 3, wherein determining the query image based on the one or more gradient values of the plurality of frame images comprises determining one or more scores based on a score function operating on the one or more gradient values.

5. The system of claim 1, wherein the structure image feature point information comprises feature point information inside an original image region and feature point information outside the original image region.

6. The system of claim 5, wherein the feature point information outside the original image region is formed by feature point information that does not match the original image feature point information.

7. The system of claim 6, wherein the functions further comprise:

determining one or more highest structure image IDs of highest matching scores by at least determining a structure image ID of the one or more highest structure image IDs based on a comparison of the feature point information outside the original image region corresponding to a highest structure image ID of the one or more highest structure image IDs with the feature point information of the query image.

8. The system of claim 7, wherein receiving the query image comprises receiving location information with the query image, and wherein the reference meta-information database further stores at least a structure location related to the original image.

9. The system of claim 1, wherein the functions further comprise:

determining first directional angle information of a user terminal directed toward the structure image.

10. The system of claim 9, wherein the functions further comprise:

receiving one or more second images;

determining a second query image from the one or more second images;

extracting one or more second feature points from the second query image;

generating second feature point information based on the one or more second features points;

determining that the matching structure image ID is associated with the second image based on a comparison of the second feature point information with the structure image feature-point information;

determining second directional angle information of the user terminal based on the one or more second images; and

determining a direction of movement of the user terminal based on the first directional angle information and the second directional angle information.

15. The method of claim 1, wherein the structure image feature point information comprises feature point information inside an original image region and feature point information outside the original image region.

16. The method of claim 15, wherein the feature point information outside the original image region is formed by feature point information that does not match the original image feature point information.

17. The method of claim 15, further comprising:

determining one or more highest structure image IDs of highest matching scores by at least determining a structure image ID of the one or more structure image IDs based on a comparison of the feature point information outside the original image region corresponding to a highest structure image ID of the one or more highest structure image IDs with the feature point information of the query image.

18. The method of claim 17, wherein receiving the query image comprises receiving location information with the query image, and wherein the method further comprises

receiving location information with the one or more images;

and

storing at least a structure location related to the original image.

19. The method of claim 1, further comprising:

determining first angle information of a user terminal directed toward the structure image.

20. The method of claim 19, further comprising:

receiving one or more second images;

determining a second query image from the one or more second images;

extracting one or more second feature points from the second query image;

generating second feature point information based on the one or more second features points;

determining that the matching structure ID is associated with the second image based on a comparison of the second feature point information with the structure image feature-point information;

determining second directional angle information of the user terminal based on the one or more second images; and

determining a direction of movement of the user terminal based on the first directional angle information and the second directional angle information.

11. A method, comprising:

receiving a query image;

extracting one or more feature points from the query image;

storing at least a structure image identifier of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information associated with the structure image ID;

generating feature point information based on the one or more feature points;

determining a matching structure image ID by comparing the feature point information with the structure image feature point information; and

providing the additional information associated with the matching structure image ID.

12. The method of claim 11, wherein the query image is related to an input video, and wherein receiving the query image comprises:

receiving the input video;

extracting a plurality of frame images from the input video; and

determining the query image from the plurality of frame images.

13. The method of claim 12, where receiving the query image comprises determining the query image based on one or more gradient values of the plurality of frame images.

14. The method of claim 13, wherein determining the query image based on the one or more gradient values of the plurality of frame images comprises determining one or more scores based on a score function operating on the one or more gradient values.