System and method for providing additional information using image matching

• US 9,536,175 B2
• Filed: 03/04/2013
• Issued: 01/03/2017
• Est. Priority Date: 11/16/2012
• Status: Active Grant

• Current Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Original Assignee: Enswers Incorporated (RR Donnelley & Sons Company)
• Inventors: Cho, Hoon-young, Cho, Min-hyung, Lee, Jaehyung
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/441,310
• Publication Number:
  US 20150286894A1
• Time in Patent Office: 1,401 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04N 21/278 Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content

Abstract

The present invention relates to a system and method for providing additional information using image matching. The present invention provides a system for providing additional information using image matching, including: a image input unit configured to receive a video or an image input from a user terminal; a query image determination unit configured to determine a query image, i.e., a query target, from the video or image received from the image input unit; a feature point extraction unit configured to extract feature points of the query image determined by the query image determination unit, and to generate feature point information of the extracted feature points; a reference meta-information database configured to store a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information to be provided to the user terminal in association with the structure image ID; an image matching unit configured to determine a matching structure image ID by comparing the feature point information of the query image generated by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database; and an additional information provision unit configured to provide the additional information stored in association with the structure image ID determined by the image matching unit to the user terminal, and a method using the same.

Claims

1. A system, comprising:

an image input unit configured to receive one or more images;

a query image determination unit configured to determine a query image from the received one or more images;

a feature point extraction unit configured to extract one or more feature points from the query image, and to generate feature point information based on the one or more feature points;

a reference meta-information database configured to store at least a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information associated with the structure image ID;

an image matching unit configured to determine a matching structure image ID by comparing the feature point information with the structure image feature point information; and

an additional information provision unit configured to provide the additional information associated with the matching structure image ID.

2. The system of claim 1, wherein the one or more images are related to an input video, and wherein the query image determination unit comprises:

a frame extraction unit configured to extract a plurality of frame images from the input video; and

a query image selection unit configured to determine the query image from the plurality of frame images.

3. The system of claim 2, wherein the query image selection unit determines the query image based on one or more gradient values of the plurality of frame images.

4. The system of claim 3, wherein the query image is determined based

on one or more scores determined based on a score function that operates on the one or more gradient values.

5. The system of claim 1, wherein:

the structure image feature point information comprises feature point information inside an original image region and feature point information outside the original image region.

6. The system of claim 5, wherein the feature point information outside the original image region is formed by feature point information that does not match the original image feature point information.

7. The system of claim 6, wherein the image matching unit is further configured to determine one or more highest structure image IDs of highest matching scores by at least:

determining a structure image ID of the one or more highest structure image IDs based on a comparison of the feature point information outside the original image region corresponding to a highest structure image ID of the one or more highest structure image IDs with the feature point information of the query image.

8. The system of claim 7, wherein:

the image input unit receives location information with the one or more images; and

the reference meta-information database further stores at least a structure location related to the original image.

9. The system of claim 1, further comprising a direction calculation unit configured to determine first directional angle information of a user terminal directed toward the structure image.

10. The system of claim 9, wherein:

the image input unit is further configured to receive one or more second images;

the query image determination unit is further configured to determine a second query image, from the one or more second images;

the feature point extraction unit is further configured to extract one or more second feature points from the second query image and to generate second feature point information based on the one or more second feature points;

the image matching unit is further configured to determine that the matching structure image ID is associated with the second image based on a comparison of the second feature point information with the structure image feature-point information;

the direction calculation unit is further configured to determine second directional angle information of the user terminal based on the one or more second images, and to determine a direction of movement of the user terminal based on the first directional angle information and the second directional angle information; and

the additional information provision unit is further configured to provide the stored additional information to the user terminal based on the matching structure image ID and the direction of movement.

11. A method, comprising:

receiving one or more images;

determining a query image from the received one or more images;

extracting one or more feature points from the determined query image;

storing at least a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information associated with the structure image ID;

generating feature point information based on the one or more feature points;

determining a matching structure image ID by comparing the feature point information with the structure image feature point information; and

providing the additional information associated with the matching structure image ID.

12. The method of claim 11, wherein the one or more images are related to an input video, and wherein determining the query image further comprises:

extracting a plurality of frame images from the input video; and

determining the query image from the plurality of frame images.

13. The method of claim 12, where determining the query image comprises determining the query image based on one or more gradient values of the plurality of frame images.

14. The method of claim 13, wherein determining the query image based on the one or more gradient values of the plurality of frame images comprises determining one or more score based on a score function operating on the one or more gradient values.

15. The method of claim 11, wherein the structure image feature point information comprises feature point information inside an original image region and feature point information outside the original image region.

16. The method of claim 15, wherein the feature point information outside the original image region is formed by feature point information that does not match the original image feature point information.

17. The method of claim 15, further comprising:

determining one or more highest structure image IDs of highest matching scores by at least determining a structure image ID of the one or more structure image IDs based on a comparison of the feature point information outside the original image region corresponding to a highest structure image ID of the one or more highest structure image IDs with the feature point information of the query image.

18. The method of claim 17, further comprising:

receiving location information with the one or more images; and

storing at least a structure location related to the original image.

19. The method of claim 11, further comprising:

determining first angle information of a user terminal directed toward the structure image.

20. The method of claim 19, further comprising:

receiving one or more second images;

determining a second query image from the one or more second images;

extracting one or more second feature points from the second query image;

generating second feature point information based on the one or more second features points;

determining that the matching structure ID is associated with the second image based on a comparison of the second feature point information with the structure image feature-point information;

determining second directional angle information of the user terminal based on the one or more second images; and

determining a direction of movement of the user terminal based on the first directional angle information and the second directional angle information.