METHOD AND SYSTEM FOR MOBILE VISUAL SEARCH USING METADATA AND SEGMENTATION

• US 20140254934A1
• Filed: 03/05/2014
• Published: 09/11/2014
• Est. Priority Date: 03/06/2013
• Status: Active Grant

• Current Assignee: Streamoid Technologies
• Original Assignee: Streamoid Technologies
• Inventors: LAXMINARAYANA BHAT, PRASAD HORABAILU, LAKSHMAN, HARICHARAN, ANANTHAPADMANABHA RAO, SANTHOSH LAKSHMANARAO, MANTHANI, SRIDHAR
• Granted Patent: US 9,323,785 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location

Abstract

The embodiments herein provide a method and system for providing an image based mobile augmented reality. The system comprises a mobile device comprising a camera for capturing plurality of images, a display screen for displaying results, a GPS module for determining geographical coordinates, an input interface and a client application including a client-side database; and a central server comprising a server-side database and an image matching and ranking module comprising one or more image matching algorithms for performing an image search. The method comprises capturing a query image, transmitting query data to the central server, applying one or more image matching and ranking algorithms to identify database images matching to the search query, extracting relevant information, presenting the extracted relevant information as search results to the user and providing an option to the user to interactively communicate with the system.

Claims

1. A method for image searching using geographical coordinates and time, the method comprises:

acquiring a query image from a mobile device;

extracting image features of the query image;

extracting image features of the database images;

calculating a matching score for the database image, and wherein the matching score is calculated based on differences between the extracted image features of the query image and the extracted image features of the database image;

identifying matching database images based on the matching score;

extracting geographical coordinates of the identified matching database images;

extracting geographical coordinates of the query image, and wherein the geographical coordinates of the query image is matched with the geographical coordinates of the identified matching database images;

extracting a time stamp of the query image;

extracting a time information of meta data in the identified matching database image, and wherein the time stamp of the query image is matched with the time information of meta data in the identified matching database image;

computing a joint matching score, and wherein the joint matching score is computed by mapping the calculated matching score of the database image, the matched results of the geographical coordinates of the query image and the geographical coordinates of the query image, and the matched time stamp of the query image with the time information of meta data in the database image;

defining a threshold matching score to identify a match between the query image and the database image, and wherein the database image is identified as a matching image to the query image when the computed joint matching score of the database images is less than the pre-defined threshold matching score;

ranking the database images based on the computed joint matching score; and

displaying one or more database images by ranking based on the relevance.

2. The method according to claim 1, wherein the time stamp of the query image is a time of capturing an image through the mobile device, and wherein the time information of the database images is a time of storing an image in the database.

3. The method according to claim 1, wherein the database images are ranked based on the computed joint matching score without any threshold check.

4. The method according to claim 1, wherein the joint matching score is computed based on a linear combination of the scores, and wherein the liner combination of scores comprises:

matching score of the database image;

matching score of the geographical coordinates of the query image and the geographical coordinates of the query image; and

matching score of the time stamp of the query image with the time information of meta data in the database image.

5. A method for multi-stage image searching by segmenting an image into multiple segments, the method comprises:

segmenting a plurality of database images into a plurality of segments;

generating global signatures fin each segment of the database images and storing the global signatures corresponding to each segment;

segmenting a query image into a plurality of segments;

generating global signatures for the entire query image and for each segments of the query image;

storing the global signatures corresponding to the entire query image and for each segments of the query image;

matching global signatures of the query image or the global signatures of each segments of the query image with the global signatures of each segments of the database images;

computing global signature matching scores for the database images;

generating a single common list for the database images;

identifying a subset of database images or by pooling by ranking to a single common list based on the computed global signature matching scores;

matching local features of the query image with the local features of the identified subset of database images

displaying one or more database images by ranking based on local features matching scores.

6. The method according to claim 5, wherein the pooling stage allows a predetermined maximum number of database images to be identified for subsequent image matching stage, in order to limit the computational complexity.

7. The method according to claim 5, wherein the pooling stage allows all the database images whose segment-wise global signatures match the segment-wise global signatures of the query image up to a certain predetermined distortion threshold.

8. The method according to claim 5, wherein the query image is split into at-least two segments along the longer dimension of the query image.

9. The method according to claim 5, wherein the query image is segmented into the plurality of segments along object boundaries using object boundary detection modules.

10. The method according to claim 5, wherein the query image is segmented into the plurality of segments based on a distribution of feature points in the query image, and wherein the distribution of feature points in the query is used to group a set of pixels into the plurality of segments.

11. A method for multi-stage image searching by applying inverted index search on a segmented query image, the method comprises:

segmenting a plurality of database images into multiple segments;

determining a list of quantization indices corresponding to features present in each of the segments in the database images;

segmenting a query image into a plurality of segments;

determining a list of quantization indices corresponding to the features present in query image and the features present in each of the segments in the query images;

identifying a first subset of database images that have non-zero occurrence of the determined query feature quantization indices through an inverted index search;

computing a global signature for the query image and for each of the segments of the query image;

computing a global signature for the query image and for each of the segments of the database images that are identified through the inverted index search;

matching the global signatures of the query image or the global signatures of the segments of the query image with the global signatures of the database images segments that are identified through the inverted index search;

identifying a second subset of database images that matches with the global signatures of the query image or the segments of the query image;

matching local features of the query image with the local features of the database images in the second subset; and

displaying one or more database images by ranking based on the local features matching scores.

12. The method according to claim 11, wherein the second subset selection process comprises identifying a predetermined maximum number of database images based on the global matching scores, in order to limit a computational complexity of the subsequent local feature matching process.

13. The method according to claim 11, wherein the second subset selection process selects database images based on global matching scores up to a certain predetermined threshold, to limit the computational complexity of the subsequent local feature matching process.