SYSTEMS AND METHODS FOR IMAGE-FEATURE-BASED RECOGNITION

• US 20150227557A1
• Filed: 02/10/2015
• Published: 08/13/2015
• Est. Priority Date: 02/10/2014
• Status: Active Grant

• Current Assignee: Geenee GmbH
• Original Assignee: Geenee Ug
• Inventors: HOLZSCHNEIDER, Philipp, MAGERS, Thorsten Otto Wolfgang, Priehn, Hannes
• Granted Patent: US 9,122,706 B1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Methods and systems are described herein that allow a user to capture a single image snapshot from video, print, or the world around him or her, and obtain additional information relating to the media itself or items of interest displayed in the snapshot. A fingerprint of the snapshot is used as a query and transmitted to the server. Image Feature-Based Recognition, as described herein, uses a feature index to identify a smaller set of candidate matches from a larger database of images based on the fingerprint. Novel methods and systems using a distance metric and a radical hash table design exploit probabilistic effects and allow distinct image features to be preferred over redundant ones, allowing only the more distinctive data points to remain resident within the index, yielding a lean index that can be quickly used in the identification process.

Claims

1. A server for identifying and transmitting to a mobile device additional information relating to a query image obtained by a user with a mobile device, the server comprising:

a reference image database storing one or more reference feature descriptors associated with one or more reference images, wherein at least one reference image feature is mapped to a plurality of addresses in the reference image database using a plurality of functions; and

a processor for executing one or more modules comprising;

a query receiving module for receiving, from the mobile device, a first query including one or more query image feature descriptors;

a candidate feature descriptors retrieval module for;

calculating, for each of the one or more query image feature descriptors, a plurality of addresses using the plurality of functions, and

for each of the plurality of addresses, calculating a number of duplicate reference image feature descriptors stored within a pre-defined region of the reference image database;

a candidate images retrieval module for;

selecting a set of reference image feature descriptors based on the calculated number of duplicate reference image feature descriptors;

an image matching module for;

comparing the selected set of reference image feature descriptors with the one or more query image feature descriptors to determine if any reference image mages the query image;

the server further configured to perform the steps of

if a matching reference image is determined, obtaining additional information associated with the at least one matching reference image from an annotation database, and

transmitting the additional information to the mobile device for displaying as additional information relating to the at least one item of interest to the user.

2. The server of claim 1, wherein the processor is further configured to execute a hash table management module.

3. The server of claim 2, wherein the hash table management module further comprises a collision resolving module, for handling hash collisions;

and an overflow handling module for handling the association of a new data point (e.g., a set of feature descriptors) with an address in a hash table.

4. A computer-implemented method for identifying and displaying additional information relating to a first query image obtained by a user with a mobile device, the method comprising:

receiving, from the mobile device, a first query including one or more query image feature descriptors corresponding to at least one item of interest to the user included in the first query image;

responsive to receiving the first query, performing a similarity search, wherein the similarity search comprises;

for each of the query image feature descriptors, calculating a plurality of addresses using a plurality of functions;

for each of the plurality of addresses, calculating a number of duplicate reference image feature descriptors stored within a certain region of a database, and wherein at least one reference image feature is mapped to a plurality of addresses in the reference image database using the plurality of functions; and

determining one or more candidate images based on the number of duplicate reference image feature descriptors within the certain region of the reference image database associated with each of the plurality of addresses, the reference image database storing one or more reference image feature descriptors, each of the reference image feature descriptors associated with at least one reference image

comparing the feature descriptors for the one or more candidate images to the query image feature descriptors to determine if there is a match; and

if there is a match, acquiring additional information associated with the match; and

transmitting, to the mobile device, the additional information associated with the match for displaying to the user.

5. The computer-implemented method of claim 4,

wherein the number of duplicate reference image feature descriptors within a pre-defined region associated with an address reflects a degree of similarity between the duplicate reference image feature descriptors and the first query image feature descriptors mapped to the address;

and wherein the determination of one or more candidate images comprises;

determining that the number of duplicate reference image feature descriptors exceeds a pre-determined threshold; and

acquiring the one or more candidate images associated with the duplicate reference image feature descriptors.

6. The computer-implemented method of claim 5, further comprising:

receiving, from the mobile device, a second query including one or more second query image feature descriptors;

comparing the one or more first query image feature descriptors with the one or more second query image feature descriptors; and

based on a result of the comparison;

determining not to perform the similarity search; and

acquiring, based on a mapping between the first query and the one or more candidate images stored in a query cache, the one or more candidate images;

7. The method of claim 4, wherein

wherein reference image feature descriptors are stored within a database based on a hash table including a plurality of buckets, each bucket being associated with an address;

wherein the hash table is associated with a family of hash functions, the family of hash functions mapping one feature descriptor data point to a first set of addresses associated with a first set of buckets out of the plurality of buckets, and each of the first set of buckets storing the one second feature descriptor data point.

8. A computer-implemented method for identifying and displaying additional information relating to an image obtained by a user with a mobile device, the method comprising:

acquiring at least one image using the mobile device, the at least one image comprising at least one item of interest to the user about which the user wishes to receive additional information;

extracting one or more first features from the at least one image, the one or more first features corresponding to the at least one item of interest;

generating a first fingerprint representing the one or more first features;

transmitting, to a server, a first query including the first fingerprint, the fingerprint allowing the server to acquire one or more second images and determine, based on one or more second images selected based on the first query, that the server has additional information relating to the at least one item of interest to the user; and

receiving, from the server, the additional information relating to the at least one item of interest to the user and displaying the additional information to the user.

9. The computer-implemented method of claim 8, further comprising:

receiving, via the first interface on the mobile device, a user'"'"'s selection of a portion of the first image;

wherein extracting one or more first features from the at least one image comprises extracting one or more first features corresponding to the selected portion; and

wherein generating a first fingerprint comprises generating a first fingerprint representing the one or more first features extracted from the selected portion of the first image.