Methods and systems for object recognition

• US 9,489,401 B1
• Filed: 06/16/2015
• Issued: 11/08/2016
• Est. Priority Date: 06/16/2015
• Status: Active Grant

• Current Assignee: My Eyespy Pty., Ltd.
• Original Assignee: My Eyespy Pty., Ltd.
• Inventors: Garcia, Juan, Bakhshandeh, Reza
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): Allen, Nicholas
• Application Number:
  US14/741,186
• Time in Patent Office: 511 Days
• Field of Search: 707/694
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

An object identification system comprises at least one database of reference images. Each reference image comprises an identifiable reference object and is associated in the database with a corresponding text description of the reference object. The system also comprises image query resolution data structures, each containing a collection of records having an image descriptor of one of the reference images, and searchable using a corresponding search algorithm. Upon receiving a query comprising an input image including a query object, the system generates a query image descriptor of the input image corresponding with one or more search algorithms, and searches the corresponding image query resolution data structure to identify a closest match record. A best available match record is identified from the closest match records identified in the image query resolution data structures, and a text description of the query object is retrieved corresponding with the associated reference image.

Claims

1. An object identification method comprising:

providing one or more databases which contain;

a corpus of reference images, each reference image comprising an identifiable reference object and being associated in the database with a corresponding text description of the reference object; and

a plurality of image query resolution data structures, each comprising a collection of records wherein each record includes an image descriptor of one of the reference images, and wherein each data structure is searchable using a corresponding one of a set of predetermined search algorithms to identify a closest match record within the data structure based upon the image descriptors,

populating the corpus of reference images, at least in part, by one or more crawler processes, each configured to traverse a corresponding selected subset of the internet, to identify and extract reference images comprising reference objects, and to analyze image-associated content to determine the corresponding text description of the reference object comprised within each extracted reference image;

updating the image query resolution data structures to incorporate records for the extracted reference images;

receiving a query comprising an input image which includes a query object;

for each of one or more applicable search algorithms selected from the set of predetermined search algorithms, generating a query image descriptor of the input image and searching the corresponding image query resolution data structure to identify a closest match record corresponding with the query image descriptor;

selecting a best available match record from the closest match records identified in the image query resolution data structures;

retrieving the text description associated with the reference image corresponding with the best available match record; and

providing, as an output object identification, the retrieved text description.

2. The method of claim 1 wherein the set of predetermined search algorithms includes one or more algorithms selected from the group comprising:

a barcode detection algorithm, a QR code detection algorithm;

a text detection algorithm;

a movie poster detection algorithm;

a product detection algorithm; and

a general image detection algorithm.

3. The method of claim 1 wherein generating a query image descriptor comprises one or more of:

extracting barcode features from the input image;

extracting QR code features from the input image;

extracting text features from the image;

extracting Speeded Up Robust Features (SURF) feature vectors from the input image; and

extracting Scale Invariant Features Transform (SIFT) feature vectors from the input image.

4. The method of claim 1 wherein the one or more crawler processes comprise a plurality of content-specific crawler processes, each of which is configured to extract content-specific images and to determine associated descriptive text.

5. The method of claim 4 wherein the content-specific crawler processes are configured to crawl web sites selected from:

online movie databases;

online stores;

online encyclopedias; and

specific-interest-related web sites.

6. The method of claim 1 wherein the corpus of reference images is further populated, at least in part, using results from image search queries.

7. The method of claim 1 wherein the image query resolution data structures comprise one or more of:

a barcode index data structure;

a QR code index data structure;

a text index data structure;

a k-dimensional tree structure of feature vectors; and

a n-Nearest Neighbor graph structure of feature vectors.

8. The method of claim 1 further comprising searching a history database containing previously received input images and associated text descriptions.

9. The method of claim 1 further comprising comparing a quality measure of the best available match against a predetermined threshold requirement and, in the event that the quality measure fails to meet the threshold requirement, obtaining a third-party text description of the image by performing one or more further steps selected from the group comprising:

performing a reverse image search of the input image using a remote image search engine to obtain a text description corresponding with the image;

submitting the input image to a remote online crowd-sourcing service along with a request for a text description of the query object to obtain a text description corresponding with the image; and

submitting the input image to a remote third-party automated image identification service to obtain a text description corresponding with the image.

10. The method of claim 8 further comprising a step of storing the input image in association with the third-party text description in the history database, which may be searched for an improved match to a subsequently-received input image, relative to the best available match.

11. The method of claim 1, further comprising:

receiving user feedback in relation to accuracy of the retrieved text description; and

storing the user feedback for subsequent review by an operator.

12. The method of claim 11 further comprising:

receiving an instruction from the operator to update the corpus of reference images based upon the user feedback; and

updating the corpus of reference images.

18. The method of claim 1 wherein the subset of the internet traversed by one or more of the crawler processes is selected based upon subject matter of associated content.

19. The method of claim 18 wherein the subject matter of associated content is selected from a group comprising:

cars;

shoes;

jewelry;

animals;

plants;

toys; and

clothing.

20. The method of claim 1 wherein the subset of the internet traversed by one or more of the crawler processes is selected based upon syntactic or semantic structure of associated content.

21. The method of claim 20 wherein the subset of the internet traversed by one or more of the crawler processes based upon syntactic or semantic structure of associated content is selected from the group comprising:

movie databases;

online stores; and

online encyclopedias.

13. An object identification server comprising:

at least one processor;

at least one non-volatile storage device comprising one or more databases which contain;

a corpus of reference images, each reference image comprising an identifiable reference object and being associated in the database with a corresponding text description of the reference object; and

a plurality of image query resolution data structures, each comprising a collection of records wherein each record includes an image descriptor of one of the reference images, and wherein each data structure is searchable using a corresponding one of a set of predetermined search algorithms to identify a closest match record within the data structure based upon the image descriptors,

at least one computer-readable memory device operatively associated with the processor; and

a network communications interface operatively associated with the processor,

wherein the memory device contains computer-executable instruction code which, when executed via the processor, causes the processor to effect a method comprising steps of;

populating the corpus of reference images, at least in part, by one or more crawler processes, each configured to traverse a corresponding selected subset of the internet, to identify and extract reference images comprising reference objects, and to analyze image-associated content to determine the corresponding text description of the reference object comprised within each extracted reference images;

updating the image query resolution data structures to incorporate records for the extracted reference images;

receiving a query comprising an input image which includes a query object;

for each of one or more applicable search algorithms selected from the set of predetermined search algorithms, generating a query image descriptor of the input image and searching the corresponding image query resolution data structure to identify a closest match record corresponding with the query image descriptor;

selecting a best available match record from the closest match records identified in the image query resolution data structures;

retrieving the text description associated with the reference image corresponding with the best available match record; and

providing, as an output object identification, the retrieved text description.

14. A non-transitory computer-readable medium comprising executable program instructions which, when executed by a corresponding processor, cause the processor to implement a method comprising steps of:

acquiring an image which includes a query object;

generating and transmitting, to an object identification server according to claim 13, a query comprising the image;

receiving, from the object identification server, an object identification comprising a text description of the query object; and

displaying, to a user, the text description of the query object.

15. The non-transitory computer-readable medium of claim 14 comprising further executable program instructions which, when executed by the processor, cause the processor to implement the method comprising a further step of:

enabling the user to select a region of the image containing the query object prior to generating and transmitting the query.

16. The non-transitory computer-readable medium of claim 14 comprising further executable program instructions which, when executed by the processor, cause the processor to implement the method comprising a further step of:

after displaying the text description of the query object, enabling the user to obtain further information about the query object.

17. The non-transitory computer-readable medium of claim 16 wherein the further information comprises one or more of:

additional descriptive information of the query object; and

purchasing information in relation to the query object,

wherein the further information is obtained from one or more available information resources using a search query based upon the text description of the query object.