Automatic recognition of images

• US 8,744,196 B2
• Filed: 01/10/2011
• Issued: 06/03/2014
• Est. Priority Date: 11/26/2010
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Banerjee, Serene, Kuchibhotla, Anjaneyulu Seetha Rama, Sharma, Avinash
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US12/987,179
• Publication Number:
  US 20120134576A1
• Time in Patent Office: 1,240 Days
• Field of Search: 382/181, 382/100, 707/104.1
• US Class Current: 382/224
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/2136 based on sparsity criteria, e.g. with an overcomplete basis
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods

Abstract

Presented is a method of automatically performing an action, based on graphical input. The method comprises: receiving, for a user, an input image; comparing the input image with the contents of a user-customized database comprising a plurality of records, each record representing a predefined class of image, wherein the user has previously associated records in the database with respective specified actions; attempting to recognize the image, based on the similarity of the input image to one of the predefined classes of image represented in the user-customized database; and if the image is recognized, performing the action previously associated by the user with the class. Also presented is apparatus for recognizing an image and a method of constructing a user-customized database.

Claims

1. A method of automatically performing an action, based on graphical input, the method comprising:

receiving, for a user, an input image;

comparing the input image with the contents of a user-customized database comprising a plurality of records, each record representing a predefined class of image, wherein the user has previously associated records in the database with respective specified actions;

attempting to recognize the image, based on the similarity of the input image to one of the predefined classes of image represented in the user-customised database; and

if the image is recognized, performing the action previously associated by the user with the class.

2. The method of claim 1, further comprising:

if the image is not recognized, prompting the user for an action to be associated with a class to which the image belongs; and

adding a record for the class to the database, associated with the action.

3. The method of claim 1, further comprising:

if the image is not recognized, prompting the user to indicate a record in the database to which the image corresponds; and

updating the database, based on the input image, so that the image is in future associated with the same action as the record indicated by the user.

4. The method of claim 1, wherein the step of comparing the input image with the contents of the user-customized database comprises identifying in the image fiducial points suitable for object recognition, comprising:

detecting a first set of points comprising corner points in the image;

detecting a second set of points comprising the contours of edges in the image;

determining the fiducial points as a subset of the points in the first set which are located near to a point in the second set.

5. The method of claim 4, wherein the step of detecting edge contours comprises:

detecting the contours as a set of candidate edge-pixels; and

refining the contours, by eliminating some of the candidate edge-pixels to reduce the width of the contours.

6. The method of claim 4, wherein the step of detecting corner points comprises detecting points which belong to two edges, wherein the angle between the two edges falls in a predefined range about 90 degrees.

7. A method of automatically performing an action, based on graphical input, the method comprising:

receiving, for a user, an input image;

comparing the input image with the contents of a user-customized database comprising a plurality of records, each record representing a predefined class of image, wherein records in the database are associated with respective specified actions; and

attempting to recognize the input image by class, based on the similarity of the input image to one of the predefined classes of image represented in the user-customized database;

wherein the steps of comparing the input image with the contents of the database and attempting to recognize the image comprise;

classifying the input image into one of a plurality of classes, each class corresponding to one of the records in the database, the classification process comprising;

identifying fiducial points in the input image;

populating a spatial histogram with the fiducial points, to generate a test feature vector;

receiving, for each class, one or more reference feature vectors, extracted from respective one or more reference images for that class;

solving a system of linear equations to produce a solution whereby the test feature vector is modeled as a linear combination of the reference feature vectors; and

assigning the input image to a first class according to the solution.

8. The method of claim 7, wherein:

the step of solving a linear system of equations comprises, for each class, approximating the test feature vector by a linear combination of the one or more reference feature vectors in that class; and

in the step of assigning the input image to a first class, the input image is assigned to the class yielding the most accurate approximation.

9. The method of claim 7, wherein, in the step of populating the spatial histogram, the origin of the histogram is positioned at a centroid of the fiducial points.

10. The method of claim 7, further comprising calculating a measure of confidence that the input image belongs to the assigned first class,

wherein, if the measure of confidence is below a preset threshold, the assignment of the input image to the first class is re-evaluated.

11. The method of claim 10, wherein the re-evaluation of the assignment comprises:

for each of the fiducial points identified in the input image;

selecting that fiducial point; and

populating a spatial histogram with the remaining fiducial points, to generate a test feature vector,

wherein an origin of the histogram is positioned at the selected fiducial point;

receiving, for at least the first class, a reference feature vector for each fiducial point of each reference image in that class;

comparing each of the test feature vectors with each of the reference feature vectors; and

assigning the input image to a class according to the result of the comparison.

12. The method of claim 1, further comprising constructing the user-customized database, comprising:

receiving, from a user, an image;

receiving, from the user, an indication of an action to be associated with the image;

creating a record in the database to represent that image; and

associating the record with the action.

13. A computer program on a non-transitory computer-readable medium, said program comprising computer program code to control a physical computing device to perform the method of claim 1.

14. Apparatus for recognizing an image, comprising:

an input for receiving an input image to be recognized;

a first database comprising a plurality of records, each record representing a class of image and associating the class with an action;

a comparison unit adapted to compare the input image with the contents of the first database;

a recognition processor to associate the input image with a class of image represented in said first database, based on a similarity between the input image and one of the classes represented in the database; and

an output, for providing an indication of the action associated with the class corresponding to the input image.

15. The apparatus of claim 14, further comprising a processor to perform the action associated with the class corresponding to the input image.

16. The apparatus of claim 15, wherein the action comprises the processor launching a specific software application associated with the class corresponding to the input image.

17. The apparatus of claim 14, wherein the action comprises multiple actions.

18. The image of claim 1,

wherein attempting to recognize the input image comprises identifying the input image as belonging to one of the predefined classes of image, based on the similarity of the input image to one of the predefined classes of image represented in the user-customised database;

wherein, if the image is recognized, performing the action comprises performing an action previously associated by the user with the corresponding class to which the input image belongs.

19. The method of claim 1, wherein the action comprises launching a specific software application associated by the user with the class to which the input image is recognized as belonging.

20. The method of claim 1, wherein the action comprises multiple actions.