Label-embedding for text recognition

• US 9,008,429 B2
• Filed: 02/01/2013
• Issued: 04/14/2015
• Est. Priority Date: 02/01/2013
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Rodriguez-Serrano, Jose Antonio, Perronnin, Florent C.
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US13/757,014
• Publication Number:
  US 20140219563A1
• Time in Patent Office: 802 Days
• Field of Search: 382/103, 382/105, 382/107, 382/190, 382/282, 358/538, 358/462
• US Class Current: 382/182
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

A system and method for comparing a text image and a character string are provided. The method includes embedding a character string into a vectorial space by extracting a set of features from the character string and generating a character string representation based on the extracted features, such as a spatial pyramid bag of characters (SPBOC) representation. A text image is embedded into a vectorial space by extracting a set of features from the text image and generating a text image representation based on the text image extracted features. A compatibility between the text image representation and the character string representation is computed, which includes computing a function of the text image representation and character string representation.

Claims

1. A method for comparing a text image and a character string comprising:

embedding a character string into a vectorial space, comprising extracting a set of features from the character string and generating a character string representation based on the extracted character string features;

embedding a text image into a vectorial space, comprising extracting a set of features from the text image and generating a text image representation based on the extracted text image features; and

computing a compatibility between the text image representation and character string representation comprising computing a function of the text image representation and character string representation, the function including an embedding parameter w which is a DE-dimensional vector or a D×

E matrix W which embeds the text image representation and character string representation into a new space, where D is the dimensionality of the text image representation and E is the dimensionality of the character string representation,

wherein at least one of the embedding and the computing of the compatibility is performed with a processor.

2. The method of claim 1, wherein the function of the text image representation and character string representation is in a bilinear form.

4. The method of claim 1, wherein the computing of the compatibility comprises determining whether the computed function meets a predetermined threshold.

5. The method of claim 1, wherein the character string comprises a sequence of characters, each character in the sequence being drawn from a finite set of characters.

6. The method of claim 1, wherein the character string is in a standard format selected from ASCII, UNICODE, and UTF-8 where each character of the character string is represented in the form of a binary code.

7. The method of claim 1, wherein the character string comprises a license plate number.

8. The method of claim 1, wherein the text image comprises a captured image of a license plate.

9. The method of claim 1, wherein the character string embedding comprises extraction of a spatial pyramid bag-of-characters.

14. The method of claim 1, wherein parameters of the compatibility function have been learned through optimization of a ranking objective function.

15. The method of claim 1, further comprising learning parameters of the compatibility function using a training set of text images that are each labeled with a respective character string corresponding to text present in the image.

16. The method of claim 1, wherein the text image embedding comprises extraction and aggregation of local descriptors using Fisher vectors in which a gradient of the log-likelihood of the descriptor is measured with respect to parameters of a Gaussian Mixture Model.

17. The method of claim 1, further comprising outputting information based on the computed compatibility between the text image representation and the character string representation.

18. The method of claim 1, wherein the character string comprises a vocabulary of character strings, and the method comprises outputting a character string from the vocabulary of character strings having a character string representation with a highest computed compatibility with the text image representation.

19. The method of claim 1, wherein the text image comprises a collection of text images and the method comprises outputting a text image from the collection of text images having a text image representation with a highest computed compatibility with the text image representation.

20. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer, causes the computer to perform the method of claim 1.

21. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

3. A method for comparing a text image and a character string comprising:

embedding a character string into a vectorial space, comprising extracting a set of features from the character string and generating a character string representation based on the extracted character string features;

embedding a text image into a vectorial space, comprising extracting a set of features from the text image and generating a text image representation based on the extracted text image features; and

computing a compatibility between the text image representation and character string representation comprising computing a function of the text image representation and character string representation, wherein the function is of the form;

F(x,y;

W)=θ

(x)^TWφ

(y) 

 

(5)

where F represents the compatibility between an image and a character string, given a matrix of weights W,

θ

(x) represents one of the text image representation and the character string representation and φ

(y) represents the other of the text image representation and the character string representation, and T represents the transpose operator,

or where an approximation W≈

U^TV is used, with Uε

<sup>R×

D</sup>, Vε

<sup>R×

E</sup>, where R<

D, is a function of the form;

F(x,y;

W)=(Uθ

(x))^T(Vφ

(y)) 

 

(6); and

wherein at least one of the embedding and the computing of the compatibility is performed with a processor.

10. A method for comparing a text image and a character string comprising:

embedding a character string into a vectorial space, comprising;

extracting a spatial pyramid bag-of-characters comprising partitioning the character string into a plurality of regions,

for each of the regions, extracting features based on the characters present in the region,

extracting a representation of each of the regions based on the respective extracted features, and

generating a character string representation, the character string representation being derived from the region representations;

embedding a text image into a vectorial space, comprising extracting a set of features from the text image and generating a text image representation based on the extracted text image features; and

computing a compatibility between the text image representation and character string representation comprising computing a function of the text image representation and character string representation,

wherein at least one of the embedding and the computing of the compatibility is performed with a processor.

11. The method of claim 10, wherein the partitioning of the character string into a plurality of regions and extracting a representation of each of the regions is repeated at least once, wherein in each of the repeats, the partitioning includes partitioning one of the previously-generated regions.

12. The method of claim 10, wherein the partition generates a spatial pyramid of regions which includes at least three levels, each of at least two of the levels including a plurality of the regions generated by partitioning the regions of a previous level in the spatial pyramid.

13. The method of claim 12, wherein the spatial pyramid includes at least seven regions.

22. A system for comparing a text image and a character string comprising:

a text string representation generator for generating a character string representation based on features extracted from a character string, the character string consisting of a sequence of characters, the character string representation comprising a spatial pyramid bag of characters representation generated by partitioning the character string into at least two regions and partitioning each of the at least two regions into at least two smaller regions, the character string representation being based on representations of features extracted from the regions;

a text image representation generator for generating a text image representation based on features extracted from a text image;

a comparator for computing a compatibility between the text image representation and the character string representation;

an output component for outputting information based on the computed compatibility between at least one character string representation and at least one text image representation; and

a processor which implements the text string representation generator, text image representation generator;

comparator, and output component.

23. A method for comparing a text image and a character string comprising:

for at least one character string comprising a sequence of characters, extracting a set of features from the character string, comprising partitioning the character string to form a spatial pyramid of regions, and for each region, generating a representation of the region comprising counting occurrences of each of a finite set of characters in the region and generating a region representation based on the counts;

generating a character string representation based on the region representations;

extracting a set of features from the text image and generating a text image representation based thereon; and

computing a compatibility between the text image representation and the character string representation comprising embedding at least one of the character string representation and the text image representation with a matrix of learned parameters, the compatibility being a function of the at least one embedded representation,

wherein at least one of the embedding and the computing of the compatibility is performed with a processor.

24. The method of claim 23, wherein in the partitioning, for at each of a plurality of levels of the spatial pyramid, regions of a previous level are each partitioned into at least two smaller regions.