Method and system to perform text-to-image queries with wildcards

• US 9,626,594 B2
• Filed: 01/21/2015
• Issued: 04/18/2017
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Soldevila, Albert Gordo, Rodrguez-Serrano, Jos Antonio, Perronnin, Florent
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/601,802
• Publication Number:
  US 20160210532A1
• Time in Patent Office: 818 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/262 using context analysis, e.g. lexical, syntactic or semantic context

Abstract

A system and method for comparing a text image with or without a wildcard character and a character string are provided. The method includes embedding a character string into a vectorial space by extracting a set of features from the character string and generating a character string representation based on the extracted features, such as a spatial pyramid bag of characters (SPBOC) representation. A text image is embedded into a vectorial space by extracting a set of features from the text image and generating a text image representation based on the text image extracted features. A similarity between the text image representation and the character string representation is computed, which includes computing a function of the text image representation and character string representation.

Claims

1. A system for comparing a queried character string including a wildcard character and a text image comprising:

a character string image representation generator configured to embed a character string image representation of the queried character string including the wildcard into a common vectorial space based on features extracted from the queried character string including the wildcard character, the queried character string consisting of a sequence of a plurality of characters including the wildcard character, and the character string image representation including a spatial pyramid bag of characters representation including information indicating a location of the wildcard character relative to other characters included in the queried character string;

a text image representation generator configured to embed a text image representation of the text image into the common vectorial space, the text image based on features extracted from the text image, and the text image including a sequence of a plurality of textual characters;

a comparator configured to compute a similarity between the text image representation embedded in the common vectorial space and the character string image representation embedded in the common vectorial space;

an output component configured to output information based on the computed similarity between the character string image representation embedded in the common vectorial space and the text image representation embedded in the common vectorial space; and

a processor which implements the text string representation generator, text image representation generator, comparator, and output component.

3. The method of claim 1, further comprising learning parameters of the similarity function using a training set of text images that are each labeled with a respective character string including a wildcard character corresponding to text present in the image.

2. A computer implemented method for comparing a queried character string and a text image comprising:

for at least one queried character string including a sequence of characters including a wildcard character, extracting a set of features from the queried character string by partitioning the queried character string to form a spatial pyramid of regions and for each region generating a representation of the region including counting occurrences of each of a finite set of characters in the region;

generating a character string image representation of the queried character string based on the region representations, the character string image representation including information indicating a location of the wildcard character relative to other characters included in the queried character string;

embedding the character string image representation into a common vectorial space;

extracting a set of features from the text image and generating a text image representation based thereon;

embedding the text image into the common vectorial space; and

computing a similarity function between the text image representation and the character string representation comprising embedding at least one of the character string representation and the text image representation with a matrix of learned parameters, the similarity being a function of the at least one embedded representation, and

the learned parameters of the similarity function learned using a training set of text images that are each labeled with a respective character string including a wildcard character corresponding to text present in the image.