Handwritten word spotter system using synthesized typed queries

• US 8,731,300 B2
• Filed: 11/16/2012
• Issued: 05/20/2014
• Est. Priority Date: 09/18/2008
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Rodriguez Serrano, Jose A., Perronnin, Florent C.
• Primary Examiner(s): CONWAY, THOMAS A
• Application Number:
  US13/678,961
• Publication Number:
  US 20130156302A1
• Time in Patent Office: 550 Days
• Field of Search: None
• US Class Current: 382/186
• CPC Class Codes: G06F 18/243 relating to the number of classes
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 30/19187 Graphical models, e.g. Bayesian networks or Markov models
• CPC Class Codes: G06V 30/2276 with probabilistic networks, e.g. hidden Markov models

Abstract

A wordspotting system and method are disclosed for processing candidate word images extracted from handwritten documents. In response to a user inputting a selected query string, such as a word to be searched in one or more of the handwritten documents, the system automatically generates at least one computer-generated image based on the query string in a selected font or fonts. A model is trained on the computer-generated image(s) and is thereafter used in the scoring the candidate handwritten word images. The candidate or candidates with the highest scores and/or documents containing them can be presented to the user, tagged, or otherwise processed differently from other candidate word images/documents.

Claims

1. A method comprising:

receiving a query string;

generating a plurality of computer-generated images based on the query string with a computer typographic font, each of the plurality of computer-generated images being generated by varying the computer typographic font, the font variations having been identified based on a precision for retrieving word images which match at least one selected query string;

training a model based on the plurality of computer-generated images;

scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and

based on the scores, identifying a subset of the word images.

2. The method of claim 1, further comprising:

outputting the identified subset of word images.

3. The method of claim 1, wherein the receiving of the query string is performed prior to training of the model based on the plurality of computer-generated images.

4. The method of claim 1, wherein the model comprises a first model which is trained on the plurality of computer-generated images and a second model which has been trained on a collection of sample handwritten word images and wherein the score of each word image is a function of a computed likelihood of the word image with respect to the first model and a computed likelihood of the word image with respect to the second model.

5. The method of claim 1, wherein the model comprises a semi-continuous hidden Markov model and optionally further comprises a Gaussian mixture model.

6. The method of claim 1, wherein the generating of the plurality of computer-generated images based on the query string comprises generating binary images of a fixed pixel height in pixels and wherein the candidate handwritten word images are normalized to the same fixed pixel height and are binary images.

7. A non-transitory computer program product encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1.

8. The method of claim 1, further comprising:

extracting the collection of candidate handwritten word images from at least one document.

9. The method of claim 8, wherein the extracting comprises scanning at least one handwritten document and segmenting the scanned handwritten document.

10. The method of claim 1, wherein the training of the model based on the plurality of computer-generated images comprises extracting features from patches of the plurality of computer-generated images.

11. The method of claim 10, wherein the extracting features from patches of the plurality of computer-generated images comprises translating the image stepwise with a window of fixed height and width and at each step, extracting a patch.

12. The method of claim 1, wherein:

the generating of the plurality of computer-generated images based on the query string includes, for each of a set of fonts, automatically generating images based on the query string;

the training of the model based on the plurality of computer-generated images includes modeling the query string with a semi-continuous hidden Markov model, a subset of the parameters of the semi-continuous hidden Markov model being estimated based on features extracted from the images in the different fonts, and other parameters of the semi-continuous hidden Markov model being previously trained on sample handwritten word images without consideration of the query string;

the scoring of the candidate handwritten word images includes scoring candidate handwritten word images of the collection against the trained semi-continuous hidden Markov model; and

the identifying of the subset of the word images based on the scores includes labeling one or more of the candidate handwritten word images, or a document containing one or more of the candidate handwritten word images.

13. The method of claim 12, wherein the subset of parameters includes transition probabilities and mixture weights and the other parameters include means and covariance matrices.

14. The method of claim 1, wherein the collection of candidate handwritten word images comprises images extracted from at least one document.

15. The method of claim 14, further comprising:

implementing a computer implemented process for at least one document from which a word image of the identified subset of the candidate handwritten word images was extracted.

16. The method of claim 15, wherein the computer implemented process comprises displaying at least that portion of the document from which the word image was extracted.

17. The method of claim 15, wherein the computer implemented process comprises processing a document which contains one of the subset of candidate word images differently from a document which does not.

18. A computer implemented processing system comprising:

a synthesizer which synthesizes a plurality of computer-generated images based on a received query string with a computer typographic font, each of the plurality of computer-generated images being generated by varying the computer typographic font, the font variations having been identified based on a precision for retrieving word images which match at least one selected query string;

a model which is trained on features extracted from the plurality of computer-generated images; and

a scoring component which scores candidate handwritten word images of a collection of candidate handwritten word images against the model and, based on the scores, identifies a subset of the handwritten word images.

19. The processing system of claim 18, further comprising:

a document segmentor which processes an input document image to identify the candidate handwritten word images.

20. The processing system of claim 18, further comprising:

a features extractor which extracts the features of the plurality of computer-generated images and the candidate handwritten word images.

21. The processing system of claim 18, further comprising:

a document tagger which labels documents or portions thereof which contain at least one of the subset of handwritten word images.

22. The processing system of claim 18, wherein the model comprises a semi-continuous hidden Markov model which includes a background model trained on sample handwritten word images prior to receipt of a query string.

23. A method comprising:

receiving a query string;

generating a plurality of computer-generated images based on the query string, each of the plurality of computer-generated images being generated by varying a selected computer typographic font;

training a model based on the plurality of computer-generated images in the different fonts;

scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and

based on the scores, identifying a subset of the word images.

24. The method of claim 23, further comprising identifying the set of fonts based on a precision of the set of fonts for retrieving word images which match at least one selected query string.