HANDWRITTEN WORD SPOTTER USING SYNTHESIZED TYPED QUERIES

• US 20100067793A1
• Filed: 09/18/2008
• Published: 03/18/2010
• Est. Priority Date: 09/18/2008
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Serrano, Jose A. Rodriguez, Perronnin, Florent C.
• Granted Patent: US 8,335,381 B2
• Time in Patent Office: Days
• US Class Current: 382/179
• CPC Class Codes: G06F 18/243 relating to the number of classes
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 30/19187 Graphical models, e.g. Bayesian networks or Markov models
• CPC Class Codes: G06V 30/2276 with probabilistic networks, e.g. hidden Markov models

Abstract

A wordspotting system and method are disclosed for processing candidate word images extracted from handwritten documents. In response to a user inputting a selected query string, such as a word to be searched in one or more of the handwritten documents, the system automatically generates at least one computer-generated image based on the query string in a selected font or fonts. A model is trained on the computer-generated image(s) and is thereafter used in the scoring the candidate handwritten word images. The candidate or candidates with the highest scores and/or documents containing them can be presented to the user, tagged, or otherwise processed differently from other candidate word images/documents.

Claims

1. A method comprising:

receiving a query string;

generating at least one computer-generated image based on the query string;

training a model based on the at least one computer generated image;

scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and

based on the scores, identifying a subset of the word images.

2. The method of claim 1, further comprising:

outputting at least one of the identified subset of word images.

3. The method of claim 1, further comprising:

extracting the collection of candidate handwritten word images from at least one document.

4. The method of claim 3, wherein the extracting comprises scanning at least one handwritten document and segmenting the scanned handwritten document.

5. The method of claim 1, wherein the collection of candidate handwritten word images comprises images extracted from at least one document.

6. The method of claim 5, further comprising:

implementing a computer implemented process for at least one document from which a word image of the identified subset of the candidate handwritten word images was extracted.

7. The method of claim 6, wherein the computer implemented process comprises displaying at least that portion of the document from which the word image was extracted.

8. The method of claim 6, wherein the computer implemented process comprises processing a document which contains one of the subset of candidate word images differently from a document which does not.

9. The method of claim 1, wherein the receiving of the query string is performed prior to training of the model based on the at least one computer generated image.

10. The method of claim 1, wherein the model comprises a first model which is trained on the computer-generated image and a second model which has been trained on a collection of sample handwritten word images and wherein the score of each word image is a function of a computed likelihood of the word image with respect to the first model and a computed likelihood of the word image with respect to the second model.

11. The method of claim 1, wherein the model comprises a semi-continuous hidden Markov model and optionally further comprises a Gaussian mixture model.

12. The method of claim 1, wherein the generating of the at least one computer-generated image comprises generating the image in a selected font.

13. The method of claim 12, wherein the generating of the at least one computer-generated image based on the query string comprises generating an binary image of a fixed pixel height in pixels wherein the candidate handwritten word images are normalized to the same fixed pixel height and are binary images.

14. The method of claim 12, wherein the generating of the at least one computer-generated image comprises generating a plurality of computer-generated images based on the query string, each of the plurality of computer-generated images being generated in a different one of a set of selected fonts.

15. The method of claim 14, further comprising identifying the set of fonts based on a precision of the set of fonts for retrieving word images which match at least one selected query string.

16. The method of claim 1, wherein the training a model based on the at least one computer-generated image comprises extracting features from patches of the computer-generated image.

17. The method of claim 16, wherein the extracting features from patches of the computer-generated image comprises translating the image stepwise with a window of fixed height and width and at each step, extracting a patch.

18. A computer program product encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1.

19. A computer implemented processing system comprising:

a synthesizer which synthesizes at least one computer-generated image based on a received query string;

a model which is trained on features extracted from the at least one computer-generated image;

a scoring component which scores candidate handwritten word images of a collection of candidate handwritten word images against the model and, based on the scores, identifies a subset of the handwritten word images.

20. The processing system of claim 19, further comprising:

a document segmentor which processes an input document image to identify the candidate handwritten word images.

21. The processing system of claim 19, further comprising:

a features extractor which extracts the features of the at least one computer-generated image and the candidate handwritten word images.

22. The processing system of claim 19, further comprising:

a document tagger which labels documents or portions thereof which contain at least one of the subset of handwritten word images.

23. The processing system of claim 19, wherein the model comprises a semi-continuous hidden Markov model which includes a background model trained on sample handwritten word images prior to receipt of a query string.

24. A computer implemented method for wordspotting comprising:

receiving a query string to be searched for in a collection of candidate handwritten word images extracted from one or more documents;

for each of a set of fonts, automatically generating an image based on the query string;

modeling the query string with a semi-continuous hidden Markov model, a subset of the parameters of the semi-continuous hidden Markov model being estimated based on features extracted from the images in the different fonts, and other parameters of the semi-continuous hidden Markov model being previously trained on sample handwritten word images without consideration of the query string;

scoring candidate handwritten word images of the collection against the trained semi-continuous hidden Markov model; and

based on the scoring, labeling one or more of the candidate handwritten word images, or a document containing one or more of the candidate handwritten word images.

25. The method of claim 24, wherein the subset of parameters includes transition probabilities and mixture weights and the other parameters include means and covariance matrices.