Applying super resolution for quality improvement of OCR processing

• US 9,256,922 B2
• Filed: 12/20/2013
• Issued: 02/09/2016
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: ABBYY Development LLC
• Original Assignee: ABBYY Development LLC
• Inventors: Loginov, Vasily, Zagaynov, Ivan
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US14/137,913
• Publication Number:
  US 20140369556A1
• Time in Patent Office: 781 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

Systems and methods for improving the quality of recognition of the object based on a series of frame images of objects are described herein. A plurality of images depicting the same object are received. A first image is selected from the plurality of images. The first image may be an image with the highest quality from plurality of images. For each image in the plurality of images, motion estimation of elements of an image in the plurality of images and the first image is performed. Based on the results of motion estimation, motion compensation and signal accumulation of the object in the images in the plurality of images using the first image are performed. A high resolution image of the object obtained based on the motion compensation and signal accumulation is generated. Character recognition on the resulting high resolution image is performed.

Claims

1. A method for improving quality of recognition of objects in images, the method comprising:

receiving a plurality of images, each image depicting same object;

selecting a first image from the plurality of images;

estimating motion of elements of the object in the first image relative to images in the plurality of images;

based on results of motion estimation, performing motion compensation of elements of the one or more objects using the first image for the images in the plurality of images;

accumulating signal from the compensated images into the first image resulting in a high resolution image; and

performing object recognition using the resulting high resolution image.

2. The method of claim 1, wherein the estimating motion step further comprising:

partitioning of each image into non-overlapping blocks of pixels;

comparing blocks of pixels of a current image with corresponding blocks of pixels from the first image; and

calculating motion vectors for each block of pixels, wherein a motion vector describes shift of a block between two images, wherein the shift, calculated for each block of pixels in an image, is estimated movement for a currently analyzed image.

3. The method of claim 2, wherein calculating motion vectors comprises using a multi-scale motion vector search method, wherein the multi-scale motion vector search method comprises calculating Gaussian pyramids.

4. The method of claim 2, further comprising removing from analysis unreliable blocks of pixels which do not satisfy a reliability criteria.

5. The method of claim 4, wherein a reliability criterion of a block is described by equation:

|V_C−

V|<

T, wherein V_C is a motion vector of a currently analyzed block, V is a middle vector of neighboring blocks adjacent to the currently analyzed block, and T is a predetermined threshold value.

6. The method of claim 4, wherein a reliability criterion of a block is described by equation:

Var (I)≧

kσ

, wherein I is the variance of pixel intensity of an analyzed image, and kσ

is level of noise.

7. The method of claim 2, further comprising removing blocks of pixels that only contain background.

8. The method of claim 2, further comprising removing from motion vector calculation blocks of pixels located along image perimeter.

9. The method of claim 2, further comprising smoothing each vector motion component for removal of noise from a field of motion vectors by using a filter.

10. The method of claim 2, further comprising applying bilinear interpolation to the motion vectors.

11. The method of claim 2, further comprising:

removing from calculation of motion vectors unreliable blocks of pixels, wherein the unreliable blocks of pixels do not satisfy reliability criteria;

substituting each of the removed blocks of pixels with a block whose pixel components are calculated based on information of neighboring reliable blocks of pixels using an interpolation method.

12. The method of claim 1, further comprising stabilizing a device that captured the plurality of images.

13. The method of claim 1, wherein the images are sequentially compensated for motion, and wherein signal accumulation is performed on compensated images.

14. The method of claim 1, wherein the step of performing motion compensation comprises calculating a plurality of motion vectors, and a selecting a motion vector from the plurality of motion vectors.

15. The method of claim 14, wherein the motion vector is selected based on a block matching algorithm, wherein the block matching algorithm describes combination of a block of pixels from a currently analyzed image with a corresponding block in the first image.

16. The method of claim 15, wherein the block matching algorithm is based on resulting values of an evaluation metric.

17. The method of claim 15, wherein selecting the motion vector from the plurality of motion vectors is based on criteria of best combination of blocks of pixels, the selected motion vector providing the selected motion vector using a selected difference metric.

18. The method of claim 1, wherein the steps of estimating motion and motion compensation further comprise:

comparing current image with the selected first image;

creating a compensated image through signal compensation based on the pairwise comparison.

19. The method of claim 1, further comprising calculating motion vectors using a model of perspective motion transformation.

20. The method of claim 1, further comprising increasing resolution by changing image size.

21. The method of claim 20, wherein the resolution is increased using a geometric increase of a source image.

22. The method of claim 20, wherein the increase in resolution is achieved by increase using geometric up-scaling with by-pixel motion compensation using bicubic interpolation.

23. The method of claim 22, wherein the geometric increase with by-pixel motion compensation using the bicubic interpolation comprises:

increasing a first processed image from the plurality of images without motion; and

increasing subsequent images with motion vectors.

24. The method of claim 1, further comprising:

sorting the images from the plurality of images using initial estimate of quality of each image based on a number of defects contained in an image, wherein an image with highest quality is selected as the first image.

25. The method of claim 1, wherein determining signal accumulation of the object includes adding information from currently analyzed image into the first image, wherein the first image is a first captured image from the plurality of images.

26. The method of claim 1, wherein signal accumulation of the object includes averaging a signal obtained from the first image and currently analyzed image from the plurality of images using weights.

27. The method of claim 1, wherein accumulating signal of the object further includes adding results of processing the first image and currently analyzed image into the first image.

28. A system comprising:

one or more data processors; and

one or more storage devices storing instructions that, when executed by the one or more data processors, cause the one or more data processors to perform operations comprising;

receiving a plurality of images, each image depicting same object;

selecting a first image from the plurality of images;

estimating motion of elements of the object in the first image relative to images in the plurality of images;

based on results of motion estimation, performing motion compensation of elements of the one or more objects using the first image for the images in the plurality of images;

accumulating signal from the at least some compensated images into the first image resulting in a high resolution image; and

performing object recognition using the resulting high resolution image.

29. The system of claim 28, wherein the estimating motion step further comprising:

partitioning of each image into non-overlapping blocks of pixels;

comparing blocks of pixels of a current image with corresponding blocks of pixels from the first image; and

calculating motion vectors for each block of pixels, wherein a motion vector describes shift of a block between two images, wherein the shift, calculated for each block of pixels in an image, is estimated movement for a currently analyzed image.

30. The system of claim 29, wherein calculating motion vectors comprises using a multi-scale motion vector search method, wherein the multi-scale motion vector search method comprises calculating Gaussian pyramids.

31. The system of claim 29, further comprising removing from analysis unreliable blocks of pixels which do not satisfy reliability criteria.

32. The system of claim 31, wherein a reliability criterion of a block is described by equation:

|V_C−

V|<

T, wherein V_C is a motion vector of a currently analyzed block, V is a middle vector of neighboring blocks adjacent to the currently analyzed block, and T is a predetermined threshold value.

33. The system of claim 31, wherein a reliability criterion of a block is described by equation:

Var (I)≧

kσ

, wherein I is the variance of pixel intensity of an analyzed image, and kσ

is level of noise.

34. The system of claim 28, further comprising stabilizing a device that captured the plurality of images.

35. The system of claim 29, further comprising removing blocks of pixels that only contain background.

36. The system of claim 29, further comprising removing from motion vector calculation blocks of pixels located along image perimeter.

37. The system of claim 29, further comprising smoothing each vector motion component for removal of noise from a field of motion vectors by using a filter.

38. The system of claim 29, further comprising applying bilinear interpolation to the motion vectors.

39. The system of claim 29, further comprising:

removing from calculation of motion vectors unreliable blocks of pixels, wherein the unreliable blocks of pixels do not satisfy reliability criteria;

substituting each of the removed blocks of pixels with a block whose pixel components are calculated based on information of neighboring reliable blocks of pixels using an interpolation method.

40. The system of claim 28, wherein the images are sequentially compensated for motion, and wherein signal accumulation is performed on compensated images.

41. A non-transitory computer-readable storage medium having machine instructions stored therein, the instructions being executable by a processor to cause the processor to perform operations comprising:

receiving a plurality of images, each image depicting same object;

selecting a first image from the plurality of images;

estimating motion of elements of the object in the first image relative to images in the plurality of images;

based on results of motion estimation, performing motion compensation of elements of the one or more objects using the first image for the images in the plurality of images;

accumulating signal from the compensated images into the first image resulting in a high resolution image; and

performing object recognition using the resulting high resolution image.