Systems and methods for detection and high-quality capture of documents on a cluttered tabletop with an automatically controlled camera

• US 9,747,499 B2
• Filed: 03/03/2015
• Issued: 08/29/2017
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Kim, Chelhwon, Chiu, Patrick, Tang, Hao
• Primary Examiner(s): Findley, Christopher
• Application Number:
  US14/637,391
• Publication Number:
  US 20160259971A1
• Time in Patent Office: 910 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Described are systems and methods for recognizing paper documents on a tabletop using an overhead camera mounted on pan-tilt servos. The described automated system first finds paper documents on a cluttered desk based on a text probability map, constructed using multiple images acquired at fixed grid positions, and then captures a sequence of high-resolution overlapping frames of the located document(s), which are then fused together and perspective-rectified, using computed homography, to reconstruct a high quality and fronto-parallel document image that is of sufficient quality required for optical character recognition. The extracted textual information may be used, for example, for indexing and search, document repository and/or language translation applications.

Claims

1. A computer-implemented method being performed in a computerized system comprising a processing unit, a memory and a camera, the camera being mounted on a turret operatively coupled to the processing unit, the computer-implemented method comprising:

a. scanning a surface with the camera to acquire a first plurality of images of the surface;

b. using the acquired first plurality of images of the surface to determine at least one location of a text on the surface;

c. capturing a second plurality of images using the camera based on the determined location of the text on the surface; and

d. extracting the text using the second captured plurality of images.

2. The computer-implemented method of claim 1, wherein the processing unit is configured to cause the turret to move the camera during the scanning and capturing.

3. The computer-implemented method of claim 1, wherein in a. the first plurality of images is acquired by moving the camera to a plurality of fixed positions along a predetermined path, wherein each image of the first plurality of images corresponds to a fixed position of the plurality of fixed positions, where the image was acquired.

4. The computer-implemented method of claim 3, wherein a. further comprises, for each image in the first plurality of images of the surface, computing feature points and a text response map.

5. The computer-implemented method of claim 4, wherein the each text response map is computed using a probability histogram, the probability histogram being pre-computed based on the feature points.

6. The computer-implemented method of claim 4, wherein a. further comprises stitching the text response maps corresponding to the first plurality of images into a single text response map based on the plurality of fixed positions corresponding to the first plurality of images.

7. The computer-implemented method of claim 6, wherein a. further comprises detecting text blobs in the single text response map.

8. The computer-implemented method of claim 7, wherein a. further comprises identifying at least one large rectangular shaped blob from the detected text blobs as the location of the text on the surface.

9. The computer-implemented method of claim 1, wherein in b. the second plurality of images is captured by moving the camera to a plurality of fixed positions along a predetermined path, wherein each image of the second plurality of images corresponds to a fixed position of the plurality of fixed positions, where the image was captured.

10. The computer-implemented method of claim 9, wherein the determined location of the text on the surface is a location of a text blob and wherein the predetermined path is a center line through a bounding box of the corresponding text blob.

11. The computer-implemented method of claim 9, wherein the images of the second plurality of images overlap with one another.

12. The computer-implemented method of claim 11, further comprising stitching and fusing the images of the second plurality of images to obtain a second stitched image.

13. The computer-implemented method of claim 12, further comprising fitting lines around a boundary of a text blob in the second stitched image and estimating vanishing points.

14. The computer-implemented method of claim 12, further comprising performing a perspective rectification on the second stitched image based on a computed homography.

15. The computer-implemented method of claim 14, wherein in d. the text is extracted using an optical character recognition performed on the perspective rectified second stitched image.

16. A non-transitory computer-readable medium embodying a set of computer-executable instructions, which, when executed in a computerized system comprising a processing unit, a memory and a camera, the camera being mounted on a turret operatively coupled to the processing unit, cause the computerized system to perform a method comprising:

a. scanning a surface with the camera to acquire a first plurality of images of the surface;

b. using the acquired first plurality of images of the surface to determine at least one location of a text on the surface;

c. capturing a second plurality of images using the camera based on the determined location of the text on the surface; and

d. extracting the text using the second captured plurality of images.

17. The non-transitory computer-readable medium of claim 16, wherein the set of computer-executable instructions configures the processing unit to cause the turret to move the camera during the scanning and capturing.

18. The non-transitory computer-readable medium of claim 16, wherein in a. the first plurality of images is acquired by moving the camera to a plurality of fixed positions along a predetermined path, wherein each image of the first plurality of images corresponds to a fixed position of the plurality of fixed positions, where the image was acquired.

19. The non-transitory computer-readable medium of claim 16, wherein a. further comprises, for each image in the first plurality of images of the surface, computing feature points and a text response map.

20. A computerized system comprising a processing unit, a memory and a camera, the camera being mounted on a turret operatively coupled to the processing unit, the memory storing a set of computer-executable instructions causing the computerized system to perform a method comprising:

a. scanning a surface with the camera to acquire a first plurality of images of the surface;

b. using the acquired first plurality of images of the surface to determine at least one location of a text on the surface;

c. capturing a second plurality of images using the camera based on the determined location of the text on the surface; and

d. extracting the text using the second captured plurality of images.