Method and subsystem for identifying document subimages within digital images

• US 10,503,997 B2
• Filed: 06/28/2016
• Issued: 12/10/2019
• Est. Priority Date: 06/22/2016
• Status: Active Grant

• Current Assignee: ABBYY Development LLC
• Original Assignee: ABBYY Production LLC (ABBYY Software)
• Inventors: Zagaynov, Ivan Germanovich, Loginov, Vasily Vasilyevich, Lobastov, Stepan Yurievich
• Primary Examiner(s): Rudolph, Vincent
• Assistant Examiner(s): Patel, Pinalben
• Application Number:
  US15/195,759
• Publication Number:
  US 20170372134A1
• Time in Patent Office: 1,260 Days
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/469 Contour-based spatial representations, e.g. vector-coding
• CPC Class Codes: G06V 10/473 using gradient analysis
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

The current document is directed to automated methods and systems, controlled by various constraints and parameters, that identify document sub-images within digital images. Certain of the parameters constrain contour identification and document-subimage-hypothesis generation. The currently described methods and systems identify contours within the digital image, partition the identified contours into four contour sets corresponding to four different regions of the original digital image, construct hypotheses based on these contours for the edges or boundaries of a digital sub-image, and evaluate the hypotheses in order to select a most highly scored hypotheses as a representation of the borders of a digital sub-image within the original received digital image.

Claims

1. An image-processing subsystem to identify a document sub-image within a digital image, the image-processing subsystem comprising:

a memory; and

a processor, operatively coupled to the memory, being configured to;

receive the digital image;

identify contours corresponding to intensity edges within the digital image;

generate four contour sets that each includes either predominantly vertically oriented or predominantly horizontally oriented contours selected from the identified contours, wherein to generate the four contour sets, the processor is to partition the identified contours into a first set of predominantly vertically oriented contours and a second set of predominantly horizontally oriented contours, select contours from the first set of predominantly vertically oriented contours to add to right and left contour sets, and select contours from the second set of predominantly horizontally oriented contours to add to upper and lower contour sets;

generate hypotheses, each comprising data that describes a four-sided polygon positioned and oriented with respect to the received digital image, by combining contours selected from each of the right, left, upper, and lower contour sets from the four contour sets;

score the generated hypotheses;

select a hypothesis among the generated hypotheses based on the generated scores; and

store the selected hypothesis in memory as an indication of boundaries of the document sub-image.

2. The image-processing subsystem of claim 1 wherein the image-processing subsystem identifies contours corresponding to intensity edges within the digital image by:

employing multiple parameter values that control contour identification; and

identifying contours by

identifying seed pixels within the digital image coincident with intensity edges, and

for each of multiple seed pixels, constructing an incipient contour that includes the seed pixel, and iteratively extending both ends of the incipient contour along an intensity edge to generate an identified contour.

3. The image-processing subsystem of claim 1,

wherein selecting contours from the first set of predominantly vertically oriented contours to add to the right contours set further comprises selecting those contours that he within a right-hand region of the received digital image having a width greater than half the width of the received digital image, a right-hand border of the right-hand region coincident with a right-hand border of the received digital image;

and

wherein selecting contours from the first set of predominantly vertically oriented contours to add to the left contours set further comprises selecting those contours that he within a left-hand region of the received digital image having a width greater than half the width of the digital image, a left-hand border of the left-hand region coincident with a left-hand border of the received digital image.

4. The image-processing subsystem of claim 1,

wherein selecting contours from the second set of predominantly horizontally oriented contours to add to the upper contours set further comprises selecting those contours that are within an upper-hand region of the received digital image with a height greater than half the height of the digital image, a top border of the upper-hand region coincident with a top border of the received digital image;

and

wherein selecting contours from the second set of predominantly horizontally oriented contours to add to the lower contours set further comprises selecting those contours that are within a lower region of the received digital image with a height greater than half the height of the digital image, a bottom border of the upper-hand region coincident with a bottom border of the received digital image.

5. The image-processing subsystem of claim 1, wherein, following generation of the four contour sets that each includes either predominantly vertically oriented or predominantly horizontally oriented contours selected from the identified contours, the image-processing subsystem:

filters each of the right, left, upper, and lower contour sets from the four contour sets, using a filtering criteria, to remove contours so that each contour set contains no more than a threshold number of contours.

6. The image-processing subsystem of claim 5, wherein the filtering criteria include at least one or more of:

contour length;

extent of contour curvature;

or

number of pixels overlain by contour in the received digital image.

7. The image-processing subsystem of claim 5, wherein, following filtering, each contour set is sorted by one of length and a number of pixels overlain by contour in the received digital image.

8. The image-processing subsystem of claim 1, wherein the image-processing subsystem generates hypotheses by:

iteratively generating a next combination of four contours selected from the four contour sets;

attempting to use the four contours of the generated combination to generate a four-sided polygon; and

when a four-sided polygon is successfully generated, when the four-sided polygon is convex, and when no vertex of the four-sided polygon lies outside a rectangle representing an extended received-digital-image boundary, encoding the four-sided polygon in a hypothesis data structure for scoring until no additional combinations of four contours selected from the four contour sets can be generated.

9. The image-processing subsystem of claim 8, wherein the image processing subsystem attempts to use the four contours of the generated combination to generate a four-sided polygon by extending one or more line segments fitted to one or more of the contours.

10. The image-processing subsystem of claim 1, wherein the image-processing subsystem uses an encoding of a four-sided polygon that corresponds to a generated hypothesis to score the generated hypothesis by:

for each four-sided-polygon side, computing a side-quality metric and a side weight;

for each four-sided-polygon angle, computing an angle-quality metric;

computing a four-sided-polygon area-quality metric;

when a ratio of width to height for a document corresponding to the document sub-image is known, computing a side-correspondence quality metric; and

combining the side-quality metrics, side weights, area-quality metric, and correspondence quality metric to generate a numerical score for the generated hypothesis.

11. The image-processing subsystem of claim 10,

wherein the value assigned to a side-quality metric increases with an increase of the ratio of the length of the side to the length of a corresponding edge of the received digital image;

wherein the value assigned to an angle-quality metric increases as the angle nears 90°

;

wherein the value assigned to the area-quality metric increases as the ratio of the area of the four-sided polygon to the area of the received digital image increases; and

wherein the side-correspondence quality metric increases as the width-to-height ratio of a rectangle corresponding to the four-sided polygon approaches the known ratio of width to height for the document corresponding to the document sub-image.

12. The image-processing subsystem of claim 10, wherein the weight computed for a side is related to

the difference between a sum of pixel weights of pixels of a contour corresponding to the side and a sum of pixel weights associated with pixels associated with the side that are not associated with the contour;

and

a ratio of the number of pixels of the contour corresponding to the side to the number of pixels associated with the side.

13. The image-processing subsystem of claim 12, wherein a pixel weight is computed from the projection of an intensity gradient associated with the pixel and a vector perpendicular to the side.

14. A method that identifies a document sub-image within a digital image, the method comprising:

receiving the digital image;

identifying contours corresponding to intensity edges within the digital image;

generating four contour sets that each includes either predominantly vertically oriented or predominantly horizontally oriented contours selected from the identified contours by partitioning the identified contours into a first set of predominantly vertically oriented contours and a second set of predominantly horizontally oriented contours, selecting contours from the first set of predominantly vertically oriented contours to add to right and left contour sets, and selecting contours from the second set of predominantly horizontally oriented contours to add to upper and lower contour sets;

generating hypotheses, each comprising data that describes a four-sided polygon positioned and oriented with respect to the received digital image, by combining contours selected from each of the right, left, upper, and lower contour sets from the four contour sets;

scoring the generated hypotheses;

selecting a hypothesis among the generated hypotheses based on the generated scores; and

storing the selected hypothesis in memory as an indication of boundaries of a document sub-image within the received digital image.

15. The method of claim 14, wherein identifying contours corresponding to intensity edges within the digital image further comprises:

employing multiple parameter values that control contour identification; and

identifying contours by

identifying seed pixels within the digital image coincident with intensity edges, and

for each of multiple seed pixels, constructing an incipient contour that includes the seed pixel, and iteratively extending both ends of the incipient contour along an intensity edge to generate an identified contour.

16. The method of claim 14, wherein generating hypotheses further comprises:

iteratively generating a next combination of four contours selected from the four contour sets;

attempting to use the four contours of the generated combination to generate a four-sided polygon; and

when a four-sided polygon is successfully generated, when the four-sided polygon is convex, and when no vertex of the four-sided polygon lies outside a rectangle representing an extended received-digital-image boundary, encoding the four-sided polygon in a hypothesis data structure for scoring until no additional combinations of four contours selected from the four contour sets can be generated.

17. The method of claim 16, wherein attempting to use the four contours of the generated combination to generate a four-sided polygon further includes extending one or more line segments fitted to one or more of the contours.

18. The method of claim 14, further including using an encoding of a four-sided polygon that corresponds to a generated hypothesis to score the generated hypothesis by:

for each four-sided-polygon side, computing a side-quality metric and a side weight;

for each four-sided-polygon angle, computing an angle-quality metric;

computing a four-sided-polygon area-quality metric;

when a ratio of width to height for a document is known, computing a side-correspondence quality metric; and

combining the side-quality metrics, side weights, area-quality metric, and—

correspondence quality metric to generate a numerical score for the generated hypothesis.

19. A non-transitory computer-readable medium for identifying a document sub-image within a digital image, the non-transitory computer-readable medium having recorded thereon instructions that when executed by one or more computer processors, perform operations comprising:

receiving the digital image;

identifying contours corresponding to intensity edges within the digital image;

generating four contour sets that each includes either predominantly vertically oriented or predominantly horizontally oriented contours selected from the identified contours by partitioning the identified contours into a first set of predominantly vertically oriented contours and a second set of predominantly horizontally oriented contours, selecting contours from the first set of predominantly vertically oriented contours to add to right and left contour sets, and selecting contours from the second set of predominantly horizontally oriented contours to add to upper and lower contour sets;

generating hypotheses, each comprising data that describes a four-sided polygon positioned and oriented with respect to the received digital image, by combining contours selected from each of the right, left, upper, and lower contour sets from the four contour sets;

scoring the generated hypotheses;

selecting a hypothesis among the generated hypotheses based on the generated scores; and

storing the selected hypothesis in memory as an indication of boundaries of the document sub-image.