Colour document layout analysis with multi-level decomposition

• US 8,532,374 B2
• Filed: 12/03/2008
• Issued: 09/10/2013
• Est. Priority Date: 12/05/2007
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Chen, Yu-Ling, Irrgang, Steven Richard, Chong, Eric Wai-Shing
• Primary Examiner(s): Wang, Claire X
• Assistant Examiner(s): JAMES, THOMAS AIDEN
• Application Number:
  US12/327,247
• Publication Number:
  US 20090148039A1
• Time in Patent Office: 1,742 Days
• Field of Search: None
• US Class Current: 382/164
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/40062 Discrimination between different image types, e.g. two-tone, continuous tone

Abstract

Disclosed is a method of classifying segmented contents of a scanned image of a document. The method comprise partitioning the scanned image into color segmented tiles at pixel level. The method then generates superpositioned segmented contents, each segmented content representing related color segments in at least one color segmented tile. Statistics are then calculated for each segmented content using pixel level statistics from each of the tile color segments included in segmented content, and then determines a classification for each segmented content based on the calculated statistics. The segmented content may be macroregions. The macroregions may form part of a multi-layered document representation of the document. Each of a plurality of tiles of predetermined size of the image are converted into a representation having a plurality of layers, the representation corresponding to at least one said tiles comprising multiple colored layers, each tile comprising a superposition of the corresponding colored layers. For each of the colored layers, merging is performed with adjacent ones of the tiles, thereby generating a multi-layered document representation.

Claims

1. A method of classifying regions of a scanned document image, said method comprising the steps of:

(a) partitioning the scanned image into a plurality of tiles;

(b) determining at least one dominant colour for each of the plurality of tiles;

(c) generating superpositioned regions based on the dominant colours, each said superpositioned region representing a group of tiles, wherein at least one tile is grouped into two superpositioned regions and each dominant colour is represented by at most one of the superpositioned regions, and wherein the generation of superposition regions comprises;

(ca) converting each of the tiles into a representation having a plurality of layers, the representation corresponding to at least one said tiles comprising multiple coloured layers, each said tile comprising a superposition of the corresponding said coloured layers; and

(cb) merging, for each of said coloured layers, adjacent ones of said tiles, thereby generating the superpositioned regions and a multi-layered document representation of the document;

(d) calculating statistics for each said superpositioned region using pixel level statistics from each of the tiles included in said superpositioned region; and

(e) determining a classification for each superpositioned region based on the calculated statistics.

2. A method according to claim 1, wherein each superpositioned region comprises a macroregion.

3. A method according to claim 2, wherein step (d) further comprises examining statistics of said macroregions and merging macroregions having a size considered to be noise.

4. A method according to claim 3, wherein the size considered to be noise is determined by using at least one of a coverage value and a border count approach.

5. A method according to claim 4, wherein the coverage value comprises those macroregions that cover less than a predetermined number of tiles of the document image.

6. A method according to claim 4, wherein the border count approach comprises those macroregions which have a tile border count greater than a predetermined percentage of the total number of pixels in the macroregion.

7. A method according to claim 2, wherein step (e) comprises the steps of:

(ea) ascribing an initial classification to each said macroregion based on colour and shape statistics associated with said macroregion; and

(eb) assessing relationships between said macroregions to optimise the classification by merging sufficiently related ones of said macroregions.

8. A method according to claim 7, wherein step (ea) comprises classifying a macroregion according to at least one of the following statistics:

(1) at least one of an average and variance of a the number of pixels in dominant colours included in the macroregion;

(2) a total number of dominant colours included in the macroregion;

(3) a ratio of the number of dominant colours to a bounding box area surrounding the macroregion;

(4) at least one edge ratio of the dominant colours in the macroregion;

(5) an average colour value for the macroregion;

(6) a number of tiles quantised to 3 or more colours;

(7) a number of other macroregions with which the macroregion shares tiles;

(8) a total number of pixels on the tile borders of the macroregion;

(9) a contrast level between the dominant colours in the macroregion and other dominant colours from the same tile; and

(10) a total number of dominant colours in the macroregion.

9. A method according to claim 7, wherein the initial classification ascribed each macroregion to one of three classes being:

text, flat colour and image.

10. A method according to claim 9, wherein step (eb) comprises the steps of:

(eba) consolidating macroregions classified as image;

(ebb) detecting blends within the image and thereby merging corresponding macroregions;

(ebc) reclassifying macroregions based upon an extent of overlap between such 10 macroregions;

(ebd) merging foreground fragmented macroregions of the image to the same classification; and

(ebe) merging proximate macroregions having similar colours and the same classification.

11. A method according to claim 10, wherein step (ebc) comprises determining a cost function associated with pairs of macroregions and minimising the cost function to classify macroregions as text or flat.

12. A method according to claim 10, wherein step (ebd) comprises assessing the fragmentation using at least one of:

(1) a colour distance between the two macroregions;

(2) an overlap factor determined using a number of tiles in which two macroregions are present;

(3) a tile border factor associated with a number of pixels from each macroregion on a tile border;

(4) an edge ratio similarity factor based upon edge ratios of macroregions; and

(5) an amount of evidence factor based upon the incidence of evidence contributing to a likelihood of merging.

13. A method according to claim 2, further comprising the step of:

(f) forming a final output segmentation of said image by hierarchically associating at least one said macroregion within another said macroregion of different classification.

16. The method of claim 1, further comprising simplifying the superpositioned regions by:

(f) determining a text confidence value for each superpositioned region, representing a likelihood that said superpositioned region contains predominantly text-like content; and

(g) merging together different superpositioned regions with text confidence values lower than a threshold to create a simplified representation.

14. A non-transitory computer readable storage medium having a program recorded thereon, the program being executable by a computer to classify regions of a scanned document image, said program comprising:

code for partitioning the scanned image into a plurality of tiles;

code for determining at least one dominant colour for each of the plurality of tiles;

code for generating superpositioned regions based on the dominant colours, each said superpositioned region representing a group of tiles, wherein at least one tile is grouped into two superpositioned regions and each dominant colour is represented by at most one of the superposition regions, and wherein the code of generation of superpositioned regions comprises;

(a) code for converting each of the tiles into a representation having a plurality of layers, the representation corresponding to at least one said tiles comprising multiple coloured layers, each said tile comprising a superposition of the corresponding said colour layers; and

(b) code for merging, for each of said coloured layers, adjacent ones of said tiles, thereby generating the superpositioned regions and a multi-layered document representation of the document;

code for calculating statistics for each said superpositioned region using pixel level statistics from each of the tiles included in said superpositioned region; and

code for determining a classification for each superpositioned region based on calculated statistics.

15. A system for analysing and classification of a document, said system comprising:

a scanning device arranged to scan the document to form a raw pixel image of the document;

a computer apparatus configured to receive the image of the document and to partition the image into a plurality of tiles of predetermined size and to process the tiles in raster tile order, the processing comprising;

determining at least one dominant colour for each of the plurality of tiles

generating superpositioned regions based on the dominant colours, each said superpositioned region representing a group of tiles wherein at least one tile is grouped into two superpositioned regions and each dominant colour is represented by at most one of the superpositioned regions, said generating comprising;

(a) converting each of the tiles into a representation having a plurality of layers, the representation corresponding to at least one said tiles comprising multiple coloured layers, each said tile comprising a superposition of the corresponding said coloured layers; and

(b) merging, for each of said coloured layers, adjacent ones of said tiles, thereby generating the superpositioned regions and a multi-layered document representation of the document;

calculating statistics for each said superpositioned region using pixel level statistics from each of the tiles included in said superpositioned region; and

determining a classification for each superpositioned region based on the calculated statistics.