Image processing apparatus, image processing method and computer program

• US 20080056572A1
• Filed: 08/27/2007
• Published: 03/06/2008
• Est. Priority Date: 08/29/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nielsen, Frank
• Granted Patent: US 7,986,831 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering

Abstract

An image processing apparatus for extracting pictures-in-picture information contained in an image, includes an image segmentation unit segmenting an input image into regions, and a region extraction unit extracting a pictures-in-picture region containing the pictures-in-picture information on the basis of a contour of a segmented region.

Claims

1. An image processing apparatus for extracting pictures-in-picture information contained in an image, comprising:

image segmenting means for segmenting an input image into regions; and

region extracting means for extracting a pictures-in-picture region containing the pictures-in-picture information on the basis of a contour of a segmented region.

2. The image processing apparatus according to claim 1, further comprising:

quadrilateral detecting means for detecting a quadrilateral from the pictures-in-picture region extracted by the region extracting means; and

image converting means for subjecting a detected quadrilateral to inverse projective transformation so as to obtain an image viewed from a front, and adjusting an aspect ratio thereof.

3. The image processing apparatus according to claim 1, wherein

the image segmenting means includes:

an incidence graph creating means for creating an incidence graph in which adjacent nodes are connected by an edge, from an input image formed with nodes, the node being a pixel or a polygon formed from a cluster of pixels;

an incidence graph evaluating means for assigning a weight to the edge on the basis of a result of comparison between attribute values of each node connected by the edge, and sorting the edges in the incidence graph on the basis of their weights; and

a merge processing means for extracting a pair of the nodes sandwiching the edge in sorted order of the edges, and evaluating whether or not the pair of the nodes should be merged on the basis of a statistical processing algorithm, to merge the nodes,

wherein the input image is segmented into a plurality of image regions respectively formed from the corresponding nodes.

4. The image processing apparatus according to claim 3, further comprising:

small region merging means for merging a small region left behind as a result of the merge processing performed on the image regions by the merge processing means.

5. The image processing apparatus according to claim 3, wherein

the incidence graph evaluating means assign a difference between the attribute values of the nodes connected by the edge in the incidence graph as the weight of the edge, and sorts the edges by weight in increasing order.

6. The image processing apparatus according to claim 3, wherein

the merge processing means judge whether or not the nodes connected by the edge in the incidence graph should be merged on the basis of a predicate derived from a phenomenon of concentration inequality in areas of image regions forming the nodes.

7. The image processing apparatus according to claim 3, wherein

the image segmenting means perform first image segmenting processing for exporting a foreground region including individual pictures in picture from a source image, and second image segmenting processing for exporting a background region excluding the pictures in picture from the source image, in a two-step manner.

8. The image processing apparatus according to claim 7, further comprising:

a differential processing means for perform differential-processing on the input image,

wherein the incidence graph creating means assign, to a pixel which frequency band is less than a predetermined value in a differentiated image outputted from the differential processing means, an edge for connection with another pixel, to create a partial incidence graph, and

the merge processing means merge the nodes on a border or touching thereto to merge the background into a single region, to export the foreground image region including the pictures in picture, and to export the background image region on which the pictures in picture are placed.

9. The image processing apparatus according to claim 1, wherein

the region extracting means discard the small region contained in another region, and extracts a region formed from an outermost contour.

10. An image processing method for extracting pictures-in-picture information contained in an image, comprising:

an image segmenting step of segmenting an input image per region; and

a region extracting step of extracting a pictures-in-picture region containing the pictures-in-picture information on the basis of a contour of a segmented region.

11. The image processing method according to claim 10, further comprising:

a quadrilateral detecting step of detecting a quadrilateral from the pictures-in-picture region extracted in the region extracting step; and

an image converting step of subjecting the detected quadrilateral to inverse projective transformation into an image viewed from the front, and adjusting an aspect ratio thereof.

12. The image processing method according to claim 10, wherein

the image segmenting step includes:

an incidence graph creating step of creating an incidence graph in which adjacent nodes are connected through an edge, from the input image formed from a node being a pixel (or a polygon formed from a cluster of pixels);

an incidence graph evaluating step of giving the edge a weight on the basis of a result of comparison between attribute values of each node connected through the edge, and sorting edges in the incidence graph on the basis of the corresponding weights; and

a mesh-growing step of extracting a pair of nodes sandwiching an edge in sorted order of the edges, and evaluating whether or not the pair of nodes should be merged on the basis of a statistical processing algorithm, to mesh-grow the nodes,

wherein the input image is segmented into a plurality of image regions formed from individual nodes.

13. The image processing method according to claim 12, further comprising:

a small region merging step of merging a small region remaining as a result of the mesh growing performed on the image regions in the mesh-growing step.

14. The image processing method according to claim 12, wherein

the incidence graph evaluating step gives a difference between attribute values of each node connected through the edge in the incidence graph as a weight of the edge, and sorts weights in increasing order.

15. The image processing method according to claim 12, wherein

the image region mesh-growing step judges whether or not the nodes connected through the edge in the incidence graph should be merged on the basis of a predicate derived from a phenomenon of concentration inequality in areas of image regions forming the nodes.

16. The image processing method according to claim 12, wherein

the image segmenting stepper forms first image segmenting processing for exporting a foreground region including individual pictures in picture from a source image, and second image segmenting processing for exporting a background region excluding the pictures in picture from the source image, in a two-step manner.

17. The image processing method according to claim 16, further comprising:

a differential processing step of differential-processing the input image,

wherein the incidence graph creating step gives, only to a pixel belonging to a frequency band which is less than a predetermined value in a differentiated image obtained in the differential processing step, an edge for connection with another pixel, to create a partial incidence graph, and

the mesh-growing step mesh-grows nodes on a border to merge the background into a single region, to export the foreground image region including the pictures in picture, and to export the background image region on which the pictures in picture are placed.

18. The image processing method according to claim 10, wherein

the region extracting step discards a small region containing another region, and extracts only a region formed from an outermost contour.

19. A computer program described in a computer-readable form for execution of processing for extracting pictures-in-picture information contained in an image on a computer, the computer program causing the computer to execute:

an image segmenting processing for segmenting an input image into regions;

a region extracting processing for extracting a pictures-in-picture region containing the pictures-in-picture information on the basis of a contour of a segmented region;

a quadrilateral detecting processing for detecting a quadrilateral from the pictures-in-picture region extracted in the region extracting procedure; and

an image converting processing for subjecting the detected quadrilateral to inverse projective transformation into an image viewed from the front, and adjusting an aspect ratio thereof.

20. An image processing apparatus for extracting pictures-in-picture information contained in an image, comprising:

an image segmentation unit segmenting an input image into regions; and

a region extraction unit extracting a pictures-in-picture region containing the pictures-in-picture information on the basis of a contour of a segmented region.

21. An image processing method for extracting pictures-in-picture information contained in an image, comprising:

segmenting an input image per region; and

extracting a pictures-in-picture region containing the pictures-in-picture information on the basis of a contour of a segmented region.