Image processing system for digital collage

• US 7,532,771 B2
• Filed: 12/12/2006
• Issued: 05/12/2009
• Est. Priority Date: 11/12/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: de Boer, William H, Taylor, Stuart, Rother, Carsten
• Primary Examiner(s): Kassa; Yosef
• Application Number:
  US11/609,802
• Publication Number:
  US 20070110335A1
• Time in Patent Office: 882 Days
• Field of Search: 382/115, 382/118, 382/178, 382/284, 382/294, 358/537, 358/540, 358/450
• US Class Current: 382/284
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

It is required to provide a framework for an automated process for forming a visually appealing collage from a plurality of input images. It is required to provide a framework for this type of automated process which is flexible and robust and which can easily be interfaced to a related software application. An image synthesis framework is provided with a modular architecture having a first module, a plurality of prior compute modules and an image synthesis module. The first module provides an application programming interface, the prior compute modules compute information about input images, and the image synthesis module uses the computed information together with the input images to form a digital collage.

Claims

1. An image processing system comprising:

one or more processors coupled to one or more memories;

an input arranged to receive a plurality of digital images for automatically forming a digital collage;

a first module implemented by the one or more processors, providing an application programming interface arranged to be used by an application in order to control the image processing system, said first module being configured to modify dimensions of the received images so that each of the images matches a normalized dimension requirement;

a plurality of prior compute modules, implemented by the one or more processors each having an interface to the first module, and each arranged to compute information to be used in determining rank and identifying regions of interest in the plurality of images; and

an image synthesis module, having an interface to the first module, implemented by the one or more processors, and arranged to automatically form the digital collage from at least some of the plurality of digital images on the basis of the information computed by the prior compute modules.

2. An image processing system as claimed in claim 1 wherein the prior compute modules are given weights that are adjustable by a user of the application for controlling importance of corresponding information computed by the prior computer modules for determining rank and regions of interest of the images.

3. An image processing system as claimed in claim 1 wherein the first module is arranged to rank the received digital images on a basis of how much information each image contains and rejecting near duplicates.

4. An image processing system as claimed in claim 1 wherein the output of each prior compute module comprises a potential matrix, being a two dimensional array of values, each value being associated with an image block.

5. An image processing system as claimed in claim 4 wherein one of the prior compute modules is arranged to produce a potential matrix comprising weights according to a Gaussian distribution.

6. An image processing system as claimed in claim 4 wherein one of the prior compute modules is arranged to produce, for a given image, a saliency potential matrix comprising contrast values calculated from that image.

7. An image processing system as claimed in claim 4 wherein one of the prior compute modules is arranged to produce, for a given image, a face potential matrix comprising information about faces detected in that image.

8. An image processing system as claimed in claim 1 wherein the first module is also arranged to determine a region of interest for each of the plurality of digital images based on the information computed by one or more of the prior compute modules.

9. An image processing system as claimed in claim 1 wherein the application programming interface comprises a function to enable a particular received image'"'"'s calculated region of interest to be modified.

10. An image processing system as claimed in claim 1 wherein the application programming interface comprises a function to enable a rank of a received image determined by the first module to be over-ridden.

11. An image processing system as claimed in claim 8 wherein an amount of information contained in the region of interest must be above a specified threshold relative to the total amount of information contained in the image.

12. An image processing system as claimed in claim 8 wherein, when a face is identified as a region of interest, the face is included as a whole in the region of interest.

13. An image processing system as claimed in claim 8 wherein the image synthesis module applies an optimization process to pack as many of the regions of interest as possible into a predetermined width and height of the digital collage, wherein as many collage pixels as possible are covered by some input image without overlapping of the regions of interest.

14. An image processing system as claimed in claim 1 further comprising an application arranged to control the image processing system, that application being arranged to display a user interface at a display, the user interface comprising items to control selection of prior compute modules and selection of relative weights to be used in association with results of the selected prior compute modules.

15. A method of image processing comprising the steps of:

receiving, by a first module implemented by one or more processors, a plurality of digital images for inclusion in a digital collage;

providing at the first module at least an application programming interface implemented by the one or more processors and arranged to be used by an application in order to control the image processing method;

using a plurality of prior compute modules, implemented by the one or more processors, each prior computer module having an interface to the first module, to compute information to be used in determining a rank and a region of interest for each of the images; and

using an image synthesis module, implemented by the one or more processors and having an interface to the first module, to automatically form the digital collage from at least some of the plurality of digital images on the basis of the information computed by the prior compute modules by positioning as many images with highest rank into an area specified for the digital collage without allowing the regions of interest to overlap and while covering as many pixels of the area as possible.

16. A method as claimed in claim 15 which comprises using the first module to modify the dimensions of the received digital images so that each of the images matches a normalized dimension requirement.

17. A method as claimed in claim 15 which comprises using the first module to rank the received digital images on the basis of at least one metric using the information computed by one or more of the prior compute modules.

18. A method as claimed in claim 15 which comprises using each prior compute module to produce a potential matrix, being a two dimensional array of values, each value being associated with an image block.

19. A method as claimed in claim 18 further comprising giving weighting values to the prior compute modules that are adjustable by a user of the application for controlling importance of corresponding information computed by the prior computer modules for determining rank and regions of interest of the images.

20. A method of image processing comprising:

receiving, by a first module implemented by one or more processors, a plurality of digital images for use in automatically forming a digital collage;

providing an application programming interface, implemented by the one or more processors, arranged to be used by an application in order to control the image processing method;

modifying, by a second module implemented by the one or more processors, dimensions of the received digital images so that each of the images matches a normalized dimension requirement of a uniform size;

ranking, by a third module implemented by the one or more processors, the images relative to each other on a basis of how much information is contained in each image and rejecting near duplicates;

computing, by a fourth module implemented by the one or more processors, a region of interest for each received image according to specified constraints;

positioning, by a fifth module implemented by the one or more processors, as many images with highest rank into an area specified for the digital collage without allowing the regions of interest to overlap and while covering as many pixels of the area as possible; and

blending, by a sixth module implemented by the one or more processors, adjacent images positioned in the area using an edge-sensitive blending process whereby seams are created between the images that switch automatically between cutting along natural boundaries and blending transparently according to the presence or absence, respectively, of underlying sharp edges.