Model based image editing and correction

• US 5,990,901 A
• Filed: 06/27/1997
• Issued: 11/23/1999
• Est. Priority Date: 06/27/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Monson, John A., Lawton, Daryl T.
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): NGUYEN, THU V
• Application Number:
  US08/883,718
• Time in Patent Office: 879 Days
• Field of Search: 345/429-432, 345/433, 345/434, 345/435, 382/115-118, 382/308, 382/311
• US Class Current: 345/581
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/44 Morphing

Abstract

A system and method is described for automatically editing digital images. An image to be edited is selected and registered with a model that is an abstract of features and attributes common to a category of images to which the selected image belongs. In the illustrated embodiment, a human face is used as an example of an image to be edited and the matching model is a representation of features and attributes common to all human faces. Many other images of objects may also be edited using the techniques of the invention, as long as a general class of objects to which an image belongs is amenable to being represented by a model comprising a set of common features or attributes. A library of different models is available to the user for selecting the model matching the image to be edited. The library includes different models of the same object in different orientations and may also include models of different objects. After the appropriate model is selected, the user is prompted to register the details of the selected image with the generalized features and attributes of the selected model. Editing processes or tools, which are linked to the model, are then automatically appled to the image in order to perform a desired editing function. In addition to applying editing techniques to existing images in a larger image, the invention enables a user to paste and blend images into another, larger image--e.g., create a morphing effect by pasting one persons head over the head of another in an original image.

Claims

1. A method for automatically editing digital images comprising the steps of:

selecting for editing an object within an original image;

selecting a model from a library of models that best matches the selected object;

prompting a user to register the selected object with the selected model;

storing information mapping attributes of the selected model to the selected object; and

, transforming the original image to an edited image by automatically applying one or more editing tools to the selected object based on the mapped attributes of the selected model.

2. The method of claim 1 wherein the step of transforming the original image to the edited image includes a step of pasting a replacement object over the selected object of the original image, where placement of the pasted replacement image into the original image is constrained by the registration between the selected object and model.

3. The method of claim 2 wherein the replacement object has attributes of a same type as those of selected model and object, but the attributes of the replacement object am visually distinct from the attributes of the selected object so that the pasting of the replacement object over the selected object results in an appearance of the selected object being morphed to a different visual appearance.

4. The method of claim 2 wherein the editing tools include gradient extraction and filling, texture mapping, boundary blending, shading and compositing.

5. The method of claim 2 wherein the replacement object is a pre-stored image linked to the editing tools.

6. The method of claim 1 wherein the step of prompting the user to register the selected object with the selected model includes the steps of (1) displaying nodes of the model whose locations represent locations of the attributes in the model that are to be mapped to the selected object and (2) the user identifying locations of attributes of the selected object corresponding to the locations of the attributes of the model, which are identified by the nodes.

7. The method of claim 1 wherein the step of selecting a model from the library of models includes the step of identifying an editing effect to be performed on the selected object.

8. The method of claim 1 wherein the step of registering the selected object with the selected model includes the steps of initiating the registration by first selecting attributes indicating the size of the image;

determining from the size of the selected object the amount of detail in the selected object likely to be discernible by the user; and

tailoring the number of attributes of the model the user is prompted to match to the selected object so as to complement the likely amount of visual detail in the selected object to be edited.

9. The method of claim 1 including wherein each of the models has two and three dimensional attributes.

10. The method of claim 9 wherein the two-dimensional attributes of the selected model are registered with the selected object of the original image and warped to fit the size and orientation of the selected object established by the registration.

11. The method of claim 10 wherein the two-dimensional attributes of each of the models are embedded in one or more grids, which include (1) nodes that correspond to locations of attributes of the selected model that the user interactively registers to pixel locations in the original image that represent like attributes of the selected object, (2) arcs that describe piecewise splines of the model, (3) regions of the model defined by closed paths of the arcs and (4) iterators, which describe areas surrounding the nodes, arcs and regions for blending or texturing.

12. The method of claim 1 wherein the registration of the selected object with the selected model includes the step of marking an object that occludes the image to be edited.

13. A computer system comprising;

a library of object models;

a data structure for recording the registration of one of the models from the library with an object selected from an original image displayed on a user interface;

an editing tool associated with the model;

a processor for applying the editing tool to the selected object based on the registration of the selected object to the selected model; and

, a display memory containing an edited image resulting from a transformation of the original image by the editing tool.

14. The computer system of claim 13 wherein the editing tool is registered to the selected model, which in turn is registered to the selected object to be edited in the original image.

15. The computer system of claim 14 wherein the editing tool includes a pre-stored image for pasting over the selected object in the original image.

16. The computer system of claim 14 wherein the selected object is a first object and the computer system includes a data structure mapping attributes of the selected model to a second object for constraining the pasting of the second object over the first object, where the editing tool includes a technique for blending the pasted second object into a original image.

17. The computer system of claim 14 wherein the editing tool include gradient extraction and filling, texture mapping, boundary blending, shading and compositing.

18. The computer system of claim 16 wherein the registered attributes of the second object are visually distinct from the registered attributes of the first object, resulting in the appearance of the first object being morphed in to the second object when the second object is pasted over the first object.

19. The computer system of claim 13 including means for creating the data structure by prompting the user to register the selected object with the selected model, where the prompting includes (1) displaying nodes of the selected model whose locations represent locations of attributes common to all images that are properly matched to the model and (2) recording the locations in the original image of attributes of the selected object identified by the user in response to the displayed nodes.

20. The computer system of claim 13 wherein the library of models is indexed to a menu of editing effects displayed to the user for the purpose of selecting the appropriate one of the models from the library for registering with the selected object.

21. The computer system of claim 13 wherein the data structure for registering the selected model with the selected object maps attributes of the model to pixels of the original image.

22. The computer system of claim 13 wherein each of the models comprises a data structure defining a three-dimensional model from which two-dimensional attributes are derived for registering with the selected object in the original image.

23. The computer system of claim 22 wherein the two-dimensional attributes include one or more two-dimensional grids containing nodes of the model that define locations of attributes of the model in a two-dimensional projection of the three-dimensional model.

24. The computer system of claim 23 including means for displaying the nodes in a sequence for prompting the user to interactively register each of the nodes with one of attributes of the selected object in the original image.

25. The computer system of claim 13 wherein the selected model includes a two-dimensional grid containing (1) nodes demarking locations of selected attributes of the model, (2) arcs that describe piecewise splines of the model, (3) regions of the model and (4) iterators, which describe areas surrounding the nodes, arcs and regions for constraining the application of blending or texturing tools to the original image.

26. The computer system of claim 13 including a copy of an object occluding the object to be edited, where the copy is pasted over the pre-stored image.

27. In a computer for receiving and displaying digital images, an article of manufacture comprising;

a medium that is readable by the computer and that carries instructions for the computer to perform a process comprising the steps of;

selecting in an original image an object to be edited;

selecting from a library of models a model that is linked to an editing tool for the desired editing effect to be applied to the selected object;

registering attributes of the model with the same attributes of the selected object;

automatically applying the editing tool to the selected object based on a mapping of the attributes of the selected model to the selected object to create an edited image, which is the original image modified to include the desired editing effect resulting from the application of the editing tool to the selected object; and

, displaying the edited image in place of the original image.

28. The article of manufacture of claim 27 wherein the step of registering the selected object with the selected model includes the step of prompting a user to register the model with the selected object at two or more nodes of the model that represent attributes common to all objects that are properly matched to the model.

29. The article of manufacture of claim 27 wherein the step of prompting the user to register the selected object with the selected model includes the steps of (1) displaying nodes of the model whose locations represent locations of attributes common to all images that are properly matched to the model and (2) identifying locations of attributes of the selected object corresponding to the attributes of the model represented by the displayed the nodes.

30. The article of manufacture of claim 27 wherein the editing tool is registered to the selected model, which in turn is registered to the selected object to be edited in the original image.

31. The article of manufacture of claim 30 wherein one of the editing tools includes a tool for pasting a pre-stored image over the selected object in the original image, where the pre-stored image is part of the editing tool.

32. The article of manufacture of claim 27 wherein the selected object is a first object and the article includes a data structure for registering the model to a second object, which enables the second object to be pasted over the first object, where the editing tools include tools for blending the pasted second object into the original image.

33. The article of manufacture of claim 31 wherein the editing tool includes a texturing tool for blending the pasted, pre-stored image into the original image to create an edited image.

34. The article of manufacture of claim 32 wherein attributes of the second image registered to the selected model are visually distinct from attributes of the first object that are registered to the model, resulting in the appearance of the first object being morphed into the second object when the second object is pasted over the first object.

35. The article of manufacture of claim 27 wherein the models in the library of models are indexed to a menu of editing effects, which is displayed to the user for selection of the desired editing effect.

36. The article of manufacture of claim 27 including initiating the registration of the selected object with the model by first registering attributes of the model and the selected object that indicate the size of the selected object;

and tailoring any additional registration of attributes of the model and the selected object to complement the likely amount of detail visually discernible to the user based on the size of the selected object.

37. The article of manufacture of claim 27 including wherein each of the models has two and three dimensional attributes.

38. The article of manufacture of claim 37 wherein the two-dimensional attributes of the model are first registered with the selected object and then warped to complement the shape and orientation of the selected object.

39. The article of manufacture of claim 37 wherein the two-dimensional attributes of each of the models are embedded in one or more grids and include (1) nodes that mark the locations of attributes in the model which are mapped to attributes in the selected image during the registration process, (2) arcs that describe piecewise splines of the model, (3) regions of the model and (4) iterators, which describe areas surrounding the nodes, arcs and regions for constraining the application of blending or texturing tools to the original image.

40. The article of manufacture of claim 27 wherein the registration of the selected object with the selected model includes the step of marking an object that occludes the image to be edited.