Image-based modeling and photo editing

• US 20040095357A1
• Filed: 05/20/2003
• Published: 05/20/2004
• Est. Priority Date: 05/21/2002
• Status: Active Grant

• Current Assignee: Smarter Systems, Inc.
• Original Assignee: Mok3 Inc.
• Inventors: Chen, Max, Oh, Byong Mok, Durand, Fredo
• Granted Patent: US 7,199,793 B2
• Time in Patent Office: Days
• US Class Current: 345/589
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

The invention provides a variety of tools and techniques for adding depth information to photographic images, and for editing and manipulating images that include depth information. The tools for working with such images include tools for “painting” in a depth channel, for using geometric primitives and other three-dimensional shapes to define depth in a two-dimensional image, and tools for “clone brushing” portions of an image with depth information while taking the depth information and lighting into account when copying from one portion of the image to another. The tools also include relighting tools that separate illumination information from texture information.

Claims

1. A data structure for representing an image including depth information, the data structure comprising two or more layers, each layer comprising:

an object representing a reference camera;

a collection of color channel values; and

a collection of depth channel values, each depth channel value in the collection of depth channel values corresponding to a color channel value in the collection of color channel values.

2. The data structure of claim 1, wherein each layer further comprises a collection of alpha-channel values, each alpha channel value in the collection of alpha channel values corresponding to a color channel value in the collection of color channel values.

3. The data structure of claim 1, wherein each layer further comprises a collection of texture channel values, each texture channel value in the collection of texture channel values corresponding to a color channel value in the collection of color channel values.

4. The data structure of claim 1, wherein each layer further comprises a collection of illuminance channel values, each illuminance channel value in the collection of illuminance channel values corresponding to a color channel value in the collection of color channel values.

5. The data structure of claim 1, wherein each layer further comprises a collection of normal channel values, each normal channel value in the collection of normal channel values corresponding to a color channel value in the collection of color channel values.

6. The data structure of claim 1, wherein the collection of color channel values comprises a two-dimensional array of color channel values, each entry in the two dimensional array representing the color channel value of a pixel in an image.

7. The data structure of claim 1, wherein the object representing the reference camera comprises a world-to-image projection matrix that specifies a field of view of the reference camera.

8. A method of interactively editing an image containing depth information and reference camera information, the method comprising:

a) displaying the image from a viewpoint defined by an interactive camera;

b) receiving an edit to the image;

c) transforming the edit to a viewpoint defined by the reference camera; and

d) applying the transformed edit to the image.

9. The method of claim 8, wherein receiving an edit comprises receiving an edit to color information associated with the image.

10. The method of claim 8, wherein receiving an edit comprises receiving an edit to alpha information associated with the image.

11. The method of claim 8, wherein receiving the edit comprises receiving an edit to depth information associated with the image.

12. The method of claim 11, wherein receiving an edit to depth information comprises:

providing a user with an interactive drawing tool that specifies edits to depth information; and

receiving edits to depth information made by the user using the interactive drawing tool.

13. The method of claim 12, wherein the interactive drawing tool specifies a selected value for depth for a selected portion of the image.

14. The method of claim 12, wherein the interactive drawing tool incrementally adds to the depth for a selected portion of the image.

15. The method of claim 12, wherein the interactive drawing tool incrementally subtracts from the depth for a selected portion of the image.

16. A method of assigning depth to an object in an image, the method comprising:

a) interactively determining a ground reference for the image; and

b) assigning depth information to the object based on the ground reference.

17. The method of claim 16, wherein interactively determining a ground reference comprises specifying a horizon line in the image.

18. The method of claim 16, wherein interactively determining a ground reference comprises specifying a scale factor on depth.

19. The method of claim 16, wherein assigning depth information comprises using an interactive depth painting tool to specify the depth information.

20. The method of claim 16, wherein assigning depth information comprises using geometric primitives to specify the depth information.

21. The method of claim 16, wherein assigning depth information comprises using a three-dimensional model of an object to specify the depth information.

22. The method of claim 16, wherein the ground reference comprises a planar surface.

23. The method of claim 16, wherein the ground reference comprises a non-planar surface.

24. A method of painting in an image that includes depth information, the method comprising:

a) providing a source position and a destination position in the image;

b) identifying a destination region in the image relative to the destination position;

c) determining a source region in the image relative to the source position and corresponding to the destination region;

d) transforming the image information of the source region relative to the depth information of the source region to image information relative to the depth information of the destination region; and

e) copying the transformed image information to the destination region.

25. The method of claim 24, wherein the image comprises two or more layers.

26. The method of claim 25, wherein the source position is in a first layer, and destination position is in a second layer.

27. The method of claim 24, wherein transforming the image further comprises transforming the image relative to lighting information of the source and destination regions.

28. The method of claim 24, wherein transforming the image further comprises transforming the image relative to texture information of the source and destination regions.

29. The method of claim 24, wherein:

the destination region is defined relative to a destination reference camera;

the source region is defined relative to a source reference camera; and

transforming the image information further comprises transforming the image information from a viewpoint defined by the source reference camera to a viewpoint defined by the destination reference camera.

30. The method of claim 24, further comprises temporarily displaying the area locally around the destination region initialized with image information that approximates the area locally around the source region.

31. The method of claim 30, wherein the approximation comprises a geometric mapping.

32. The method of claim 31, wherein the geometric mapping comprises a planar mapping.

33. The method of claim 24, wherein transforming the image occurs substantially concurrently with a user identifying the destination region.

34. The method of claim 33, wherein transforming the image comprises computing a parameterization only for selected active pixels concurrently with the user identifying the destination region.

35. The method of claim 24, further comprising factoring the image information into a texture component and an illumination component, and wherein copying the transformed image comprises copying the texture component of the transformed image and applying the illumination component of the destination region.

36. A method for determining a texture component and an illumination component of an image, the method comprising:

determining sizes of a small-scale feature in the image and a large-scale feature in the image; and

using an edge-preserving filter to place small-scale features into the texture component and large-scale features into the illumination component.

37. The method of claim 36, wherein determining sizes comprises interactively selecting a feature size of a texture.

38. The method of claim 36, wherein determining sizes further comprises determining a size and shape of a filter kernel, and wherein the edge-preserving filter evaluates image information over an area defined by the filter kernel.

39. The method of claim 38, wherein the image comprises depth information, and wherein determining a size and shape of the filter kernel comprises using the depth information to determine the size and shape of the filter kernel.

40. The method 38 wherein the image comprises normal information, and wherein determining a size and shape of the filter kernel comprises using the normal information to determine the size and shape of the filter kernel.

41. A system for interactively editing a three-dimensional image, the system comprising:

a computer comprising a processor, memory, and a display, the memory containing instructions that, when executed by the processor, cause the computer to;

receive an input image;

interact with a user to segment the input image into a plurality of layers; and

interact with a user to apply depth information to the plurality of layers.

42. The system of claim 41, wherein the input image comprises a two-dimensional image.

43. The system of claim 41, wherein the input image comprises a three-dimensional image including depth information.

44. The system of claim 41, wherein the instructions, when executed by the processor, further cause the computer to interact with the user to edit the plurality of layers.

45. The system of claim 41, wherein the instructions, when executed by the processor, further cause the computer to interact with the user to relight the plurality of layers.

46. The system of claim 41, wherein the instructions, when executed by the processor, further cause the computer to interact with the user to copy a source portion of one of the plurality of layers to a destination portion of one of the plurality of layers, transforming the source portion in response to differences in the depth information of the source portion and the destination portion.