Iterative three-dimensional process for creating finished media content

• US 6,160,907 A
• Filed: 10/10/1997
• Issued: 12/12/2000
• Est. Priority Date: 04/07/1997
• Status: Expired due to Fees

• Current Assignee: Synapix Incorporated
• Original Assignee: Synapix Incorporated
• Inventors: Robotham, John S., French, Michael T., Rawley, Curt A.
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US08/948,865
• Time in Patent Office: 1,159 Days
• Field of Search: 382/282, 382/284, 382/285, 382/154, 382/299, 382/308, 382/278, 382/173, 345/418, 345/420, 345/419, 345/421, 345/425, 345/426, 345/430, 345/431, 345/432, 345/473, 345/474
• US Class Current: 382/154
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

An integrated media production technique for refining a finish quality media content. The technique permits a media production to be defined as a computerized choreography model of a virtual stage containing data objects representing live/recorded image elements as well as synthetic-abstract elements and multiple versions thereof. Elements of the choreography model are iteratively rendered, blended, adjusted, and selected until a desired quality result is achieved.

Claims

1. A method for developing finished media content from an input visual image sequence, the input visual image sequence containing a visual representation of one or more visual objects, the method comprising the steps of:

(a) analyzing portions of the input visual image sequence to derive a scene model in three spatial dimensions with respect to a three-dimensional object space by performing the steps of;

(i) defining a common three-dimensional object space;

(ii) defining an image-based data object containing a segmented pixel representation corresponding to a portion of at least one frame of the input visual image sequence within the three-dimensional object space;

(iii) defining an abstraction-based data object containing an abstract model for at least a portion of one of the visual objects contained in the input visual image sequence within the same three-dimensional object space;

(b) defining a choreography specification for the finished media content that includes a specification for how the data objects are to be used in creating the finished media content;

(c) selecting an image-based data object to be modified and accepting user input parameters modifying how the selected image-based data obiect is to be used in the choreography specification for generating the finished media content;

(d) selecting an abstraction-based data object to be modified and accepting user input parameters modifying how the selected abstraction-based data object is to be used in the choreography specification for generating the finished media content;

(e) generating a two-dimensional (2D) visual image of the finished media content from the choreography specification, scene model, at least one pixel representation, and at least one abstract model; and

(f) iteratively refining the finished media content by performing certain selected ones of steps (a) through (e) in an order as selected by user input until the finished media content is of acceptable finish quality.

2. A method as in claim 1 wherein step (b) additionally comprises:

choreographing representations of objects within the context of a three-dimensional (3D) virtual stage.

3. A method as in claim 1 wherein at least one parameter of a virtual stage is defined by analyzing the input visual image.

4. A method as in claim 1 additionally comprising the step of, after step (a):

storing a link in correlation mesh data object wherein the link indicates a correspondence between an image-based data object and an abstraction-based data object.

5. A method as in claim 1 wherein step (a) further comprises the step of:

defining a spatial position for the pixel representation with respect to a virtual stage.

6. A method as in claim 1 wherein the input visual image is a visual image stream comprising a time sequence of two or more image frames.

7. A method as in claim 6 wherein step (b) additionally comprises the step of:

defining a choreography specification that includes a specification of one or more time-sequenced manipulations of at least one of the abstract models in a virtual stage.

8. A method as in claim 7 wherein step (e) further comprises the step of:

generating a time-based sequence of 2D output visual images from at least one of the choreography specification, the abstract models, and the pixel representations.

9. A method as in claim 1 wherein step (f) additionally comprises the step of:

during a given iteration, accepting user input specifying which of a plurality of pixel representations and which of a plurality of abstract models are used in the output generating step.

10. A method as in claim 1 additionally comprising the step of:

defining a camera model including a specification of a viewpoint of a camera used to generate the input visual image.

11. A method as in claim 10 wherein step (e) further comprises the step of:

generating the output visual image from the viewpoint of the camera as specified by the camera model.

12. A method as in claim 10 wherein the camera model additionally specifies at least one camera parameter taken from the group consisting of camera position, movement, resolution, field of view, depth of field, focal length, lens f-stop, exposure interval and orientation.

13. A method as in claim 1 wherein the visual object in the input visual image comprises a physical object.

14. A method as in claim 1 wherein the visual object in the input visual image comprises a synthetic object.

15. A method as in claim 1 wherein the abstract models comprise models of multiple physical objects.

16. A method as in claim 1 wherein the abstract models comprise a model of at least one synthetic object.

17. A method as in claim 1 additionally comprising the step of:

defining a light model specifying at least one light parameter of a light that can be used to generate the output visual image, with at least one light parameter taken from the group consisting of light position, light movement, light intensity, light color, and light orientation.

18. A method as in claim 1 additionally comprising the step of:

generating an audio stream synchronized with the output visual image stream.

19. A method as in claim 1 wherein the step (e) of generating an output visual image additionally comprises the step of:

determining at least one visual effect on at least one visual object.

20. A method as in claim 1 wherein step (b) further comprises additionally comprises specifying a visual effect upon at least one of the visual objects.

21. A method as in claim 20 wherein the visual effect comprises a visual effect selected from the group consisting of shadows, reflections, and refractions.

22. A method as in claim 20 wherein the visual effect comprises a visual effect selected from the group consisting of atmospheric and object geometry distortions.

23. A method as in claim 20 wherein the visual effect comprises an inter-object morphing effect.

24. A method as in claim 20 wherein the visual effect comprises changes to at least one object surface attribute.

25. A method as in claim 24 wherein the object surface attribute is selected from the group consisting of surface texture, reflectivity, and opacity.

26. A method as in claim 1 additionally comprising the step of:

generating an input specification and parameters for an external 3D rendering system used in the output generating step (e).

27. A method as in claim 1 additionally comprising the step of:

accepting a user input specifying how a visual image specified by more than one pixel representation and abstract model is to be blended into a single composite output visual image.

28. A method as in claim 1 wherein step (b) of defining a choreography specification additionally comprises the step of:

specifying the blending of multiple pixel representations and abstract models of a given visual object.

29. A method as in claim 28 wherein the blending specification step additionally comprises the step of:

rendering each of a plurality of abstract models first to a two-dimensional 2D pixel representation, each 2D pixel representation including a plurality of pixel data elements, and then blending the multiple 2D pixel representations.

30. A method as in claim 29 wherein step blending specification step additionally comprises the step of:

determining an image blending attribute for the visual object.

31. A method as in claim 30 wherein the image blending attribute is at least one attribute selected from the group consisting of depth from camera, image transparency, and surface normal.

32. A method as in claim 30 wherein step (e) additionally comprises the step of:

blending multiple 2D pixel representations by comparing the image blending attribute on a pixel by pixel basis.

33. A method as in claim 30 wherein step (e) additionally comprises the step of:

blending multiple 2D pixel representations by comparing the image blending attribute on a pixel region by pixel region basis.

34. A method as in claim 30 wherein step (e) additionally comprises the step of:

blending multiple 2D pixel representations by comparing the image blending attribute on a sub-pixel by sub-pixel basis.

35. A method as in claim 4 additionally comprising the step of:

during a given iteration, accepting user input specifying changes to the correlation mesh in response to display of the 2D finish representation to the user.

36. A method as in claim 1 additionally comprising the step of:

during a given iteration, accepting user input specifying changes to parameters of an automatic segmenting process that performs step (a)(i).

37. A method as in claim 1 additionally comprising the step of:

during a given iteration, accepting user input specifying changes to parameters of an automatic abstract modeling process that performs step (a)(ii).

38. A method as in claim 1 wherein step (a)(i) additionally comprises the steps of:

creating multiple pixel representation versions of a given input visual image, the multiple pixel representation versions being at different levels of image resolution; and

using different ones of the pixel representation versions in given iterations of performing step (a)(i).

39. A method as in claim 38 additionally comprising the step of:

presenting an interactive display of an intermediate finish representation from a given iteration whereby a user specifies which one of the pixel representation versions are to be used in a subsequent iteration.

40. A method as in claim 38 wherein the different levels of image resolution are for different spatial resolutions.

41. A method as in claim 38 wherein the different levels of image resolution are for different color space resolutions.

42. A method as in claim 1 wherein the step of defining an abstract model additionally comprises the steps of:

creating multiple abstract model versions of a given input visual image, the abstract model versions being at different levels of modeling detail; and

using different ones of the abstract model versions in given iterations of performing step (f).

43. A method as in claim 42 additionally comprising the step of:

presenting an interactive display of an intermediate finish representation from a given iteration whereby a user specifies which one of the abstract model versions are to be used in a subsequent iteration.

44. A method as in claim 42 wherein the different levels of modeling detail are different levels of geometry detail.

45. A method as in claim 42 wherein the different levels of modeling detail are different representations of surface textures.

46. A method as in claim 7 additionally comprising the step of:

providing an application programming interface that permits access to the pixel representation(s), abstract model(s) choreography specification, correlation mesh, or visual image(s).

47. A method as in claim 7 additionally comprising the step of:

modifying the output audio stream in accordance with changes made to at least one object in the virtual stage through the choreography step.

48. A data processing system for generating finish quality media from one or more source images of a scene comprising:

a scene model developed by analyzing portions of the source images, the scene model defining

(i) a three-dimensional virtual space;

(ii) an image-based data object containing a segmented pixel representation corresponding to a portion of at least one frame of the source images;

(iii) an abstraction-based data object containing an abstract model for at least a portion of one of the visual objects contained in at least one of the source images, the abstraction-based data object being defined with respect to the three-dimensional virtual space;

a choreography specification for defining how the finish quality media is to be developed from the scene model and both the image-passed and abstraction-based data objects;

means for accepting user input parameters modifying how selected data objects are to be used with the choreography specification in generating the finish quality media;

an image renderer, connected to receive the scene model, the data objects, and the choreography specification as inputs, for rendering a two-dimensional visual representation of each data object;

means for accepting user inputs in response to the two-dimensional visual representation produced by the image renderer, the user inputs defining how the data objects and the choreography specification are to he used in iteratively refining the finish quality media.

49. A system as in claim 48 wherein the scene model is developed from two source images.

50. A system as in claim 49 wherein the two source images comprise two live/recorded images.

51. A system as in claim 49 wherein the two source images comprise a live/recorded image and a synthetic image.

52. A system as in claim 49 wherein the two source images comprise two synthetic images.

53. A system as in claim 48 wherein the image renderer additionally blends the rendered data objects using depth information derived from the source images.

54. A system as in claim 48 wherein the image renderer additionally blends the rendered data objects using transparency information.

55. A system as in claim 48 wherein the scene model includes image-based data objects and abstract data objects.