Media production with correlation of image stream and abstract objects in a three-dimensional virtual stage

• US 6,084,590 A
• Filed: 10/10/1997
• Issued: 07/04/2000
• Est. Priority Date: 04/07/1997
• Status: Expired due to Fees

• Current Assignee: Synapix Incorporated
• Original Assignee: Synapix Incorporated
• Inventors: Robotham, John S., Rawley, Curt A.
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): GOOD JOHNSON, MOTILEWA
• Application Number:
  US08/949,166
• Time in Patent Office: 998 Days
• Field of Search: 345/473, 345/419
• US Class Current: 345/419
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

In a media production system, two-dimensional images captured from physical objects are analyzed to create three-dimensional representations of the physical objects within a virtual stage. Representations of physical and synthetic objects are manipulated and choreographed in the virtual stage within an integrated system which also generates a media product in the form of a two-dimensional image stream of the physical and synthetic objects with optional synchronized audio tracks. A correlation is maintained between object models in the virtual stage and the original two-dimensional images so that information from those images can be included in the final product. To facilitate direction of combined media products, a composite display of the three-dimensional virtual stage and a two-dimensional preview are provided. Also provided are a score display providing temporal representations of elements within the scene and an object catalog of object oriented representations of elements within the scene.

Claims

1. A method of media production comprising:

capturing at least one image stream from physical objects in a physical object space; and

in a data processing system, defining a three-dimensional virtual stage;

analyzing the image stream to define representations of physical objects in the virtual stage;

maintaining a correlation between representations of objects in the virtual stage and corresponding segments of the at least one image stream, the correlation being stored in a correlation mesh data structure linking representations of objects in the virtual stage to corresponding segments of the at least one image stream;

choreographing representations of objects within the virtual stage; and

providing a choreography specification for generation of a two-dimensional image stream of the virtual stage with the choreographed objects therein.

2. A method as claimed in claim 1 further comprising generating a two-dimensional image stream of the choreographed objects as viewed from a virtual camera specified within the virtual stage.

3. A method as claimed in claim 1 further comprising generating a two-dimensional image stream, details of the generated image stream being derived directly from the image stream captured from the physical objects.

4. A method as claimed in claim 1 wherein the choreographed representations of objects include three-dimensional representations of physical objects in the physical object space and three-dimensional representations of synthetic objects.

5. A method as claimed in claim 1 further comprising providing representations of camera and lighting, choreographing positions and orientations of camera and lighting, and generating the two-dimensional image stream as a function of the choreographed camera and lighting.

6. A method as claimed in claim 1 further comprising including abstract models as proxies of physical objects and, in generating the two-dimensional image stream, obtaining detail for those physical objects from the captured image stream.

7. A method as claimed in claim 1 further comprising, after choreographing and generating a two-dimensional image stream, capturing another image stream from physical objects in the virtual stage for use in generating a two-dimensional image stream from a choreography specification.

8. A method as claimed in claim 1 further comprising choreographing synthetic objects with physical objects in the virtual stage and, subsequent to the choreography and generation of a two-dimensional image stream, modifying the representations of synthetic objects.

9. A method as claimed in claim 1 further comprising displaying a three-dimensional representation of physical and synthetic objects within the virtual stage and displaying a preview of the two-dimensional image stream.

10. A method as claimed in claim 9 further comprising displaying the three-dimensional representation as viewed from a vantage point other than a virtual camera location for which the preview is displayed.

11. A method as claimed in claim 9 further comprising displaying temporal representations of the choreography specification.

12. A method as claimed in claim 11 further comprising displaying an object catalog of physical and synthetic objects within the virtual stage.

13. A method as claimed in claim 1 further comprising displaying temporal representations of the choreography specification.

14. A method as claimed in claim 1 further comprising associating audio tracks with at least one object in the virtual stage.

15. A method as claimed in claim 14 further comprising modifying the audio tracks as the step of choreographing representations of objects changes acoustic properties of the virtual stage.

16. A method as claimed in claim 1 wherein the representations of physical objects and representations of synthetic objects are object oriented models.

17. A method as claimed in claim 1 wherein the representations of objects include two-dimensional representations.

18. A method as claimed in claim 1 further comprising defining a three-dimensional path within the virtual stage that represents the motion associated with at least one feature of an object represented in the virtual stage.

19. A method as claimed in claim 1 further comprising generating in the choreography specification inter-object effects including shadows and reflections between plural objects represented in the virtual stage.

20. A method as in claim 1 additionally comprising the step of:

accessing segments of the at least one image stream by reference to at least one object representation in the virtual stage through a corresponding link in the correlation mesh.

21. A method as in claim 1 additionally comprising the step of:

accessing at least one object representation in the virtual stage by reference to at least one image stream through a corresponding link in the correlation mesh.

22. A method as in claim 1 additionally comprising the step of:

presenting an interactive display of intermediate results of storing the correlation mesh data structure whereby a user specifies which one of multiple representations of object in the virtual stage are linked to a corresponding segment of the at least one image stream.

23. A method of media production comprising, in a data processing system:

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least one physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) storing in a correlation mesh data structure representations of links between objects in the virtual stage, at least one such correlation link correlating at least one of the pixel representations of the physical object with at least one corresponding segment of the at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream.

24. A method as claimed in claim 23 further comprising providing abstract models of multiple physical objects.

25. A method as claimed in claim 23 further comprising providing abstract models of synthetic objects.

26. A method as claimed in claim 23 further comprising correlating audio tracks with a three-dimensional path.

27. A method as claimed in claim 23 further comprising displaying a representation of the three-dimensional virtual stage, a representation of the choreography specification, and a catalog of image streams and abstract models of objects.

28. A method as claimed in claim 23 further comprising generating two-dimensional images from a combination of the choreography specification, the abstract model and the two-dimensional pixel representation.

29. A method as claimed in claim 23 further comprising providing a preview window, a two-dimensional rendered display of the three-dimensional virtual stage from the viewpoint of a specified virtual camera based on a combination of the choreography specification, the abstract model and the pixel representation.

30. A method as claimed in claim 23 further comprising providing rendered results in a two-dimensional preview window at different levels of quality and resolution based on user control.

31. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process the creation of a two-dimensional geometric path that represents the motion in image-space associated with at least one specified feature of pixel representations correlated across successive frames in an image stream.

32. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process the parameters for an automated procedure that projects a two-dimensional path derived from analysis of movements in image-space into the related three-dimensional path within the three-dimensional virtual stage.

33. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process the creation of a three-dimensional geometric path that represents the motion associated with at least one feature of an abstract model in the three-dimensional virtual stage.

34. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process the assignment of a three-dimensional path to an abstract model in the three-dimensional virtual stage, such that the movement of one abstract model tracks the movement of another abstract model.

35. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process the parameters for the alteration of surface characteristics of an abstract object.

36. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process parameters for an automated process that generates inter-object effects including shadows and reflections between multiple abstract models represented in the three-dimensional virtual stage.

37. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process parameters for an automated process that generates atmospheric effects that alter the generation of two-dimensional images from the abstract model represented in the three-dimensional virtual stage.

38. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process parameters for an automated process that alters geometric characteristics of an abstract model in the three-dimensional virtual stage over a specified sequence of time.

39. A method as claimed in claim 23 further comprising specifying and controlling during the choreography process parameters for an automated inter-object process that alters geometric characteristics of an abstract model, in the three-dimensional virtual stage across a specified sequence of time such that the final result matches specified geometric characteristics of a second abstract model and that the intermediate results appear to be a gradual transformation from the first abstract model to the second abstract model.

40. A method as claimed in claim 23 further comprising providing user access to the choreography specification, correlation database, abstract model, pixel representation and image stream.

41. A method of media production comprising, in a data processing system:

a) providing a three-dimensional virtual stage,

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

providing an abstract model of the camera'"'"'s position.

42. A method of media production comprising, in a data processing system:

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

providing an abstract model of the lighting used to illuminate the physical objects.

43. A method of media production comprising, in a data processing system;

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

providing an abstract model of acoustics.

44. A method of media production comprising, in a data processing system:

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

providing the choreography specification in a human-readable form for use, review and modification.

45. A method of media production comprising, in a data processing system;

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

creating and maintaining multiple choreography specifications that share a correlation database, abstract model, pixel representation and image stream.

46. A method of media production comprising, in a data processing system:

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

creating and maintaining multiple versions of image stream and pixel representations and allowing determination of which version to use for generating an interactive display of intermediate results from choreography manipulations.

47. A method of media production comprising, in a data processing system:

a) providing a three-dimensional virtual stage;

b) providing at least one image stream captured from one or more physical objects;

c) providing at least one pixel representation segmented from the at least one image stream that at least partially represents at least only physical object captured in the at least one image stream;

d) providing at least one abstract model of at least one of the physical objects in the at least one image stream;

e) correlating at least one of the pixel representations of the physical object with at least one image stream and with at least one corresponding abstract model of the same physical object;

f) in a choreography process, iteratively specifying and controlling how the abstract model of the physical object is manipulated within the virtual stage; and

g) generating a choreography specification that captures and allows a reproduction of the specified manipulations of at least one abstract model in the virtual stage, allowing the generation of a sequence of two-dimensional images of the virtual stage from a combination of the choreography specification, the at least one abstract model of the physical object, and the at least one of the pixel representations of the physical object in the image stream, further comprising the step of;

creating and maintaining multiple versions of abstract models and allowing determination of which version to use for generating an interactive display of intermediate results from choreography manipulations.