System and process for geometry replacement

• US 20030076321A1
• Filed: 10/23/2001
• Published: 04/24/2003
• Est. Priority Date: 10/23/2001
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Clavadetscher, Charles
• Granted Patent: US 6,724,386 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

A system and process for replacing unwanted geometry (for example, tracking markers) in a recorded scene with new geometry. A 3-D matchmove process creates a 3-D map. The 3-D map is used to determine the 3-D coordinates of objects within the recorded scene. The 3-D coordinates of the tracking markers (and/or other objects) and the replacement portions are converted by a conversion means to 2D screen space coordinates of pixels that make up both the tracking markers (and/or other objects) and the replacement portions. A list comprising the 2D screen space pixel coordinates is then exported from the conversion means into a compositing application which reads the list of 2D pixel coordinates and makes the desired replacement automatically from frame-to-frame. Tracking markers (and/or other unwanted objects) are replaced with replacement portions with suitable characteristics.

Claims

1. A process for altering images in frames of a recorded scene, the recorded scene comprising a plurality of frames, the process comprising:

converting at least one 2-D point or feature within a first frame to a set of 3-D coordinates;

designating at least one set of 3-D coordinates within the first frame as a target point;

logging 3-D variations in the recorded scene from the first frame to a second frame;

outputting a plurality of sets of 3-D coordinates corresponding to the 3-D variations in the recorded scene;

converting the outputted plurality of sets of 3-D coordinates into a first plurality of 2-D points or features corresponding to 2-D variations in the target point from the first frame to the second frame; and

inputting the first plurality of 2-D points or features into a compositing application;

wherein the compositing application automatically locates and alters the target point in the first and second frames using the first plurality of 2-D points or features.

2. The process recited in claim 1, wherein designating at least one set of 3-D coordinates within the first frame as a target point comprises manually designating the at least one set of 3-D coordinates with an input device.

3. The process recited in claim 1, wherein logging 3-D variations in the recorded scene from the first frame to a second frame comprises generating a list of the 3-D variations.

4. The process recited in claim 1, wherein outputting a first plurality of sets of 3-D coordinates comprises outputting the plurality of sets of 3-D coordinates to a conversion means.

5. The process recited in claim 1, wherein converting the outputted plurality of sets of 3-D coordinates into a first plurality of 2-D points or features comprises using conversion algorithms to convert the plurality of outputted sets of 3-D coordinates into a first plurality of 2-D points or features.

6. The process recited in claim 5, further comprising generating a list of the first plurality of 2-D points or features.

7. The process recited in claim 1, further comprising:

designating at least one set of 3-D coordinates within the first frame as a source point;

converting the outputted plurality of sets of 3-D coordinates into a second plurality of 2-D points or features corresponding to 2-D variations in the source point from the first frame to the second frame; and

inputting the second plurality of 2-D points or features into a compositing application;

wherein the compositing application automatically locates and copies the source point in the first and second frames to the target point in the first and second frames using the first and second plurality of 2-D points or features.

8. The process recited in claim 7, wherein designating at least one set of 3-D coordinates within the first frame as a source point comprises manually designating the at least one set of 3-D coordinates with an input device.

9. The process recited in claim 7, wherein converting the outputted plurality of sets of 3-D coordinates into a second plurality of 2-D points or features comprises using conversion algorithms to convert the plurality of outputted sets of 3-D coordinates into a second plurality of 2-D points or features.

10. The process recited in claim 9, further comprising generating a list of the second plurality of 2-D points or features.

11. The process recited in claim 7, wherein designating at least one set of 3-D coordinates within the first frame as a source point comprises designating two or more sets of 3-D coordinates within the first frame as source points.

12. A system for altering images in frames of a recorded scene, the recorded scene comprising a plurality of frames, the system comprising:

a processing device suitable for performing media production editing, compositing and mixing of the images in the recorded scene;

a display screen connected with the processing device for displaying the recorded scene;

a 3-D graphics software application executable by the processing device and suitable for generating a 3-D map of the recorded scene;

a conversion means for converting 3-D coordinates within the 3-D map to 2-D pixel coordinates on the display screen; and

a compositing software application executable by the processing device and suitable for locating and altering a target point in a frame of the recorded scene;

wherein the conversion means outputs the 2-D pixel coordinates to the compositing software and wherein the compositing software alters the target point using the 2-D pixel coordinates.

13. The system recited in claim 12, wherein the conversion means comprises a software plug-in associated with a 3-D animation software application.

14. The system recited in claim 12, wherein the conversion means comprises a stand-alone software application.

15. The system recited in claim 12, wherein the conversion means comprises at least one of firmware and hardware comprising hardwired logic.

16. The system recited in claim 12, wherein the processing device comprises a computer graphics workstation.

17. A process for making a recording of a scene, comprising:

designating a point or feature to be tracked within a scene;

recording the scene;

importing the recorded scene into a 3-D graphics application;

tracking the designated point or feature within the scene in 2-D screen space;

converting at least one 2-D point or feature within a first frame of the recorded scene to a set of 3-D coordinates;

designating at least one set of 3-D coordinates within the first frame as a target point;

logging 3-D variations in the recorded scene from the first frame to a second frame;

outputting a plurality of sets of 3-D coordinates corresponding to the 3-D variations in the recorded scene;

converting the outputted plurality of sets of 3-D coordinates into a first plurality of 2-D points or features corresponding to 2-D variations in the target point from the first frame to the second frame; and

inputting the first plurality of 2-D points or features into a compositing application;

wherein the compositing application automatically locates and alters the target point in the first and second frames using the first plurality of 2-D points or features.

18. The process recited in claim 17, further comprising:

designating at least one set of 3-D coordinates within the first frame as a source point;

converting the outputted plurality of sets of 3-D coordinates into a second plurality of 2-D points or features corresponding to 2-D variations in the source point from the first frame to the second frame; and

inputting the second plurality of 2-D points or features into a compositing application;

wherein the compositing application automatically locates and copies the source point in the first and second frames to the target point in the first and second frames using the first and second plurality of 2-D points or features.

19. The process recited in claim 18, wherein designating at least one set of 3-D coordinates within the first frame as a source point comprises designating two or more sets of 3-D coordinates within the first frame as source points.

20. The process recited in claim 17, wherein importing the recorded scene into a 3-D graphics application comprises scanning the recorded scene into the 3-D graphics application.