SYSTEM FOR MIXING OR COMPOSITING IN REAL-TIME, COMPUTER GENERATED 3D OBJECTS AND A VIDEO FEED FROM A FILM CAMERA

• US 20160005234A1
• Filed: 08/19/2015
• Published: 01/07/2016
• Est. Priority Date: 05/09/2012
• Status: Active Grant

• Current Assignee: Ncam Technologies Limited (Carl-Zeiss-Stiftung)
• Original Assignee: Ncam Technologies Limited (Carl-Zeiss-Stiftung)
• Inventors: BOIVIN, Samuel, MICHOUD, Brice
• Granted Patent: US 11,182,960 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method of mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera in which the body of the film camera can be moved in 3D and sensors in or attached to the camera provide real-time positioning data defining the 3D position and 3D orientation of the camera, or enabling the 3D position to be calculated.

Claims

1. A markerless system for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera, such as a video camera, to generate real-time augmented reality video for TV broadcast, cinema or video games, in which:

(d) the body of the film camera can be moved in 3D;

(b) sensors, including multiple non-optical sensors, selected from the group of accelerometer, gyroscope, or magnetometer, are in or attached directly or indirectly to the film camera and provide real-time positioning data defining the 3D position and 3D orientation of the film camera, or enable the 3D position and 3D orientation of the film camera to be calculated;

(c) a pair of stereoscopic witness cameras are fixed directly or indirectly to the film camera;

(d) that real-time positioning data is then automatically used by the system to create, recall, render or modify computer generated 3D objects;

(e) the resulting computer generated 3D objects are then mixed in or composited with the video feed from the film camera to provide augmented reality video for TV broadcast, cinema or video games;

and in which;

(f) the 3D position and orientation of the film camera is determined with reference to a 3D map of the real-world generated whilst the camera is being used to capture video, by using the real-time 3D positioning data from the non-optical sensors plus a video flow from each stereo witness camera that enables the scene to be reconstructed in 3D using stereoscopic algorithm(s) running on a processor operating on natural markers in the scene (‘

feature points’

) that have not been manually or artificially added to that scene, so that the system is capable of tracking a scene without the requirement for an initial or prior survey of the scene prior to filming with the film camera.

2. The system claim 1, in which the non-optical sensors include an accelerometer, and a gyro (6 DOF sensor).

3. The system of claim 1, in which the non-optical sensors include a 3-axis accelerometer measuring transnational acceleration in 3D, a 3-axis gyro measuring angular velocity in 3D, and magnetometer measuring absolute heading in 3D, and hence constituting a 9 DOF sensor.

4. The system of claim 1 in which the sensors include a 3D range sensor, such as structured light or time-of-flight camera that captures the depth of each pixel in a video output from the camera.

5. The system of claim 4 in which the depth of edges is refined by re-projecting the 3D range sensor depths onto the high resolution video feed of the film camera.

6. The system of claim 1 which utilizes two high-speed (such as at least 100 fps) witness cameras to enable the system to be fully initialized without a separate stage of purely surveying the scene to be tracked (called ‘

instant survey’

), but instead surveying takes place continuously whilst the camera is being used to capture video.

7. The system of claim 6 in which the two high speed witness cameras form the stereoscopic system that enables software to process the images and, even with the camera system not being moved at all, to generate an instant 3D point cloud (for example associating a large number of points in the scene to their position in a 3D space using knowledge of the separation between the two camera and epipolar geometry).

8. The system of claim 6 in which the depth of each pixel in the 3D point cloud is obtained using corresponding 2D texture patches obtained from each stereoscopic witness camera and an epi-polar line search algorithm.

9. The system of claim 1 which runs a fusion algorithm that combines optical flow data from the witness camera system with the real-time positioning data from the hardware sensors.

10. The system of claim 9 in which the fusion algorithm is based on an Extended Kalman Filter prediction/correction technique to integrate outputs from, and to re-calibrate, all the sensors, selected from a group including an accelerometer, a gyroscope, a magnetometer, a 3D range sensor, to determine the position and orientation of the camera.

11. The system of claim 10 in which the Extended Kalman Filter fusion algorithm uses confidence level data, associated with the output from each sensor, when determining how to merge the data from each sensor.

12. The system of claim 1 in which keyframes generated by the witness cameras are pan of the visual tracking process and are real-time images computed at 4 different resolution levels of the witness cameras'"'"' video feed.

13. The system of claim 1 in which a 3D range sensor is used to enhance the accuracy of a depth measurement associated with a reconstructed 3D point, or to reject that reconstructed 3D point.

14. The system of claim 13 in which the 3D range sensor is used for real-time depth keying to enable dynamic occlusion and suppress the eventual use of a green stage.

15. The system of claim 1 which includes a magnetometer to indicate magnetic North, an accelerometer to indicate gravity direction (and thus giving true vertical), a gyro to indicate if the system is tilted up/down or has panned left or right or has been rotated about the optical axis, and a 3-axis accelerometer to enable translation in 3D from a start position to be inferred.

16. The system of claim 1 in which software attempts to generate a uniformly distributed point cloud in the 3D map to greatly reduce tracking losses and increase tracking accuracy, in which more parallax is generated, so the estimated camera position is more accurate.

17. The system of claim 1 providing a camera tracking system that combines instant survey with tracking of the film camera as the director/cameraman tracks, pans, tilts the camera tracking system attached to the film camera.

18. The system of claim 1 that enables real-time, continuous surveying of a scene to generate a more complete point cloud defining the scene.

19. The system of claim 1 that attaches rotation invariant descriptors, for example using ORB (Oriented FAST and Rotated BRIEF), to feature points detected in the scene to facilitate recovery of tracking.

20. The system of claim 1 that uses a constant velocity model associated with the in formation provided by the sensors to predict the next position of the film camera using the previously correctly computed or confirmed position.

21. The system of claim 20 that uses that prediction to re-project a 3D point cloud onto the current frame, to enable a point matching algorithm to match points identified in the real-time video feed from the witness camera system and the projected points in the new, 3D point cloud.

22. The system of claim 21 that uses a Levenberg-Marquardt minimization scheme for camera tracking to minimize the error between the points identified in the real-time video feed from the witness camera system and the projected points in the new, 3D point cloud.

23. The system of claim 1 in which a user can use a 3D point cloud generated by the camera tracking system to define 3D masks, such as 3D Garbage Matting masks.

24. The system of claim 1 in which the film camera and a witness camera are calibrated for frame acquisition delay using a modulated light source, such as by comparing light intensity curves associated with a flashing LED.

25. The system of claim 1 in which the film camera is any of the following:

crane camera;

steady cam;

hand-held-camera;

dolly mounted camera, tripod-mounted camera, smartphone, augmented reality glasses.

26. A markerless method for mixing or compositing real-time, computer generated 3D objects and a video feed from a film camera, such as a video camera, to generate augmented reality video for TV broadcast, cinema or video games, in which:

(a) the body of the film camera can be moved in 3D;

(b) sensors, including multiple non-optical sensors, selected from the group of accelerometer, gyroscope, or magnetometer, are in or attached directly or indirectly to the film camera and provide real-time positioning data defining the 3D position and 3D orientation of the film camera, or enable the 3D position and 3D orientation of the film camera to be calculated;

(c) a pair of stereoscopic witness cameras are fixed directly or indirectly to the film camera;

(d) that real-time positioning data is then automatically used by the system to create, recall, render or modify computer generated 3D objects;

(e) the resulting computer generated 3D objects are then mixed in or composited with the video feed from the film camera to provide augmented reality video for TV broadcast, cinema or video games;

and in which;

(f) the 3D position and orientation of the film camera is determined with reference to a 3D map of the real-world generated whilst the camera is being used to capture video, by using the real-time 3D positioning data from the non-optical sensors plus a video flow from each stereo witness camera that enables the scene to be reconstructed in 3D using stereoscopic algorithm(s) running on a processor operating on natural markers in the scene (‘

feature points’

) that have not been manually or artificially added to that scene, so that the system is capable of tracking a scene without the requirement for an initial or prior survey of the scene prior to filming with the film camera.

27. The method as defined in claim 26 in which the real-time positioning data is recorded and time-code stamped to provide match-move data for post-production processes.

28. The method as defined in claim 26 above to enable one or more of the following:

Real-time tracking for studio cameras

Real-time tracking for Steadicam

Real-time tracking for Crane-mounted cameras

Real-time tracking for Dolly-mounted cameras

Real-time tracking for Steadicam

Real time tracing for Outside Broadcast (OB)

Real time tracking implemented on a smartphone

Real time tracking implemented on wearable computing device, such as augmented reality glasses

Using real-tame data (e.g. tracking data) for 2D post-production

Using real-time data (e.g. tracking data) for post-conversion for 3D stereoscopic content

Using real-time data (e.g. tracking data) for native 3D stereoscopic content

3D graphics insertion

3D graphics insertion for in-studio or on-set product placement

3D graphics insertion for OB

3D graphics insertion for other sponsored images

3D graphics insertion that is viewer-location specific

3D graphics insertion that is viewer-specific

3D graphics insertion that is time-specific

3D graphics insertion for filling our crowd scenes

3D graphics insertion for green screen replacement

3D graphics insertion of educational content to assist learning, in museums and interpretation centres in cultural, historic or natural sites.

Measurement of the absolute or relative size of objects in the scene.

29. The method of claim 26 in which the computer generated 3D objects are automatically scaled to the correct size using the stereoscopic algorithm(s).

30. Augmented reality video film computer-readable file embodied on a non-transitory storage medium, the augmented reality video film computer-readable file being produced using a markerless method for mixing or compositing real-time, computer generated 3D objects and a video feed from a film camera, such as a video camera, to generate the augmented reality video film computer-readable rile suitable for TV broadcast, cinema or video games, in which:

(a) the body of the film camera can be moved in 3D and sensors, including multiple non-optical sensors, selected from the group of accelerometer, gyroscope or magnetometer, in which the sensors are in or attached directly or indirectly to the film camera and provide real-time positioning data defining the 3D position and 3D orientation of the film camera, or enable the 3D position and 3D orientation of the film camera to be calculated;

(b) a pair of stereoscopic witness cameras are fixed directly or indirectly to the film camera;

(c) that real-time positioning data is then automatically used by the system to create, recall, render or modify computer generated 3D objects;

(d) the resulting computer generated 3D objects are then mixed in or composited with the video feed from the film camera to provide augmented reality video for TV broadcast, cinema or video games;

and in which;

(e) the 3D position and orientation of the film camera is determined with reference to a 3D map of the real-world generated whilst the camera is being used to capture video by using the real-time 3D positioning data from the non-optical sensors plus a video flow from each stereo witness camera that enables the scene to be reconstructed in 3D using stereoscopic algorithms running on a processor operating on natural markers in the scene (feature points) that have not been manually or artificially added to that scene, so that the system is capable of tracking a scene without the requirement for an initial or prior survey of the scene prior to filming with the film camera.