SYSTEM FOR MIXING OR COMPOSITING IN REAL-TIME, COMPUTER GENERATED 3D OBJECTS AND A VIDEO FEED FROM A FILM CAMERA

• US 20150084951A1
• Filed: 05/09/2013
• Published: 03/26/2015
• Est. Priority Date: 05/09/2012
• Status: Active Grant

• Current Assignee: Ncam Technologies Limited (Carl-Zeiss-Stiftung)
• Original Assignee: Ncam Technologies Limited (Carl-Zeiss-Stiftung)
• Inventors: Boivin, Samuel, Michoud, Brice
• Granted Patent: US 9,600,936 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method of mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera in which the body of the film camera can be moved in 3D and sensors in or attached to the camera provide real-time positioning data defining the 3D position and 3D orientation of the camera, or enabling the 3D position to be calculated.

Claims

1. A markerless system for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera, such as a video camera, to generate real-time augmented reality video for TV broadcast, cinema or video games, in which:

(a) the body of the film camera can be moved in 3D and sensors, including non-optical sensors such as an accelerometer, and/or gyroscope and/or magnetometer in or attached directly or indirectly to the film camera provide real-time positioning data defining the 3D position and 3D orientation of the film camera, or enabling the 3D position and 3D orientation of the film camera to be calculated;

(b) one or more witness cameras are fixed directly or indirectly to the film camera;

(c) that real-time positioning data is then automatically used by the system to create, recall, render or modify computer generated 3D objects;

(d) the resulting computer generated 3D objects are then mixed in or composited with the video feed from the film camera to provide augmented reality video for TV broadcast, cinema or video games;

and in which

(e) the 3D position and orientation of the film camera is determined with reference to a 3D map of the real-world generated, at least in part, by using the real-time 3D positioning data from the non-optical sensors plus an optical flow in which witness camera(s) survey a scene and software running on a processor detects natural markers in the scene (‘

feature points’

) that have not been manually or artificially added to that scene.

6. The system of claim 1 in which the sensors include a 3D range sensor, such as structured light or time-of-flight camera that captures the depth of each pixel in a video output from the camera.

8. The system of claim 6 in which the depth of edges is refined by reprojecting the 3D range sensor depths onto the high resolution video feed of the film camera.

17. The system of claim 1 which utilizes one or two high-speed (such as at least 100 fps) witness cameras to enable the system to be fully initialized without a separate stage of purely surveying the scene to be tracked (called ‘

instant survey’

), but instead surveying takes place continuously whilst the camera is being used to capture video.

18. The system of claim 17 in which the two high speed witness cameras form a stereoscopic system that enables software to process the images and, even with the camera system not being moved at all, to generate an instant 3D point cloud (for example associating a large number of points in the scene to their position in a 3D space using knowledge of the separation between the two camera and epipolar geometry).

19. The system of claim 17 in which the depth of each pixel in the 3D point cloud is obtained using corresponding 2D texture patches obtained from each stereoscopic witness camera and an epi-polar line search algorithm.

20. The system of claim 1 which runs a fusion algorithm that combines optical flow data from the witness camera system with the real-time positioning data from the hardware sensors.

21. The system of claim 20 in which the fusion algorithm is based on an Extended Kalman Filter prediction/correction technique to integrate outputs from, and to re-calibrate, all the sensors, which may include an accelerometer, a gyroscope, a magnetometer, a 3D range sensor, to determine the position and orientation of the camera.

22. The system of claim 21 in which the Extended Kalman Filter fusion algorithm uses confidence level data, associated with the output from each sensor, when determining how to merge the data from each sensor.

23. The system of claim 1 in which keyframes generated by a or the witness camera(s) are part of the visual tracking process and are real-time images computed at 4 different resolution levels of the witness camera video feed.

28. The system of any-preceding claim 1 in which a 3D range sensor is used to enhance the accuracy of a depth measurement associated with a reconstructed 3D point, or to reject that reconstructed 3D point.

29. The system of claim 28 in which the 3D range sensor is used for real-time depth keying to enable dynamic occlusion and suppress the eventual use of a green stage.

33. The system of claim 1 in which software attempts to generate a uniformly distributed point cloud in the 3D map to greatly reduce tracking losses and increase tracking accuracy, in which more parallax is generated, so the estimated camera position is more accurate.

35. The system of claim 1 providing a camera tracking system that combines real-time surveying of a 3D scene (monoscopic case) or instant survey (stereoscopic case) with tracking of the film camera as the director/cameraman tracks, pans, tilts the camera tracking system attached to the film camera.

37. The system of claim 1 that enables real-time, continuous surveying of a scene to generate a more complete point cloud defining the scene.

38. The system of claim 1 that attaches rotation invariant descriptors, for example using ORB, to feature points detected in the scene to facilitate recovery of tracking.

39. The system of claim 1 that uses a constant velocity model associated with the information provided by the sensors to predict the next position of the film camera using the previously correctly computed or confirmed position.

40. The system of claim 39 that uses that prediction to re-project a 3D point cloud onto the current frame, to enable a point matching algorithm to match points identified in the real-time video feed from the witness camera system and the projected points in the new, 3D point cloud.

41. The system of claim 40 that uses a Levenberg-Marquardt minimization scheme for camera tracking to minimize the error between the points identified in the real-time video feed from the witness camera system and the projected points in the new, 3D point cloud.

42. The system of claim 1 in which a user can use a 3D point cloud generated by the camera tracking system to define 3D masks, such as 3D Garbage Matting masks.

44. The system of claim 1 in which the film camera and a witness camera are calibrated for frame acquisition delay using a modulated light source, such as by comparing light intensity curves associated with a flashing LED.

45. The system of claim 1 in which the film camera is any of the following:

crane camera;

steady cam;

hand-held-camera;

dolly mounted camera, tripod-mounted camera, smartphone, augmented reality glasses.

2-5. -5. (canceled)

7. (canceled)

9-16. -16. (canceled)

24-27. -27. (canceled)

30-32. -32. (canceled)

34. (canceled)

36. (canceled)

43. (canceled)

46. (canceled)

47. A markerless method for mixing or compositing real-time, computer generated 3D objects and a video feed from a film camera, such as a video camera, to generate augmented reality video for TV broadcast, cinema or video games, in which:

(a) the body of the film camera can be moved in 3D and sensors, including non-optical sensors such as an accelerometer, and/or gyroscope and/or magnetometer in or attached directly or indirectly to the film camera provide real-time positioning data defining the 3D position and 3D orientation of the film camera, or enabling the 3D position and 3D orientation of the film camera to be calculated;

(b) one or more witness cameras are fixed directly or indirectly to the film camera;

(c) that real-time positioning data is then automatically used by the system to create, recall, render or modify computer generated 3D objects;

(d) the resulting computer generated 3D objects are then mixed in or composited with the video feed from the film camera to provide augmented reality video for TV broadcast, cinema or video games;

and in which

(e) the 3D position and orientation of the film camera is determined with reference to a 3D map of the real-world generated, at last in part, by using the real-time 3D positioning data from the non-optical sensors plus an optical flow in which witness camera(s) survey a scene and software running on a processor detects natural markers in the scene (‘

feature points’

) that have not been manually or artificially added to that scene.

49. The method as defined in claim 47 in which the real-time positioning data is recorded and time-code stamped to provide match-move data for post-production processes.

48. (canceled)

50. (canceled)

52-60. -60. (canceled)