Hybrid precision tracking

• US 9,171,379 B2
• Filed: 04/10/2013
• Issued: 10/27/2015
• Est. Priority Date: 04/13/2012
• Status: Active Grant

• Current Assignee: Lightcraft Technology, LLC
• Original Assignee: Lightcraft Technology, LLC
• Inventors: Mack, Newton Eliot, Mass, Philip R.
• Primary Examiner(s): Chu, Randolph I
• Application Number:
  US14/370,948
• Publication Number:
  US 20150016680A1
• Time in Patent Office: 930 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 11/04 Interpretation of pictures
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/2008 Assembling, disassembling
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

Disclosed herein are through-the-lens tracking systems and methods which can enable sub-pixel accurate camera tracking suitable for real-time set extensions. That is, the through-the-lens tracking can make an existing lower precision camera tracking and compositing system into a real-time VFX system capable of sub-pixel accurate real-time camera tracking. With this enhanced level of tracking accuracy the virtual cameras can be used to register and render real-time set extensions for both interior and exterior locations.

Claims

1. A hybrid through-the-lens tracking method, comprising:

reading current incoming camera position, orientation, and adjustable field-of-view/zooming and focusing lens information of an actual scene camera, wherein the lens information includes lens position information;

deriving current lens parameters of the actual scene camera using the lens position information;

predicting the 2D locations of actual markers in a live action scene using (a) the current lens parameters, (b) the current incoming camera position and orientation information and (c) the lens position information;

detecting the 2D locations of the actual markers in the live action scene;

the detecting including derivative detecting to find sharp edges of the actual markers, center detecting to find possible centers of circles, and center voting to detect actual centers of the circles;

calculating angular corrections for a virtual camera from the current incoming camera position and orientation information, the predicted 2D locations, and the detected 2D locations; and

using the angular corrections, correcting the orientation of the current incoming camera orientation information of the actual scene camera to match the 2D locations of the actual markers.

2. The method of claim 1 wherein the detecting further includes:

before the derivative detecting, deblurring the live action image of the actual markers to get a sharp circle; and

before the center detecting, removing parallel lines from the edge detection results to remove things that are not circles.

3. The method of claim 2 further comprising using the detected actual centers, the incoming camera position and orientation information, and the predicted 2D positions to calculate the angular correction needed to make the predicted 2D marker positions match the actual 2D marker positions.

4. The method of claim 1 further comprising using the detected actual centers to calculate the angular correction needed to make the predicted 2D positions using the measured camera angles match the actual marker positions in the live action scenes, by using the incoming camera position as the base point of the angle.

5. The method of claim 1 wherein the deriving includes using a look-up table that correlates lens position information with optical parameters of the lens of the actual scene camera.

6. The method of claim 1 wherein the lens position information is lens barrel rotation measurement information of the actual scene camera.

7. The method of claim 1 wherein the current incoming camera position, orientation, and adjustable field-of-view/zooming and focusing lens information is real time information, and the lens position information is on a per frame basis.

8. The method of claim 1 further comprising 3D rendering a 2D virtual image produced by the virtual camera to produce a 2D composited image.

9. The method of claim 8 wherein the 3D rendering is after the correcting the orientation.

10. The method of claim 1 further comprising after the correcting, filtering the angular corrections to reduce motion noise and improve tracking accuracy.

11. The method of claim 10 further comprising after the filtering, correcting the pose of the virtual camera using the filtered angular corrections.

12. The method of claim 1 wherein the angular corrections are for the pan, tilt and roll angles of the virtual camera.

13. The method of claim 1 wherein the predicting uses 3D locations of the actual markers in the live action scene.

14. The method of claim 1 wherein the focusing lens information includes lens distortion information, and further comprising rendering the virtual background with correct distortion to thereby match the live action image of the actual markers using the lens distortion information.

15. The method of claim 1 wherein the focusing lens information includes the current measured camera position and orientation, and the current lens parameters include focal length, distortion, and entry pupil location.

16. The method of claim 1 wherein the actual markers are circular markers.

17. The method of claim 1 wherein the actual markers are fiducial markers, bar code markers or natural feature markers.

18. A hybrid through-the-lens tracking method, comprising:

reading current incoming camera position, orientation, and adjustable field-of-view/zooming and focusing lens information of an actual scene camera, wherein the lens information includes lens position information;

the current incoming camera position, orientation, and adjustable field-of-view/zooming and focusing lens information being real time information;

the lens position information being on a per frame basis;

deriving current lens parameters of the actual scene camera using the lens position information;

the deriving including using a look-up table that correlates lens position information with optical parameters of the lens of the actual scene camera;

predicting the 2D locations of actual markers in a live action scene using (a) the current lens parameters, (b) the current incoming camera position and orientation information and (c) the lens position information;

detecting the 2D locations of the actual markers in the live action scene;

calculating angular corrections for a virtual camera from the current incoming camera position and orientation information, the predicted 2D locations and the detected 2D locations; and

using the angular corrections, correcting the orientation of the current incoming camera orientation information of the actual scene camera to match the 2D locations of the actual markers.

19. The method of claim 18 wherein the lens position information is lens barrel rotation measurement information of the actual scene camera.

20. The method of claim 18 further comprising 3D rendering a 2D virtual image produced by the virtual camera to produce a 2D composited image.

21. The method of claim 20 wherein the 3D rendering is after the correcting the orientation.

22. The method of claim 18 further comprising after the correcting, filtering the angular corrections to reduce motion noise and improve tracking accuracy.

23. The method of claim 22 further comprising after the filtering, correcting the pose of the virtual camera using the filtered angular corrections.

24. The method of claim 18 wherein the detecting includes:

deblurring the live action image of the actual markers to get a sharp circle;

derivative detecting to find sharp edges of the actual markers;

removing parallel lines from the edge detection results to remove things that are not circles;

center detecting to find possible centers of circles; and

center voting to detect actual centers of the circles.

25. The method of claim 24 further comprising using the detected actual centers to calculate the angular correction needed to make the predicted 2D positions using the measured camera angles match the actual marker positions in the live action scenes.

26. The method of claim 18 wherein the angular corrections are for the pan, tilt and roll angles of the virtual camera.

27. The method of claim 18 wherein the predicting uses 3D locations of the actual markers in the live action scene.

28. The method of claim 18 wherein the focusing lens information includes lens distortion information, and further comprising rendering the virtual background with correct distortion to thereby match the live action image of the actual markers using the lens distortion information.

29. The method of claim 18 wherein the focusing lens information includes the current measured camera position and orientation, and the current lens parameters include focal length, distortion, and entry pupil location.

30. The method of claim 18 wherein the actual markers are circular markers.

31. The method of claim 18 wherein the actual markers are fiducial markers, bar code markers or natural feature markers.