METHODS AND APPARATUS RELATED TO CAPTURING AND/OR RENDERING IMAGES

• US 20160373734A1
• Filed: 09/01/2016
• Published: 12/22/2016
• Est. Priority Date: 02/24/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay, Medina, Hector M.
• Granted Patent: US 9,894,350 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Camera and/or lens calibration information is generated as part of a calibration process in video systems including 3-dimensional (3D) immersive content systems. The calibration information can be used to correct for distortions associated with the source camera and/or lens. A calibration profile can include information sufficient to allow the system to correct for camera and/or lens distortion/variation. This can be accomplished by capturing a calibration image of a physical 3D object corresponding to the simulated 3D environment, and creating the calibration profile by processing the calibration image. The calibration profile can then be used to project the source content directly into the 3D viewing space while also accounting for distortion/variation, and without first translating into an intermediate space (e.g., a rectilinear space) to account for lens distortion.

Claims

1. A camera calibration system including:

a target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations;

a camera calibration apparatus including;

a camera interface for receiving an image of the surface of said target apparatus from a first camera being calibrated; and

a camera calibrator configured to compare actual locations of the markers in said image to expected locations of said markers and to generate a set of camera calibration information based on differences between the actual locations of the markers in said image and the expected locations of said markers.

2. The camera calibration system of claim 1, wherein said target apparatus includes four flat sides and a back, each of said flat sides and back including markers.

4. The camera calibration system of claim 1, wherein said target apparatus is an open box, the camera calibration system further comprising:

a camera mounting platform; and

rails for support said camera mounting platform.

5. The camera calibration system of claim 4, further comprising:

a position drive motor;

a drive shaft coupled to said camera mounting platform and to said position drive motor; and

a camera position controller for controlling the position drive motor based on one or more images captured by the first camera while positioned on said camera mounting platform to position the first camera at a predetermined location in front of said target apparatus.

6. The camera calibration system of claim 1, wherein said predetermined location is a location positioned midway between a first side and a second side of said target apparatus.

7. The camera calibration system of claim 6, wherein said target apparatus is square, has four sides of the same length and wherein said predetermined location is at the center of said square, said back of said target apparatus including a center portion marker which can to facilitate camera alignment with said center portion marker.

8. The camera calibration apparatus of claim 7, wherein each of the sides of said target apparatus are at least two feet long.

9. The camera calibration apparatus of claim 8, wherein the sides and back of said target apparatus are made of a rigid sheet material and arranged perpendicular to each other.

10. The camera calibration system of claim 1, wherein each of the markers is part of a test pattern, each marker being an intersection of points in a grid formed by a pattern of squares.

11. The camera calibration system of claim 1, wherein the camera calibrator generates first UV map correction information indicating one or more adjustments to be made to the location of nodes in a UV map based on the differences between the actual locations of the markers in said first image and the expected locations of said markers.

12. The camera calibration system of claim 1, wherein the expected location of the markers is stored in information in memory and known based on properties of at least a lens included in the first camera.

13. The camera calibration system of claim 11, wherein the first UV map correction information includes information indicating a U offset and a V offset to be applied to a first node in the UV map, said UV map being a map to be used by a playback device to map an image captured by said first camera to a surface of a mesh model.

14. The camera calibration system of claim 13, wherein the calibration apparatus further includes:

a network interface for communicating the UV map, the first UV correction information, and the mesh model to a playback device for use in rendering images captured by the first camera.

15. A method comprising:

operating a first camera to capture an image of an interior surface of a target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations;

operating a camera interface to receive the image of the interior surface of said target apparatus from the first camera being calibrated; and

operating a camera calibrator configured to compare actual locations of the markers in said image to expected locations of said markers and to generate a first set of camera calibration information corresponding to the first camera based on differences between the actual locations of the markers in said image and the expected locations of said markers.

16. The method of claim 15, further comprising:

prior to performing said step of operating the first camera to capture the image of the interior surface;

receiving, at a camera interface, images from the first camera while the first camera is mounted on a camera mounting platform positioned in front of the target apparatus;

processing at least one of the received images to detect a position of the first camera to thereby determine the position of the first camera relative to the target apparatus; and

operating a camera position controller to adjust the position of the camera mounting platform to position the first camera at a predetermined location in front of said target apparatus.

17. The method of claim 16, wherein operating the camera position controller includes:

operating the camera position controller to control a position drive motor based on one or more images captured by the first camera to move a drive shaft coupled to the camera mounting platform and to thereby adjust the position of the first camera until the first camera is positioned at a predetermined location in front of said target apparatus.

18. The method of claim 17, wherein said first set of camera calibration information includes first UV map correction information indicating one or more adjustments to be made to the location of nodes in a UV map based on the differences between the actual locations of the markers in said first image and the expected locations of said markers.

19. The method of claim 18, wherein generating said first set of camera calibration information (1536) includes generating for a first node in the UV map a U offset and a V offset to be applied to the position of the first node to adjust the location of the first node to compensate for a camera introduced distortion, said UV map being a map to be used by a playback device to map an image captured by said first camera to a surface of a mesh model.

20. A non-transitory computer storage comprising instructions which, when executed, direct a computer system comprising one or more computing devices to perform the steps of:

operating a first camera to capture an image of an interior surface of a target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations;

operating a camera interface to receive the image of the interior surface of said target apparatus from the first camera being calibrated; and

operating a camera calibrator configured to compare actual locations of the markers in said image to expected locations of said markers and to generate a first set of camera calibration information corresponding to the first camera based on differences between the actual locations of the markers in said image and the expected locations of said markers.