Methods and apparatus related to capturing and/or rendering images

• US 9,894,350 B2
• Filed: 09/01/2016
• Issued: 02/13/2018
• Est. Priority Date: 02/24/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay, Medina, Hector M.
• Primary Examiner(s): Lee, Michael
• Application Number:
  US15/255,135
• Publication Number:
  US 20160373734A1
• Time in Patent Office: 530 Days
• Field of Search: 348187
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Camera and/or lens calibration information is generated as part of a calibration process in video systems including 3-dimensional (3D) immersive content systems. The calibration information can be used to correct for distortions associated with the source camera and/or lens. A calibration profile can include information sufficient to allow the system to correct for camera and/or lens distortion/variation. This can be accomplished by capturing a calibration image of a physical 3D object corresponding to the simulated 3D environment, and creating the calibration profile by processing the calibration image. The calibration profile can then be used to project the source content directly into the 3D viewing space while also accounting for distortion/variation, and without first translating into an intermediate space (e.g., a rectilinear space) to account for lens distortion.

Claims

1. A camera calibration system including:

a target apparatus in the form of a portion of a rectangular box having flat sides which are at right angles to each other, said target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations, wherein said target apparatus includes four flat sides and a back, each of said flat sides and back including markers, adjacent flat sides contacting each other and extending outward perpendicular from the back while remaining in contact with each other to form a closed corner which extends outward from said back; and

a camera calibration apparatus including;

a camera interface for receiving an image of the interior surface of said target apparatus from a first camera being calibrated; and

a camera calibrator configured to compare actual locations of the markers in said image to expected locations of said markers and to generate a set of camera calibration information based on differences between the actual locations of the markers in said image and the expected locations of said markers.

8. The camera calibration system of claim 1, wherein each of the markers is part of a test pattern, each marker being an intersection of points in a grid formed by a pattern of squares.

9. The camera calibration system of claim 1, wherein the camera calibrator generates first UV map correction information indicating one or more adjustments to be made to the location of nodes in a UV map based on the differences between the actual locations of the markers in said image and the expected locations of said markers.

10. The camera calibration system of claim 1, wherein the expected location of the markers is stored in information in memory and known based on properties of at least a lens included in the first camera.

18. The camera calibration system of claim 1, wherein the flat sides and back form an area which is closed on the sides and back.

2. A camera calibration system including:

a target apparatus in the form of a portion of a rectangular box having flat sides which are at right angles to each other, said target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations, wherein said target apparatus includes four flat sides and a back, adjacent flat sides contacting each other and extending outward perpendicular from the back while remaining in contact with each other to form a closed corner which extends outward from said back, and wherein said target apparatus is an open box;

a camera mounting platform;

rails for supporting said camera mounting platform; and

a camera calibration apparatus including;

a camera interface for receiving an image of the interior surface of said target apparatus from a first camera being calibrated; and

a camera calibrator configured to compare actual locations of the markers in said image to expected locations of said markers and to generate a set of camera calibration information based on differences between the actual locations of the markers in said image and the expected locations of said markers.

3. The camera calibration system of claim 2, further comprising:

a position drive motor;

a drive shaft coupled to said camera mounting platform and to said position drive motor; and

a camera position controller for controlling the position drive motor based on one or more images captured by the first camera while positioned on said camera mounting platform to position the first camera at a predetermined location in front of said target apparatus.

4. The camera calibration system of claim 3, wherein said predetermined location is a location positioned midway between a first side and a second side of said target apparatus.

5. The camera calibration system of claim 4, wherein said back of target apparatus is square, has four sides of the same length and wherein one of said predetermined locations is at the center of said square, said back including a center portion marker which facilitates camera alignment with said center portion marker.

6. The camera calibration apparatus of claim 5, wherein each of the sides of said target apparatus are at least two feet long.

7. The camera calibration apparatus of claim 6, wherein the sides and back of said target apparatus are made of a rigid sheet material and arranged perpendicular to each other.

11. A camera calibration system including:

a target apparatus in the form of a portion of a rectangular box having flat sides which are at right angles to each other, said target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations;

a camera calibration apparatus including;

a camera interface for receiving an image of the interior surface of said target apparatus from a first camera being calibrated; and

a camera calibrator configured to compare actual locations of the markers in said image to expected locations of said markers and to generate a set of camera calibration information based on differences between the actual locations of the markers in said image and the expected locations of said markers, wherein the camera calibrator generates first UV map correction information indicating one or more adjustments to be made to the location of nodes in a UV map based on the differences between the actual locations of the markers in said image and the expected locations of said markers, and wherein the first UV map correction information includes information indicating a U offset and a V offset to be applied to a first node in the UV map, said UV map being a map to be used by a playback device to map an image captured by said first camera to a surface of a mesh model.

12. The camera calibration system of claim 11, wherein the calibration apparatus further includes:

a network interface for communicating the UV map, the first UV correction information, and the mesh model to a playback device for use in rendering images captured by the first camera.

13. A method comprising:

receiving, at a camera interface, images from a first camera while the first camera is mounted on a camera mounting platform positioned in front of a target apparatus, wherein said target apparatus is in the form of a portion of a rectangular box having four flat sides which are at right angles to each other and a back, adjacent flat sides contacting each other and extending outward perpendicular from the back while remaining in contact with each other to form a closed corner which extends outward from said back, said target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations;

processing at least one of the received images to detect a position of the first camera to thereby determine the position of the first camera relative to the target apparatus;

operating a camera position controller to adjust the position of the camera mounting platform to position the first camera at a predetermined location in front of said target apparatus;

operating the first camera to capture an image of the interior surface of the target apparatus;

operating the camera interface to receive the image of the interior surface of said target apparatus from the first camera being calibrated; and

operating a camera calibrator configured to compare actual locations of the markers in said image of the interior surface of the target apparatus to expected locations of said markers and to generate a first set of camera calibration information corresponding to the first camera based on differences between the actual locations of the markers in said image of the interior surface of the target apparatus and the expected locations of said markers.

14. The method of claim 13, wherein operating the camera position controller includes:

operating the camera position controller to control a position drive motor based on one or more images captured by the first camera to move a drive shaft coupled to the camera mounting platform and to thereby adjust the position of the first camera until the first camera is positioned at a predetermined location in front of said target apparatus.

15. The method of claim 14, wherein said first set of camera calibration information includes first UV map correction information indicating one or more adjustments to be made to the location of nodes in a UV map based on the differences between the actual locations of the markers in said image of the interior surface of the target apparatus and the expected locations of said markers.

16. The method of claim 15, wherein generating said first set of camera calibration information includes generating for a first node in the UV map a U offset and a V offset to be applied to the position of the first node to adjust the location of the first node to compensate for a camera introduced distortion, said UV map being a map to be used by a playback device to map an image captured by said first camera to a surface of a mesh model.

17. A non-transitory computer storage comprising instructions which, when executed by a processor, direct a system comprising one or more computing devices to perform the steps of:

receiving, at a camera interface, images from a first camera while the first camera is mounted on a camera mounting platform positioned in front of a target apparatus, wherein said target apparatus is in the form of a portion of a rectangular box having four flat sides which are at right angles to each other and a back, adjacent flat sides contacting each other and extending outward perpendicular from the back while remaining in contact with each other to form a closed corner which extends outward from said back, said target apparatus having a known three dimensional (3D) shape and including markers on an interior surface of said target apparatus at known locations;

processing at least one of the received images to detect a position of the first camera to thereby determine the position of the first camera relative to the target apparatus;

operating a camera position controller to adjust the position of the camera mounting platform to position the first camera at a predetermined location in front of said target apparatus;

operating the first camera to capture an image of the interior surface of the target apparatus;

operating the camera interface to receive the image of the interior surface of said target apparatus from the first camera being calibrated; and

operating a camera calibrator configured to compare actual locations of the markers in said image of the interior surface of said target apparatus to expected locations of said markers and to generate a first set of camera calibration information corresponding to the first camera based on differences between the actual locations of the markers in said image of the interior surface of said target apparatus and the expected locations of said markers.