CALIBRATION FOR IMMERSIVE CONTENT SYSTEMS

• US 20160253795A1
• Filed: 02/23/2016
• Published: 09/01/2016
• Est. Priority Date: 02/24/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael, Medina, Hector, Moss, Alan McKay
• Granted Patent: US 9,865,055 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Camera and/or lens calibration information is generated as part of a calibration process in video systems including 3-dimensional (3D) immersive content systems. The calibration information can be used to correct for distortions associated with the source camera and/or lens. A calibration profile can include information sufficient to allow the system to correct for camera and/or lens distortion/variation. This can be accomplished by capturing a calibration image of a physical 3D object corresponding to the simulated 3D environment, and creating the calibration profile by processing the calibration image. The calibration profile can then be used to project the source content directly into the 3D viewing space while also accounting for distortion/variation, and without first translating into an intermediate space (e.g., a rectilinear space) to account for lens distortion.

Claims

1. A head mountable display system configured to render and display immersive content, comprising:

a support adapted to be worn by a user;

one or more electronic displays carried by the support; and

one or more processors programmed to;

access first data including first video image data captured using at least a first camera;

identify a first calibration profile to apply to the first video image data, where the first calibration profile is usable to map pixel groups within the first video image data to respective portions of a two dimensional (2D) texture map, the 2D texture map corresponding to a three-dimensional (3D) surface associated with a physical calibration object;

use at least a 3D digital model of the physical calibration object to graphically render a first representation of the 3D surface;

use at least the first calibration profile to graphically render the first video image data or a portion thereof onto the first representation of the 3D surface, resulting in first rendered content; and

display the first rendered content on the one or more electronic displays.

2. The system of claim 1 wherein the one or more processors are further programmed to:

access second data including second video image data captured using at least a second camera;

identify a second calibration profile to apply to the second video image data, where the second calibration profile is usable to map pixel groups within the second video image data to respective portions of the 2D texture map;

use the 3D digital model of the physical calibration object to graphically render a second representation of the 3D surface;

use at least the second calibration profile to graphically render the second video image data or a portion thereof onto the second representation of the 3D surface, resulting in second rendered content; and

display the second rendered content on the one or more electronic displays.

3. The system of claim 2 wherein the first and second cameras capture different overlapping portions of a current viewing scene, and the one or more processors are further programmed to electronically generate an output frame for display on the one or more electronic displays by seaming together at least a portion of the first representation of the 3D surface with at least a portion of the second representation of the 3D surface.

4. The system of claim 1 wherein the first camera was used to capture the first video image data using a fisheye lens, and wherein the 3D surface corresponds to an interior curved surface of the 3D object.

5. The system of claim 1 wherein the first data includes information sufficient to determine a source camera used to capture at least a portion of the first video image data, a lens used to capture at least a portion of the first video image data, or both.

6. A method of generating camera calibration information for use in displaying immersive content, comprising:

receiving first digital image data corresponding to a first image captured using a digital video camera and a wide angle lens, the first image taken of a three-dimensional (3D) object having a plurality of markers distributed thereon;

accessing a 3D digital model corresponding to the 3D object, the 3D digital model having a two-dimensional (2D) texture map associated therewith that corresponds to a 3D surface of the 3D digital model;

identifying a plurality of pixel groups in the first image data using a processor, where each pixel group of the plurality of pixel groups includes one or more pixels of the first image data that correspond to a different one of the plurality of markers; and

generating, using a processor, a mapping between the identified pixel groups and the 2D texture map corresponding to the 3D surface, the mapping usable for calibrating second digital image data for display in a simulated 3D viewing environment.

7. The method of claim 6 wherein a graphical representation of the 3D surface is created using the 3D digital model, and wherein the first image can be applied to the 3D surface using the mapping to create the simulated 3D viewing environment.

8. The method of claim 6 wherein the second digital image data is captured using one or both of the same digital video camera and the same lens that were used to capture the first digital image data.

9. The method of claim 6 wherein the second digital image data is captured using one or both of the same type of digital video camera and the same type of lens that were used to capture the first digital image data.

10. The method of claim 6 further comprising performing a feature detection algorithm using the processor to locate the plurality of markers within the first image and identify the plurality of pixel groups.

11. The method of claim 6 wherein the lens is a fish-eye lens.

12. The method of claim 6 wherein the 3D surface comprises a concave surface.

13. A method of processing video image data for display, comprising:

receiving first data including first video image data captured using at least a first camera;

using a processor to identify a first calibration profile to apply to the first video image data, where the first calibration profile is usable to map pixel groups within the first video image data to respective portions of a two dimensional (2D) texture map, the 2D texture map corresponding to a three-dimensional (3D) surface associated with a physical calibration object; and

with a 3D renderer executing on one or more processors;

graphically rendering a first representation of the 3D surface using at least a 3D digital model of the physical calibration object; and

graphically rendering the first video image data or a portion thereof onto the first representation of the 3D surface using at least the first calibration profile.

14. The method of claim 13 further comprising:

receiving second data including second video image data captured using at least a second camera;

identifying, using a processor, a second calibration profile to apply to the second video image data, where the second calibration profile is usable to map pixel groups within the second video image data to respective portions of the 2D texture map;

with a 3D renderer executing on one or more processors;

graphically rendering a second representation of the 3D surface using the 3D digital model of the physical calibration object; and

graphically rendering the second video image data or a portion thereof onto the second representation of the 3D surface using at least the second calibration profile.

15. The method of claim 14 wherein the first and second cameras capture different overlapping portions of a current viewing scene, the method further comprising electronically generating an output frame for display by seaming together at least a portion of the first representation of the 3D surface with at least a portion of the second representation of the 3D surface.

16. The method of claim 13 further comprising displaying the representation of the 3D surface on an electronic display screen of a head-mounted device.

17. The method of claim 13 wherein the first camera was used to capture the first video image data using a fisheye lens, and wherein the 3D surface corresponds to an interior curved surface of the 3D object.

18. The method of claim 13 wherein the first data includes information sufficient to determine a source camera used to capture at least a portion of the first video image data, a lens used to capture at least a portion of the first video image data, or both.

19. A system, comprising:

one or more computing devices comprising one or more hardware processors, the one or more computing devices programmed to;

receive first digital image data corresponding to a first image captured using a digital video camera and a wide angle lens, the first image taken of a three-dimensional (3D) object having a plurality of markers distributed thereon;

access a 3D digital model corresponding to the 3D object, the 3D digital model having a two-dimensional (2D) texture map associated therewith that corresponds to a 3D surface of the 3D digital model;

process the first image data to identify a plurality of pixel groups in the first image data, where each pixel group of the plurality of pixel groups includes one or more pixels of the first image data that correspond to a different one of the plurality of markers; and

generate a mapping between the identified pixel groups and the 2D texture map corresponding to the 3D surface, the mapping usable for calibrating second digital image data for display in a simulated 3D viewing environment.

20. The system of claim 19 wherein the one or more computing devices are further programmed to perform a feature detection algorithm to locate the plurality of markers within the first image and identify the plurality of pixel groups.

21. The system of claim 20 wherein the plurality of markers comprise a plurality of grid-intersection points.

22. The system of claim 19 wherein the lens is an ultra-wide angle lens and the 3D surface comprises a concave surface.

23. The system of claim 19 wherein the 3D object is a partial sphere, partial ellipsoid, partial ovoid, or partial cone, and the 3D surface corresponds to at least a portion of an interior surface of the 3D object.