Calibration for immersive content systems

• US 9,865,055 B2
• Filed: 02/23/2016
• Issued: 01/09/2018
• Est. Priority Date: 02/24/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael, Moss, Alan McKay, Medina, Hector
• Primary Examiner(s): He, Yingchun
• Application Number:
  US15/051,550
• Publication Number:
  US 20160253795A1
• Time in Patent Office: 686 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Camera and/or lens calibration information is generated as part of a calibration process in video systems including 3-dimensional (3D) immersive content systems. The calibration information can be used to correct for distortions associated with the source camera and/or lens. A calibration profile can include information sufficient to allow the system to correct for camera and/or lens distortion/variation. This can be accomplished by capturing a calibration image of a physical 3D object corresponding to the simulated 3D environment, and creating the calibration profile by processing the calibration image. The calibration profile can then be used to project the source content directly into the 3D viewing space while also accounting for distortion/variation, and without first translating into an intermediate space (e.g., a rectilinear space) to account for lens distortion.

Claims

1. A method of processing video image data for display, comprising:

receiving, at a content playback device, first data including first video image data captured using at least a first camera, said first video image data representing a first video image;

using a processor to identify a first calibration profile corresponding to the first camera, the first calibration profile including a set of UV pairs corresponding to vertices in a 2D texture map, said 2D texture map being a 2D texture map corresponding to a three-dimensional (3D) surface of a 3D model;

with a 3D renderer executing on one or more processors;

graphically rendering the first video image data or a portion thereof

onto the 3D surface of the 3D model using at least the first

calibration profile to map at least a portion of the first video image onto the surface of the 3D model; and

displaying the rendered first video image data or portion thereof on a display.

2. The method of claim 1, further comprising:

receiving, at said content playback device, second data including second video image data captured using at least a second camera, said second video image data representing a second video image;

identifying, using the processor, a second calibration profile corresponding to the second camera, the second calibration profile including a second set of UV pairs corresponding to vertices in the 2D texture map;

with the 3D renderer executing on one or more processors;

graphically rendering the second video image data or a portion thereof

onto the 3D surface of the 3D model using at least the second calibration profile to map at least a portion of the second video image onto the surface of the 3D model.

3. The method of claim 2, wherein the first and second cameras capture different overlapping portions of a current viewing scene, the method further comprising:

electronically generating an output frame for display on said display by seaming together at least a portion of the first rendered video image data with at least a portion of the second rendered video image data.

4. The method of claim 3, further comprising:

displaying the electronically generated output frame on an electronic display screen of a head-mounted device, wherein said step of displaying the electronically generated output frame on an electronic display screen of a head-mounted device includes said step of displaying the rendered first video image data or portion thereof on the display.

5. The method of claim 1, wherein graphically rendering the first video image data or a portion thereof onto the 3D surface of the 3D model using at least the first calibration profile to map at least a portion of the first video image onto the surface of the 3D model includes:

using said first calibration profile along with the 2D texture map to paint a group of pixels in the first video image corresponding to a first portion of the 2D texture map onto a portion of the surface of the 3D model; and

wherein the first camera was used to capture the first video image data using a fisheye lens.

6. The method of claim 5,

wherein the first data includes information sufficient to determine a source camera used to capture at least a portion of the first video image data, a lens used to capture at least a portion of the first video image data, or both;

and

wherein said step of using a processor to identify the first calibration profile includes identifying the first calibration profile from said information sufficient to determine a source camera used to capture at least a portion of the first video image data, a lens used to capture at least a portion of the first video image data, or both.

7. The method of claim 6, further comprising:

operating the content playback device to request the first calibration profile from a content delivery system from which the first video image data is received.

8. The method of claim 7, further comprising:

receiving both the first calibration profile and the 2D texture map corresponding to the three-dimensional (3D) surface of the 3D model from the content delivery system.

9. The method of claim 1, further comprising:

receiving, at the content playback device, both the first calibration profile and the 2D texture map corresponding to the three-dimensional (3D) surface of the 3D model.

10. The method of claim 9, wherein the 3D model is a polygon mesh model, the method further comprising:

receiving, in addition to the first calibration profile and the 2D texture map, polygon information which is part of said 3D model and surface information which is also part of said 3D model.

11. The method of claim 9, wherein said first calibration profile includes, in addition to said set of UV pairs corresponding to vertices in the 2D texture map, a distortion correction.

12. A content playback device, comprising:

an interface configured to receive first data including first video image data captured using at least a first camera, said first video image data representing a first video image;

a processor configured to;

identify a first calibration profile corresponding to the first camera, the first calibration profile including a set of UV pairs corresponding to vertices in a 2D texture map, said 2D texture map being a 2D texture map corresponding to a three-dimensional (3D) surface of a 3D model; and

graphically render the first video image data or a portion thereof onto the 3D surface of the 3D model using at least the first calibration profile to map at least a portion of the first video image onto the surface of the 3D model; and

a display configured to display the rendered first video image data or portion thereof on the display.

13. The content playback device of claim 12,

wherein the interface is further configured to receive second data including second video image data captured using at least a second camera, said second video image data representing a second video image;

and

wherein the processor is further configured to;

identify a second calibration profile corresponding to the second camera, the second calibration profile including a second set of UV pairs corresponding to vertices in the 2D texture map; and

graphically render the second video image data or a portion thereof onto the 3D surface of the 3D model using at least the second calibration profile to map at least a portion of the second video image onto the surface of the 3D model.

14. The content playback device of claim 13, wherein the first and second cameras capture different overlapping portions of a current viewing scene;

and

wherein the processor is further configured to;

electronically generate an output frame for display on said display by seaming together at least a portion of the first rendered video image data with at least a portion of the second rendered video image data.

15. The content playback device of claim 14,

wherein the processor is further configured to control the content playback device to:

display the electronically generated output frame on an electronic display screen of a head-mounted device, wherein displaying the electronically generated output frame on an electronic display screen of a head-mounted device includes said displaying the rendered first video image data or portion thereof on the display.

16. The content playback device of claim 12,

wherein the processor is configured, as part of said graphical rendering, to use said first calibration information along with the 2D texture map to paint a group of pixels in the first video image corresponding to a first portion of the 2D texture map onto a portion of the surface of the 3D model.

17. The content playback device of claim 16,

wherein the first data includes information sufficient to determine a source camera used to capture at least a portion of the first video image data, a lens used to capture at least a portion of the first video image data, or both;

and

wherein the processor is configured to identify the first calibration profile from said information sufficient to determine a source camera used to capture at least a portion of the first video image data, a lens used to capture at least a portion of the first video image data, or both.

18. The content playback device of claim 16,

wherein the processor is further configured to control the content playback device to request the first calibration profile from a content delivery system from which the first video image data is received.

19. The content playback device of claim 18,

wherein the interface is further configured to receive, both the first calibration profile and the 2D texture map corresponding to the three-dimensional (3D) surface of the 3D model from the content delivery system.

20. The content playback device of claim 12,

wherein the interface is further configured to receive both the first calibration profile and the 2D texture map corresponding to the three-dimensional (3D) surface of the 3D model.

21. The content playback device of claim 20,

wherein the 3D model is a polygon mesh model;

and

wherein the interface is further configured to receive, in addition to the first calibration profile and the 2D texture map, polygon information which is part of said 3D model and surface information which is also part of said 3D model.