4D camera tracking and optical stabilization

• US 10,545,215 B2
• Filed: 09/13/2017
• Issued: 01/28/2020
• Est. Priority Date: 09/13/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Karafin, Jon, Pan, Gang, Nonn, Thomas, Tan, Jie
• Primary Examiner(s): Harvey, David E
• Application Number:
  US15/703,553
• Publication Number:
  US 20190079158A1
• Time in Patent Office: 867 Days
• CPC Class Codes: G01S 3/7803 Means for monitoring or calibrating
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules

Abstract

A light-field video stream may be processed to modify the camera pathway from which the light-field video stream is projected. A plurality of target pixels may be selected, in a plurality of key frames of the light-field video stream. The target pixels may be used to generate a camera pathway indicative of motion of the camera during generation of the light-field video stream. The camera pathway may be adjusted to generate an adjusted camera pathway. This may be done, for example, to carry out image stabilization. The light-field video stream may be projected to a viewpoint defined by the adjusted camera pathway to generate a projected video stream with the image stabilization.

Claims

1. A method for processing a light-field video stream, the method comprising:

at a processor, receiving a light-field video stream generated by a light-field camera;

at the processor, selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream, wherein selecting the plurality of target pixels comprises selecting as target pixels those pixels having at least one of a predetermined color characteristic or a predetermining intensity characteristic and are selected from planar regions of static, textured objects appearing in the key frames of the light-field video stream as the target pixels, and wherein selecting the plurality of target pixels further comprises;

generating a list of a plurality of targets appearing in at least one of the key frames;

generating a plane model of each of the targets for each of the key frames; and

generating a mask for each of the targets for each of the key frames, indicating one or more target pixels within each of the targets;

at the processor, using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream; and

at the processor, using the generated camera pathway to process the light-field video stream.

2. The method of claim 1, wherein selecting the plurality of target pixels further comprises:

performing superpixel segmentation of each of the key frames to identify superpixels;

calculating a motion error map for each of the key frames;

using the superpixels and motion error maps to assess texture and motion error for each of the superpixels of each of the key frames to identify a plurality of the superpixels as candidate targets;

fitting a plane to each of the candidate targets for each of the key frames; and

selecting the targets from among the candidate targets.

3. The method of claim 1, wherein using the generated camera pathway to process the light-field video stream comprises:

adjusting the camera pathway to generate an adjusted camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

wherein the method further comprises, at an output device, outputting the projected video stream.

4. A method for processing a light-field video stream, the method comprising:

at a processor, receiving a light-field video stream generated by a light-field camera;

at the processor, selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream, wherein selecting the plurality of target pixels comprises selecting as target pixels those pixels having at least one of a predetermined color characteristic or a predetermining intensity characteristic;

at the processor, using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream;

at the processor, using the generated camera pathway to process the light-field video stream; and

wherein using the target pixels to generate the camera pathway comprises;

receiving at least one selection from the group consisting of;

camera-intrinsic parameters obtained from calibration of the light-field camera;

light-field optics parameters pertinent to one or more light-field optical elements of the light-field camera; and

camera motion, generated by a sensor operating contemporaneously with capture of the light-field video stream, indicative of motion of the light-field camera during at least a segment of the light-field video stream; and

using the selection to generate the camera pathway.

5. The method of claim 4, wherein using the target pixels to generate the camera pathway comprises:

generating the camera pathway with six degrees of freedom for an entirety of the light-field video stream; and

generating a 3D mapping of the target pixels.

6. A method for processing a light-field video stream, the method comprising:

at a processor, receiving a light-field video stream generated by a light-field camera;

at the processor, selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream;

at the processor, using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream, wherein using the target pixels to generate the camera pathway comprises;

dividing the light-field video stream into a plurality of sequences, each of which begins with one of the key frames;

for each sequence;

tracking a position and/or orientation of each of the target pixels in each frame;

comparing the position and/or orientation of each of the target pixels between frames of the sequence to obtain a portion of the camera pathway for that sequence; and

obtaining the position and/or orientation of each of the target pixels for the key frame for the next sequence;

generating the camera pathway with six degrees of freedom for an entirety of the light-field video stream; and

generating a 3D mapping of the target pixels; and

at the processor, using the generated camera pathway to process the light-field video stream.

7. The method of claim 6, wherein using the generated camera pathway to process the light-field video stream comprises:

adjusting the camera pathway to generate an adjusted camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

wherein the method further comprises, at an output device, outputting the projected video stream.

8. The method of claim 7, wherein adjusting the camera pathway to generate the adjusted camera pathway comprises causing the adjusted camera pathway to be more stable than the camera pathway.

9. The method of claim 7, wherein adjusting the camera pathway to generate the adjusted camera pathway further comprises adjusting U, V coordinates within each of a plurality of frames of the light-field video stream to cause frame-to-frame motion to be relatively smooth and contiguous.

10. A non-transitory computer-readable medium for processing a light-field video stream, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

receiving a light-field video stream generated by a light-field camera;

selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream, wherein selecting the plurality of target pixels comprises;

selecting pixels with predetermined color and/or intensity characteristics by selecting pixels from planar regions of static, textured objects appearing in the key frames of the light-field video stream as the target pixels by;

generating a list of a plurality of targets appearing in at least one of the key frames;

generating a plane model of each of the targets for each of the key frames; and

generating a mask for each of the targets for each of the key frames, indicating one or more target pixels within each of the targets;

using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream; and

using the generated camera pathway to process the light-field video stream.

11. The non-transitory computer-readable medium of claim 10, wherein selecting the plurality of target pixels further comprises:

performing superpixel segmentation of each of the key frames to identify superpixels;

calculating a motion error map for each of the key frames;

using the superpixels and motion error maps to assess texture and motion error for each of the superpixels of each of the key frames to identify a plurality of the superpixels as candidate targets;

fitting a plane to each of the candidate targets for each of the key frames; and

selecting the targets from among the candidate targets.

12. The non-transitory computer-readable medium of claim 10, wherein using the generated camera pathway to process the light-field video stream comprises:

adjusting the camera pathway to generate an adjusted camera pathway such that the adjusted camera pathway is more stable than the camera pathway;

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

wherein the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by one or more processors, cause an output device to output the projected video stream.

13. A system for processing a light-field video stream, the system comprising:

a processor configured to;

receive a light-field video stream generated by a light-field camera;

select a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream by;

selecting pixels from planar regions of static, textured objects appearing in the key frames of the light-field video stream as the target pixels;

generating a list of a plurality of targets appearing in at least one of the key frames;

generating a plane model of each of the targets for each of the key frames; and

generating a mask for each of the targets for each of the key frames, indicating one or more target pixels within each of the targets;

use the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream; and

use the generated camera pathway to process the light-field video stream; and

an output device configured to output the light-field video stream.

14. The system of claim 13, wherein the processor is further configured to select the plurality of target pixels by:

performing superpixel segmentation of each of the key frames to identify superpixels;

calculating a motion error map for each of the key frames;

using the superpixels and motion error maps to assess texture and motion error for each of the superpixels of each of the key frames to identify a plurality of the superpixels as candidate targets;

fitting a plane to each of the candidate targets for each of the key frames; and

selecting the targets from among the candidate targets.

15. The system of claim 13, wherein the processor is further configured to use the target pixels to generate the camera pathway by:

receiving at least one selection from the group consisting of;

camera-intrinsic parameters obtained from calibration of the light-field camera;

light-field optics parameters pertinent to one or more light-field optical elements of the light-field camera; and

camera motion, generated by a sensor operating contemporaneously with capture of the light-field video stream, indicative of motion of the light-field camera during at least a segment of the light-field video stream; and

using the selection to generate the camera pathway.

16. The system of claim 13, wherein the processor is further configured to use the generated camera pathway to process the light-field video stream by:

adjusting the camera pathway to generate an adjusted camera pathway such that the adjusted camera pathway is more stable than the camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

and wherein the output device is further configured to output the projected video stream.

17. A system for processing a light-field video stream, the system comprising:

a processor configured to;

receive a light-field video stream generated by a light-field camera;

select a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream;

use the target pixels to generate, in three dimensions, a camera pathway with six degrees of freedom and indicative of motion of the light-field camera during generation of the light-field video stream by;

dividing the light-field video stream into a plurality of sequences, each of which begins with one of the key frames; and

for each sequence;

tracking a position and/or orientation of each of the target pixels in each frame;

comparing the position and/or orientation of each of the target pixels between frames of the sequence to obtain a portion of the camera pathway for that sequence; and

obtaining the position and/or orientation of each of the target pixels for the key frame for the next sequence; and

generating a 3D mapping of the target pixels;

use the generated camera pathway to process the light-field video stream; and

an output device configured to output the light-field video stream.

18. The system of claim 17, wherein the processor is further configured to use the generated camera pathway to process the light-field video stream by:

adjusting the camera pathway to generate an adjusted camera pathway such that the adjusted camera pathway is more stable than the camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

and wherein the output device is further configured to output the projected video stream.