System and Method for Acquiring, Editing, Generating and Outputting Video Data

• US 20100129048A1
• Filed: 11/20/2009
• Published: 05/27/2010
• Est. Priority Date: 11/25/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren, Pitts, Colvin
• Granted Patent: US 8,279,325 B2
• Time in Patent Office: Days
• US Class Current: 386/278
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 7/01 Conversion of standards , e.g. involving analogue television standards or digital television standards processed at pixel level

Abstract

Certain systems and methods are directed to acquiring, generating, manipulating and/or editing (for example, focusing or refocusing) refocusable video data/frames. The refocusable video frames may be light field video frames that may be focused and/or refocused after acquisition or recording of such video frames. In one aspect, a method of comprises: (a) selecting a first key frame, wherein the first key frame corresponds to one of a plurality of refocusable light field video frames, (b) selecting a second key frame, wherein the second key frame corresponds to one of the plurality of refocusable light field video frames which is temporally spaced apart from the first key frame such that a plurality of refocusable light field video frames are temporally disposed between the first and the second key frames, (c) determining a virtual focus parameters for the first key frame and the second key frame, and (d) generating first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus parameter for the first key frame, (ii) the virtual focus parameter for the second key frame and (iii) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data includes a plurality of video frames, each video frame including a virtual focus depth which is based on the virtual focus parameter(s) for the first key frame and/or the second key frame.

Claims

1. A method of generating and outputting video data corresponding to a plurality of video data frames, the method comprising:

receiving refocusable light field video data, wherein the refocusable light field video data includes a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame includes an optical focus;

selecting a first key frame, wherein the first key frame corresponds to one of the plurality of refocusable light field video frames;

determining a virtual focus parameter for the first key frame;

selecting a second key frame, wherein the second key frame corresponds to one of the plurality of refocusable light field video frames which is temporally spaced apart from the first key frame such that a plurality of refocusable light field video frames are temporally disposed between the first and the second key frames;

determining a virtual focus parameter for the second key frame;

generating first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus parameter for the first key frame, (ii) the virtual focus parameter for the second key frame and (iii) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data includes a plurality of video frames, each video frame including a virtual focus depth which is based on the virtual focus parameter for the first key frame and/or virtual focus parameter for the second key frame; and

outputting the first video data.

2. The method of claim 1 wherein the virtual focus parameter of the first key frame and the virtual focus parameter of the second key frame each include information which is representative of a virtual focus depth.

3. The method of claim 1 wherein the virtual focus depth corresponding to the virtual focus parameter of the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame.

4. The method of claim 1 wherein the virtual focus parameter of the first key frame includes information which is representative of focus tracking.

5. The method of claim 1 wherein the virtual focus parameter of the first key frame and the virtual focus parameter of the second key frame each include information which is representative of a location of a virtual focal plane, aperture and/or focus effect.

6. The method of claim 1 wherein the virtual focus parameter of the first key frame includes information which provides a tilted virtual focal plane.

7. The method of claim 1 further including:

generating final output video data by processing the first video data to provide compositing and/or transitions and/or filtering and/or color adjustments of a plurality of the video frames corresponding to the first video data.

8. A method of generating and outputting video data corresponding to a plurality of video data frames, the method comprising:

receiving refocusable light field video data, wherein the refocusable light field video data includes a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame includes an optical focus;

selecting a first key frame, wherein the first key frame corresponds to one of the plurality of refocusable light field video frames;

determining a virtual focus depth for the first key frame, wherein the virtual focus depth for the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame;

selecting a second key frame, wherein the second key frame corresponds to one of the plurality of refocusable light field video frames which is temporally spaced apart from the first key frame such that a plurality of refocusable light field video frames are temporally disposed between the first and the second key frames;

determining a virtual focus depth for the second key frame, wherein the virtual focus depth for the second key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the second key frame;

generating first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus depth for the first key frame, (ii) the virtual focus depth for the second key frame, (iii) an inter-frame interpolation, and (iv) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data includes a plurality of video frames, each video frame including a virtual focus depth which is based on the virtual focus depth for the first key frame, the virtual focus depth for the second key frame, and inter-frame interpolation; and

outputting the first video data.

9. The method of claim 8 wherein the inter-frame interpolation includes linear interpolation.

10. The method of claim 8 wherein the inter-frame interpolation includes non-linear interpolation.

11. The method of claim 8 wherein the virtual focus depth of the first key frame provides a tilted virtual focal plane.

12. The method of claim 8 wherein the virtual focus depth of the first key frame and the virtual focus depth of the second key frame each provide a tilted virtual focal plane.

13. The method of claim 8 wherein a virtual focus depth of at least one video frame of the first video data provides a tilted virtual focal plane.

14. The method of claim 8 further including inputting the inter-frame interpolation.

15. The method of claim 8 further including:

generating final output video data by processing the first video data to provide compositing and/or transitions and/or filtering and/or color adjustments of a plurality of the video frames corresponding to the first video data.

16. A system to edit video refocusable light field video data, system comprising:

first memory to store refocusable light field video data, wherein the refocusable light field video data includes a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame includes an optical focus;

a user interface to receive input data which is representative of a first key frame, a second key frame, and inter-frame interpolation; and

processing circuitry, coupled to the first memory and the user interface to;

determine a virtual focus depth for the first key frame, wherein the virtual focus depth for the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame;

determine a virtual focus depth for the second key frame, wherein the virtual focus depth for the second key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the second key frame; and

generate first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus depth for the first key frame, (ii) the virtual focus depth for the second key frame, (iii) an inter-frame interpolation, and (iv) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data includes a plurality of video frames, each video frame including a virtual focus depth which is based on the virtual focus depth for the first key frame, the virtual focus depth for the second key frame, and inter-frame interpolation.

17. The system of claim 16 wherein the user interface further includes a display to visually output a plurality of the refocusable light field video frames.

18. The system of claim 17 wherein the system further includes:

second memory, coupled to the processing circuitry, wherein the second memory stores the first video data.

19. The system of claim 16 wherein the processing circuitry generates final output video by processing the first video data to provide compositing and/or transitions and/or filtering and/or color adjustments of a plurality of the video frames corresponding to the first video data.

20. The system of claim 16 wherein the inter-frame interpolation includes linear interpolation.

21. The system of claim 16 wherein the inter-frame interpolation includes non-linear interpolation.

22. The system of claim 16 wherein the virtual focus depth of the first key frame provides a tilted virtual focal plane.

23. The system of claim 16 wherein a virtual focus depth of at least one video frame of the first video data provides a tilted virtual focal plane.

24. A system for editing video refocusable light field video data, system comprising:

means for storing refocusable light field video data, wherein the refocusable light field video data includes a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame includes an optical focus;

means for receiving input data which is representative of a first key frame, a second key frame, and inter-frame interpolation;

means for;

determining a virtual focus depth for the first key frame, wherein the virtual focus depth for the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame;

determining a virtual focus depth for the second key frame, wherein the virtual focus depth for the second key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the second key frame; and

generating first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus depth for the first key frame, (ii) the virtual focus depth for the second key frame, (iii) an inter-frame interpolation, and (iv) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data includes a plurality of video frames, each video frame including a virtual focus depth which is based on the virtual focus depth for the first key frame, the virtual focus depth for the second key frame, and inter-frame interpolation.