System of and Method for Video Refocusing

• US 20100128145A1
• Filed: 11/20/2009
• Published: 05/27/2010
• Est. Priority Date: 11/25/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren, Pitts, Colvin
• Granted Patent: US 8,570,426 B2
• Time in Patent Office: Days
• US Class Current: 348/231.990
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 7/01 Conversion of standards , e.g. involving analogue television standards or digital television standards processed at pixel level

Abstract

Certain systems and methods are directed to acquiring, generating, manipulating and/or editing (for example, focusing or refocusing) refocusable video data, information, images and/or frames. The refocusable video data, information, images and/or frames may be light field video data, information, images and/or frames, that may be focused and/or re-focused after acquisition or recording of such video data, information, images and/or frames. In one aspect, a method of generating video data of a scene using a video acquisition device which acquires refocusable light field video data is disclosed, the method comprising (a) acquiring first refocusable light field video data of a scene, (b) storing first refocusable video data which is representative of the first refocusable light field video data, (c) acquiring second refocusable light field video data of the scene after acquiring the first refocusable light field video data, (d) determining a first virtual focus parameter (for example, a virtual focus depth) using the second refocusable light field video data, (e) generating first video data using the stored first refocusable video data and the first virtual focus parameter, wherein the first video data includes a focus depth that is different from an optical focus depth of the first refocusable light field video data, and (f) outputting the first video data to, for example, memory, a video display, processing circuitry, and/or a recording device.

Claims

1. A method of operating a video acquisition device which acquires refocusable light field video data, the method comprising:

acquiring first refocusable light field video data of a scene;

storing first refocusable video data which is representative of the first refocusable light field video data;

acquiring second refocusable light field video data of the scene after acquiring the first refocusable light field video data;

determining a first virtual focus parameter using the second refocusable light field video data;

generating first video data using the stored first refocusable video data and the first virtual focus parameter, wherein the first video data includes a focus depth that is different from an optical focus depth of the first refocusable light field video data; and

outputting the first video data.

2. The method of claim 1 wherein generating the first video data occurs substantially concurrently with acquiring the second refocusable light field video data.

3. The method of claim 1 wherein storing first refocusable video data includes buffering up to a predetermined amount of first refocusable video data.

4. The method of claim 1 wherein storing first refocusable video data includes storing up to a predetermined amount of most recently acquired first refocusable video data.

5. The method of claim 1 wherein storing first refocusable video data includes storing an amount of first refocusable video data which corresponds to three or more seconds of first refocusable light field video data.

6. The method of claim 1 wherein the first virtual focus parameter includes information which is representative of at least one focus depth which is different from the optical focus depth of the first refocusable light field video data.

7. The method of claim 1 wherein storing first refocusable video data includes storing an amount of first refocusable video data which corresponds to less than ten seconds of first refocusable light field video data, and wherein the less than ten seconds of first refocusable light field video data corresponds to the ten seconds of first refocusable light field video data acquired substantially immediately before determining the first virtual focus parameter.

8. The method of claim 1 wherein the first virtual focus parameter includes information which is representative of maintaining a predetermined focus depth of an area of the scene.

9. The method of claim 1 wherein the first virtual focus parameter includes information which is representative of focus tracking.

10. The method of claim 9 wherein the focus tracking includes:

identifying one or more subjects of interest in the scene; and

generating the first video data having a predetermined virtual focus depth relative to the one or more subjects wherein the predetermined virtual focus depth is different from the optical focus depth of the first refocusable light field video data.

11. The method of claim 1 wherein the optical focus depth of the first refocusable light field video data varies during acquisition thereof.

12. The method of claim 11 further including:

configuring the video acquisition device to include a varying optical focus depth of the video acquisition device, wherein the varying optical focus depth of the video acquisition device allows generating the first video data having a predetermined focus depth relative to the one or more subjects;

generating the first video data having the predetermined focus depth relative to the one or more subjects wherein the varying optical focus depth is different from the predetermined focus depth relative to the one or more subjects.

13. The method of claim 1 wherein the optical focus depth of the first refocusable light field video data is fixed during acquisition thereof.

14. The method of claim 1 further including:

receiving a user input indicative of a subject in the scene after acquiring the first refocusable light field video data, wherein determining the first virtual focus parameter includes using a focus depth of the subject in the scene and the second refocusable light field video data.

15. The method of claim 1 further including:

receiving a user input indicative of a location in the scene after acquiring the first refocusable light field video data, wherein determining the first virtual focus parameter includes using a focus depth of the location in the scene and the second refocusable light field video data.

16. The method of claim 1 further including:

receiving a user input after acquiring the first refocusable light field video data, wherein the user input is indicative of a selected focus depth and, wherein determining the first virtual focus parameter includes using the selected focus depth and the second refocusable light field video data.

17. The method of claim 1 further including:

storing second refocusable video data which is representative of the second refocusable light field video data;

acquiring third refocusable light field video data of the scene after acquiring the second refocusable light field video data;

determining a second virtual focus parameter using at least a portion of the first, second and/or third refocusable light field video data;

generating second video data using the second refocusable video data and the second virtual focus parameter, wherein the second video data includes a focus depth that is different from the optical focus depth of the second refocusable light field video data of the scene; and

outputting the second video data, wherein the first and second video data are temporally contiguous.

18. The method of claim 17 wherein:

the first and second virtual focus parameters each include information which is representative of focus tracking, wherein the focus tracking includes;

identifying one or more subjects of interest in the scene,

generating the first video data having a predetermined focus depth relative to the one or more subjects, and

adjusting the focus depth of the second video data to maintain the predetermined focus relative to the one or more subjects for a predetermined period of time.

19. A light field video acquisition device which acquires refocusable light field video data, the light field video acquisition device comprising:

optics;

photosensor array to acquire first light field video data and second light field video data after acquisition of the first light field video data;

a microlens array, disposed between the optics and the photosensor array, the light rays being directed onto the photosensor array via the optics and the microlens array;

first memory to store up to a predetermined amount of;

first refocusable video data which is representative of the first light field video data,

second refocusable video data which is representative of the second light field video data,

first summary data derived from the first light field video data, and second summary data derived from the second light field video data wherein the second summary data is derived after acquisition of the first light field video data;

second memory to store video data; and

processing circuitry, coupled to the first memory and the second memory, to

(i) determine a first virtual focus parameter using the second summary data,

(ii) generate first video data using the first refocusable video data and the first virtual focus parameter, wherein the first video data includes a focus depth that is different from the optical focus depth of the acquired first light field video data, and

(iii) output the first video data to the second memory.

20. The light field video acquisition device of claim 19 further including:

an interface, coupled to the processing circuitry, to receive input focus data, wherein the processing circuitry determines the first virtual focus parameter using the second summary data and the input focus data.

21. The light field video acquisition device of claim 20 wherein the interface is a user interface which is configured to receive a user input which is representative of the input focus data.

22. The light field video acquisition device of claim 19 wherein the first virtual focus parameter includes information which is representative of focus tracking.

23. A method of generating video data of a scene using a video acquisition device which acquires refocusable light field video data, the method comprising:

acquiring a series of temporally contiguous refocusable light field video data frames, wherein each refocusable light field video data frame includes an optical focus depth;

buffering a plurality of refocusable video data frames for up to a predetermined amount of time, wherein each refocusable video data frame corresponds to an associated refocusable light field data frame;

selecting a first refocusable video data frame from the plurality of buffered refocusable video data frames;

determining a first virtual focus parameter for the selected first refocusable video data frame using data which is derived from one or more refocusable video data frames which are associated with refocusable light field video data frames that are temporally acquired after acquiring the refocusable light field video data frame which is associated with the selected first refocusable video data frame;

generating a first video data frame for the selected first refocusable video data frame using (i) data which is representative of the selected first refocusable video data frame and (ii) the first virtual focus parameter, wherein the first video data frame includes a focus depth that is different from the optical focus depth of the refocusable light field video data frame which is associated with the selected first refocusable video data frame; and

outputting the first video data frame.

24. The method of claim 23 further including:

deleting a buffered refocusable video data frame;

acquiring an additional refocusable light field video data frame;

buffering an additional refocusable video data frame, wherein the additional refocusable video data frame corresponds to the additional refocusable light field video data frame;

selecting a second refocusable video data frame from a plurality of the buffered refocusable video data frames;

determining a second virtual focus parameter for the second refocusable video data frame using data which is derived from one or more refocusable video data frames which are associated with refocusable light field video data frames that are temporally acquired after acquiring the refocusable light field video data frame which is associated with the selected second refocusable video data frame;

generating a second video data frame for the selected second refocusable video data frame using (i) data which is representative of the selected second refocusable video data frame and (ii) the second virtual focus parameter, wherein the second video data frame includes a focus depth that is different from the optical focus depth of the additional refocusable light field video data frame which is associated with the selected first refocusable video data frame;

outputting the second video data frame.

25. The method of claim 23 wherein the selected first refocusable video data frame is the earliest acquired refocusable video data frame relative to the plurality of buffered refocusable video data frames.

26. The method of claim 23 wherein the first virtual focus parameter includes information which is representative of at least one focus depth.

27. The method of claim 23 wherein buffering refocusable video data for up to a predetermined amount of time includes buffering refocusable video data for three or more seconds.

28. The method of claim 23 wherein buffering refocusable video data for up to a predetermined amount of time includes buffering refocusable video data for less than ten seconds.

29. The method of claim 23 wherein the first virtual focus parameter includes information which is representative of focus tracking.

30. The method of claim 29 wherein the focus tracking includes:

identifying one or more subjects of interest in the scene; and

generating a video data frame having a predetermined focus depth relative to the one or more subjects.

31. The method of claim 30 wherein the first virtual focus parameter includes information which defines a predetermined focus relative to the one or more subjects for a predetermined period of time.