Video refocusing

• US 10,298,834 B2
• Filed: 07/11/2016
• Issued: 05/21/2019
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pitts, Colvin, Ng, Yi-Ren
• Primary Examiner(s): Vu, Kim Y
• Assistant Examiner(s): Bloom, Nathan
• Application Number:
  US15/207,110
• Publication Number:
  US 20170134639A1
• Time in Patent Office: 1,044 Days
• Field of Search: None
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/20108 Interactive selection of 2D slice in a 3D data set
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H01L 27/14627 Microlenses
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

A video refocusing system operates in connection with refocusable video data, information, images and/or frames, which may be light field video data, information, images and/or frames, that may be focused and/or refocused after acquisition or recording. A video acquisition device acquires first refocusable light field video data of a scene, stores first refocusable video data representative of the first refocusable light field video data, acquires second refocusable light field video data of the scene after acquiring the first refocusable light field video data, determines a first virtual focus parameter (such as a virtual focus depth) using the second refocusable light field video data, generates first video data using the stored first refocusable video data and the first virtual focus parameter, wherein the first video data includes a focus depth that is different from an optical focus depth of the first refocusable light field video data, and outputs the first video data.

Claims

1. A method of processing light field video data, comprising:

acquiring light field video data of a scene, the acquired light field video data having an optical focus depth, the acquired light field video data comprising a representation having at least four dimensions and comprising at least an indication of a direction of travel of light;

in a processor, determining a focus parameter;

in the processor, generating refocused video data using the acquired light field video data and the focus parameter, wherein the generated refocused video data has a focus depth that is different from the optical focus depth of the acquired light field video data; and

at an output device, outputting the refocused video data.

2. The method of claim 1, wherein generating the refocused video data occurs substantially concurrently with acquiring the light field video data.

3. The method of claim 1, further comprising storing the acquired light field video data in a storage device;

and wherein generating refocused video data using the acquired light field video data comprises processing the stored light field video data.

4. The method of claim 1, wherein the focus parameter comprises information representing a predetermined focus depth of an area of the scene.

5. The method of claim 1, wherein the focus parameter comprises information representing focus tracking.

6. The method of claim 5, further comprising:

in the processor, identifying at least one subject of interest in the scene; and

in the processor, generating the focus parameter based on the at least one subject of interest.

7. The method of claim 6, wherein identifying at least one subject of interest in the scene comprises receiving, at a user input device, user input indicative of the at least one subject.

8. The method of claim 1, wherein the optical focus depth of the acquired light field video data varies during acquisition thereof.

9. The method of claim 8, wherein generating the refocused video data comprises generating refocused video data having a predetermined focus depth.

10. The method of claim 1, wherein the optical focus depth of the acquired light field video data is fixed during acquisition thereof.

11. The method of claim 1, further comprising:

at a user input device, receiving user input after acquiring the light field video data, wherein the user input is indicative of a selected focus depth, and wherein determining the focus parameter comprises using the selected focus depth.

12. The method of claim 1, wherein generating the refocused video data comprises generating refocused video data using a tilted refocus plane.

13. The method of claim 1, wherein:

the acquired light field video data has a first perspective;

and wherein generating the refocused video data comprises generating refocused video data having a second perspective that is shifted with respect to the first perspective.

14. The method of claim 1, wherein the determined focus parameter is time-varying.

15. The method of claim 1, wherein the light field video data comprises a plurality of key frames, at least one of the key frames comprising an indication of a time-varying focus parameter;

and wherein determining a focus parameter comprises determining the focus parameter based on at least one key frame.

16. The method of claim 1, wherein the light field video data comprises a plurality of key frames, at least one of the key frames comprising an indication of a selected subject;

and wherein determining a focus parameter comprises determining a time-varying focus parameter based on determined time-varying focus depths for the selected subject.

17. A light field video acquisition device comprising:

optics;

a photosensor array, configured to acquire light field video data of a scene, the acquired light field video data having an optical focus depth, the acquired light field video data comprising a representation having at least four dimensions and comprising at least an indication of a direction of travel of light;

a microlens array, positioned between the optics and the photosensor array so that light rays are directed onto the photosensor array via the optics and the microlens array;

processing circuitry, communicatively coupled to the photosensor array, configured to;

determine a focus parameter; and

generate refocused video data using the acquired light field video data and the focus parameter, wherein the generated refocused video data has a focus depth that is different from the optical focus depth of the acquired light field video data; and

an output device, communicatively coupled to the processing circuitry, configured to output the refocused video data.

18. The light field video acquisition device of claim 17, wherein the processing circuitry is configured to generate the refocused video data substantially concurrently with the photosensor array acquiring the light field video data.

19. The light field video acquisition device of claim 17, further comprising:

a storage device, communicatively coupled to the photosensor array, configured to store the acquired light field video data;

and wherein the processing circuitry is configured to generate the refocused video data by processing the stored light field video data.

20. The light field video acquisition device of claim 17, wherein the focus parameter comprises information representing a predetermined focus depth of an area of the scene.

21. The light field video acquisition device of claim 17, wherein the focus parameter comprises information representing focus tracking.

22. The light field video acquisition device of claim 21, wherein the processing circuitry is further configured to:

identify at least one subject of interest in the scene; and

generate the focus parameter based on the at least one subject of interest.

23. The light field video acquisition device of claim 22, further comprising:

a user input device, communicatively coupled to the processing circuitry

and configured to receive user input indicative of the at least one subject;

and wherein the processing circuitry is configured to identify the at least one subject of interest in the scene based on the received user input.

24. The light field video acquisition device of claim 17, wherein the optical focus depth of the acquired light field video data varies during acquisition thereof.

25. The light field video acquisition device of claim 24, wherein the processing circuitry is configured to generate refocused video data having a predetermined focus depth.

26. The light field video acquisition device of claim 17, wherein the optical focus depth of the acquired light field video data is fixed during acquisition thereof.

27. The light field video acquisition device of claim 17, further comprising:

a user input device, communicatively coupled to the processing circuitry and configured to receive user input after the photosensor array has acquired the light field video data, wherein the user input is indicative of a selected focus depth;

and wherein the processing circuitry is configured to determine the focus parameter using the selected focus depth.

28. The light field video acquisition device of claim 17, wherein the processing circuitry is configured to generate the refocused video data using a tilted refocus plane.

29. The light field video acquisition device of claim 17, wherein:

the acquired light field video data has a first perspective;

and wherein the processing circuitry is configured to generate refocused video data having a second perspective that is shifted with respect to the first perspective.

30. The light field video acquisition device of claim 17, wherein the determined focus parameter is time-varying.

31. The light field video acquisition device of claim 17, wherein the light field video data comprises a plurality of key frames, at least one of the key frames comprising an indication of a time-varying focus parameter;

and wherein the processing circuitry is configured to determine the focus parameter based on at least one key frame.

32. The light field video acquisition device of claim 17, wherein the light field video data comprises a plurality of key frames, at least one of the key frames comprising an indication of a selected subject;

and wherein the processing circuitry is configured to determine the focus parameter by determining a time-varying focus parameter based on determined time-varying focus depths for the selected subject.

33. A computer program product for processing light field video data, comprising:

a nontransitory storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

determining a focus parameter for acquired light field video data of a scene, the acquired light field video data having an optical focus depth, the acquired light field video data comprising a representation having at least four dimensions and comprising at least an indication of a direction of travel of light;

generating refocused video data using the acquired light field video data and the focus parameter, wherein the generated refocused video data has a focus depth that is different from the optical focus depth of the acquired light field video data; and

causing an output device to output the refocused video data.

34. The computer program product of claim 33, wherein the computer program code configured to cause at least one processor to generate the refocused video data comprises computer program code configured to cause at least one processor to generate the refocused video data substantially concurrently with acquiring the light field video data.

35. The computer program product of claim 33, further comprising computer program code configured to cause a storage device to store the acquired light field video data in a storage device;

and wherein the computer program code configured to cause at least one processor to generate the refocused video data comprises computer program code configured to cause at least one processor to process the stored light field video data.

36. The computer program product of claim 33, wherein the focus parameter comprises information representing a predetermined focus depth of an area of the scene.

37. The computer program product of claim 33, wherein the focus parameter comprises information representing focus tracking.

38. The computer program product of claim 37, further comprising computer program code, encoded on the medium, configured to cause the at least one processor to perform the steps of:

identifying at least one subject of interest in the scene; and

generating the focus parameter based on the at least one subject of interest.

39. The computer program product of claim 38, wherein the computer program code configured to cause at least one processor to identify at least one subject of interest in the scene comprises computer program code configured to cause at least one processor to receive, from a user input device, user input indicative of the at least one subject.

40. The computer program product of claim 33, wherein the optical focus depth of the acquired light field video data varies during acquisition thereof.

41. The computer program product of claim 40, wherein the computer program code configured to cause at least one processor to generate the refocused video data comprises computer program code configured to cause at least one processor to generate refocused video data having a predetermined focus depth.

42. The computer program product of claim 33, wherein the optical focus depth of the acquired light field video data is fixed during acquisition thereof.

43. The computer program product of claim 33, further comprising:

computer program code configured to cause at least one processor to receiving, from a user input device, user input after acquiring the light field video data,

wherein the user input is indicative of a selected focus depth, and

wherein the computer program code configured to cause at least one processor to determine the focus parameter comprises computer program code configured to cause at least one processor to determine the focus parameter using the selected focus depth.

44. The computer program product of claim 33, wherein the computer program code configured to cause at least one processor to generate the refocused video data comprises computer program code configured to cause at least one processor to generate refocused video data using a tilted refocus plane.

45. The computer program product of claim 33, wherein:

the acquired light field video data has a first perspective;

and wherein the computer program code configured to cause at least one processor to generate the refocused video data comprises computer program code configured to cause at least one processor to generate refocused video data having a second perspective that is shifted with respect to the first perspective.

46. The computer program product of claim 33, wherein the determined focus parameter is time-varying.

47. The computer program product of claim 33, wherein the light field video data comprises a plurality of key frames, at least one of the key frames comprising an indication of a time-varying focus parameter;

and wherein the computer program code configured to cause at least one processor to determine a focus parameter comprises computer program code configured to cause at least one processor to determine the focus parameter based on at least one key frame.

48. The computer program product of claim 33, wherein the light field video data comprises a plurality of key frames, at least one of the key frames comprising an indication of a selected subject;

and wherein the computer program code configured to cause at least one processor to determine a focus parameter comprises computer program code configured to cause at least one processor to determine a time-varying focus parameter based on determined time-varying focus depths for the selected subject.