Method and device for refocusing at least one plenoptic video

• US 10,580,210 B2
• Filed: 12/08/2016
• Issued: 03/03/2020
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: InterDigital Patent Holdings, Inc. (InterDigital, Inc.)
• Original Assignee: InterDigital Patent Holdings, Inc. (InterDigital, Inc.)
• Inventors: Hellier, Pierre, Allie, Valerie, Perez, Patrick
• Primary Examiner(s): Spinks, Antoinette T
• Application Number:
  US16/063,273
• Publication Number:
  US 20180374263A1
• Time in Patent Office: 1,181 Days
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

A method for refocusing, on at least one common point of interest, the rendering of one set of plenoptic video data provided by one plenoptic device belonging to a set of plenoptic devices capturing simultaneously a same scene. According to the present disclosure, said method comprises: obtaining (21) a common 3D reference system used for spatially locating said plenoptic device that has provided said set of plenoptic video data and at least one other device of said set of plenoptic devices, from said at least one common point of interest, determining (22) common refocusing plane parameters in said common 3D reference system, refocusing (23) the rendering of said set of plenoptic video data by converting (231) said common refocusing plane parameters into a rendering refocusing plane of a 3D reference system associated with said plenoptic device.

Claims

1. A method comprising:

obtaining a first plenoptic video data capturing a scene from a first plenoptic device;

obtaining a common 3D reference system for spatially locating the first plenoptic device and a second plenoptic device, the second plenoptic device having captured a second plenoptic video data from the scene, and the first plenoptic video data and the second plenoptic video data having been temporally aligned; and

refocusing a rendering of at least a part of said first plenoptic video data on at least one common point of interest based on the second plenoptic video data, the common 3D reference system, and said temporal alignment.

2. The method according to claim 1, wherein obtaining a common 3D reference system comprises using structure-from-motion for determining a first 3D pose for said first plenoptic device and a second 3D pose for said second plenoptic device.

3. The method according to claim 1, further comprising determining common refocusing parameters in said common 3D reference system from said at least one common point of interest, said common refocusing parameters being converted in a rendering refocusing plane of a 3D reference system associated with said first plenoptic device.

4. The method according to claim 3, wherein said refocusing is recomputed when a distance between said rendering refocusing plane and a plane perpendicular to an optical axis of said first plenoptic device, exceeds a predetermined value.

5. The method according to claim 3, wherein said determining of common refocusing plane parameters in said common 3D reference system comprises:

determining of said at least one common point of interest by obtaining information representative of at least one point of interest associated with at least one plenoptic video data of interest capturing said scene and provided by a third plenoptic device;

using respectively at least one transformation between said common 3D reference system and at least one 3D reference system, associated with said third plenoptic device, for converting said information into said common 3D reference system; and

determining said common refocusing plane parameters from said information converted into said common 3D reference system.

6. The method according to claim 5, wherein said information is provided automatically by said third plenoptic device.

7. The method according to claim 5, wherein said information is:

representative of a single point of interest associated with a single plenoptic video data of interest; and

provided, through a user interface, associated with a device rendering one of said plenoptic video data capturing said scene.

8. The method according to claim 5, wherein said information is:

representative of at least two points of interest associated with at least one plenoptic video data of interest; and

provided, through at least one user interface, associated with at least one device rendering one of said at least one plenoptic video data of interest.

9. The method according to claim 5, wherein said method comprises a temporal filtering of said information representative of at least one point of interest, said temporal filtering being applied over a part of said at least one plenoptic video data of interest.

10. The method according to claim 1, wherein when at least two successive refocusings associated respectively with at least two rendering refocusing planes are performed at two distinct instants, said method comprises generating a sequence of frames whose respective focusing planes are located between said at least two rendering refocusing planes.

20. A non-transitory computer-readable storage medium storing program code instructions adapted to perform a method according to claim 1 when executed on a computer.

11. A device comprising a processor configured to:

obtain a first plenoptic video data capturing a scene from a first plenoptic device;

obtain a common 3D reference system for spatially locating the first plenoptic device and a second plenoptic device, the second plenoptic device having captured a second plenoptic video data from the scene, and the first plenoptic video data and the second plenoptic video data having been temporally aligned; and

refocus a rendering of at least a part of said first plenoptic video data on at least one common point of interest based on the second plenoptic video data, the common 3D reference system, and said temporal alignment.

12. The device according to claim 11, wherein the processor is further configured to obtain the common 3D reference system using structure-from-motion for determining a first 3D pose for said first plenoptic device and a second 3D pose for said second plenoptic device.

13. The device according to claim 11, wherein the processor is further configured to determine common refocusing parameters in said common 3D reference system from said at least one common point of interest, and to convert said common refocusing parameters in a rendering refocusing plane of a 3D reference system associated with said first plenoptic device.

14. The device according to claim 13, wherein said refocusing is recomputed when a distance between said rendering refocusing plane and a plane perpendicular to an optical axis of said first plenoptic device, exceeds a predetermined value.

15. The device according to claim 13, wherein the processor is further configured to:

determine of said at least one common point of interest by obtaining information representative of at least one point of interest associated with at least one plenoptic video data of interest capturing said scene and provided by a third plenoptic device;

use respectively at least one transformation between said common 3D reference system and at least one 3D reference system, associated with said third plenoptic device, for converting said information into said common 3D reference system; and

determine said common refocusing plane parameters from said information converted into said common 3D reference system.

16. The device according to claim 15, wherein said information is provided automatically by said third plenoptic device.

17. The device according to claim 15, wherein said information is:

representative of a single point of interest associated with a single plenoptic video data of interest; and

provided, through a user interface, associated with a device rendering one of said plenoptic video data capturing said scene.

18. The device according to claim 15, wherein said information is:

representative of at least two points of interest associated with at least one plenoptic video data of interest; and

provided, through at least one user interface, associated with at least one device rendering one of said at least one plenoptic video data of interest.

19. The device according to claim 11, wherein when at least two successive refocusings associated respectively with at least two rendering refocusing planes are performed at two distinct instants, said method comprises generating a sequence of frames whose respective focusing planes are located between said at least two rendering refocusing planes.