METHOD AND SYSTEM FOR PRODUCING A VIRTUAL OUTPUT IMAGE FROM DATA OBTAINED BY AN ARRAY OF IMAGE CAPTURING DEVICES

• US 20130088489A1
• Filed: 06/20/2011
• Published: 04/11/2013
• Est. Priority Date: 06/29/2010
• Status: Active Grant

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Schmeitz, Harold Agnes Wilhelmus, De Bruijn, Frederik Jan
• Granted Patent: US 9,013,477 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/676 Bracketing for image capture at varying focusing conditions
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

In a method and system for providing virtual output images from an array of image capturing devices image data (Ii(s,t), I2(s,t)) is taken from the devices (Ci, C2). This image data is processed by convolving the image data with a function, e.g. the path (S) and thereafter deconvolving them, either after or before summation (SUM), with an inverse point spread function (IPSF) or a filter (HP) equivalent thereto to produce all-focus image data (I₀(s,t)).

Claims

1. Method for producing output image data for a virtual output image (Ī

₀(s,t)) from input image data of individual input images (I₁(s,t), I₂(s,t), I_K(s,t), I_N (s,t)) provided by an array of image capturing devices (C₁, C₂, C_K, C_N) for capturing a number of individual images from different viewpoints,

the method comprising adding and inverse filtering of corresponding pixel values of the individual input images based on multiple depth layers in a scene,

characterized in that a virtual camera position (u₀,v₀) is provided by an input virtual camera position signal (C₀) representing the virtual camera position (u₀, v₀) and the adding and inverse filtering is performed in dependence of the virtual camera position (u₀, v₀) to produce the image data for the virtual output image (Ī

₀ (s,t)) as seen from the virtual camera position (u₀,v₀), wherein the array of image capturing devices is an array of cameras around a display screen and the virtual camera position is a point at the display screen, wherein the virtual camera position is determined by measuring the position of eyes on the display screen.

2. Method as claimed in claim 1, wherein adding and inverse filtering comprises the method steps of:

constructing from the input image data of the individual images (I₁(s,t), I₂(s,t), I_K(s,t), I_N (s,t)) image data for a number of synthetically focused images at a number of depths (I_0z1 (s,t), I_0z2 (s,t)),

adding the image data of the number of synthetically focussed images at various depths to form the image data for an integrated blurred image (Ĩ

₀(s,t)), and thereafter

deconvolving the integrated blurred image (Ĩ

₀(s,t)) using an inverse point spread function (IPSF) for producing the image data for the virtual image (Ī

₀ (s,t)), from the input image data of the individual images ((I₁(s,t), I₂(s,t), I_K(s,t), I_N (s,t)).

3. Method as claimed in claim 2, wherein the method comprises:

a for a number of depths (z₁,z₂, . . . ) shifting the individual input images provided by the image capturing devices in proportion to the distance between the position of the particular image capturing devices and the virtual camera position (u₀,v₀) and in dependence on the specific depth (z₁, z₂, . . . ), to align the individual input images to the virtual camera position (u₀,v₀) and then

b. adding the image data for the shifted images at each depth together to form image data of a synthetically focused images at the number of depths (I_0z1(s,t)),

c adding the number of synthetically focussed images to form the integrated blurred image (Ĩ

₀(s,t))

d. generating a point spread function (PSF) by integrating expected blur kernels for a pixel

e. deconvolving the integrated blurred image(Ĩ

₀(s,t)), using the inverse point spread function (IPSF) of the point spread function (PSF), for producing the image data for the virtual output image (Ī

₀ (s,t)) at the virtual camera position (u₀,v₀) provided by the input virtual camera position signal (C₀).

4. Method as claimed in claim 1, wherein adding and inverse filtering comprises the method steps of

a. generating a projected path (S) of the image shift with respect to the virtual camera position (u₀,v₀) for each image capturing device of the array, the projected path S being the sum of translating impulse response functions, for aligning the image of a image capturing device with the virtual camera position (u₀,v₀) at a number of depths

b. convolving the individual images (I₁, I₂, . . . ) with their respective projected path (S) to form convolved images (I_conv1(s,t), I_conv2(s,t), . . . )

c. adding the convolved images (I_conv1(s,t), I_conv2(s,t), . . . ) to form an integrated blurred image (Ĩ

₀(s,t))

d. generating a point spread function (PSF) by integration of the projected paths

e. deconvolving the integrated image (Ĩ

₀(s,t)), using the inverse point spread function (IPSF) of the point spread function (PSF), for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camera position signal (C₀).

5. Method as claimed in claim 1 wherein adding and inverse filtering comprises the method steps of:

a. generating a projected path (S) of the image shift for each image capturing devices of the array with respect to the virtual camera position (u₀,v₀), the projected path (S) being the sum of translating impulse response functions for aligning the image with the virtual camera position at a number of depths

b. generating a point spread function (PSF) by integration of the projected paths (S)

c. for each image convolving the respective projected path (S) with the inverse point spread function (IPSF)

d. convolving each image (I₁(s,t), I₂(s,t)) with the convolution of path and inverse point spread function (S*IPSF) providing a convolved image

e. adding the convolved images for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camera position signal (C₀).

6. Method of claim 1, wherein adding and inverse filtering comprises the method steps of:

a. generating a projected path (S) of the image shift for each image capturing device of the array (C₁, C₂, C_K, C_N), the path being determined with respect to the virtual camera position (u₀,v₀), the projected path (S) being the sum of translating impulse response functions for aligning the image of a image capturing device with the virtual camera position (u₀,v₀) at a number of depths

b. convolving the individual camera images (I₁, I₂, . . . ) with their respective projected path (S)

c. convolving the image of each image capturing device with a highpass filter (HP) perpendicular to the projected path,

d. adding the convolved images for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camarea position signal (C₀).

7. Method as claimed in claim 1 wherein adding and inverse filtering comprises the method steps of:

a. generating a projected path (S) of the image shift for each image capturing device of the array, the path (S) being determined with respect to the virtual camera position (u₀,v₀), the projected path (S) being the sum of translating impulse response functions for aligning the image of a image capturing device with the virtual camera position (u₀,v₀) at a number of depths

b. generating a convolution filter for each image by convolving its respective projected path (S) with a highpass filter (S*HP) perpendicular to it

c. convolving the individual input images (I₁(s,t), I₂(s,t)) with their respective filter as generated in b

d. adding the convolved images for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camarea position signal (C₀).

8. Method as claimed in claim 1 wherein multiple virtual viewpoints ((u₀,v₀)B, (u₀,v₀)C) are generated simultaneously from a set of images captured by an array of image capturing devices.

12. Method as claimed in claim 1 wherein two virtual camera positions for left-right eye virtual camera positions are provided to provide two virtual output images at two virtual camera positions.

13. Computer program comprising program code means for performing a method as claimed in claim 1 when said program is run on a computer.

14. Computer program product comprising program code means stored on a computer readable medium for performing a method according to claim 1 when said program is run on a computer.

25. System as claimed in claim 1 wherein the array is subdivided into two or more subarrays.

26. System as claimed in claim 25, wherein the system is arranged to allocate different virtual camera positions to the subarrays.

27. Method as claimed in claim 1, wherein the at least one virtual camera position is automatically and dynamically determined by an eye detection position means.

9-11. -11. (canceled)

15. System for constructing an output image from input images obtained by an array of image capturing devices, the system comprising an array of image capturing devices (C₁, C₂, C_K, C_N) for capturing images from different view points, wherein the system comprises a means for capturing image data (I₁(s,t), I₂(s,t)) from the image capturing devices of the array, the system comprising means for adding and inverse filtering of corresponding pixel values of the individual input images based on multiple depth layers in a scene,

characterized in that the system comprises a means to provide a selected virtual camera position (u₀,v₀) by an input virtual camera position signal (C₀) representing the virtual camera position (u₀, v₀) and the means for adding and inverse filtering are arranged to operate as a function of the selected virtual camera position (u₀, v₀) to produce the image data for the virtual output image (Ī

₀ (s,t)) at the virtual camera position (u₀,v₀) provided by the input virtual camera position signal (C₀) wherein the array of image capturing devices is an array of cameras around a display screen and the virtual camera position is a point at the display screen, wherein the system comprises two interacting sub-systems, each subsystem comprising an array of cameras around a display screen, and the virtual camera position at the display screen of one of the systems is determined by the other of the systems by measuring the position of eyes of a viewer of the other of the systems.

16. System as claimed in claim 15, wherein the means for adding and inverse filtering comprises:

means for constructing from the input image data of the individual images (I₁(s,t), I₂(s,t), I_K(s,t), I_N (s,t)) image data for a number of synthetically focused images at a number of depths (I_0z1 (s,t), I_0z2 (s,t)),

means for adding the image data of the number of synthetically focussed images at various depths to form the image data for an integrated blurred image (Ĩ

₀(s,t)), and

means for deconvolving the integrated blurred image (Ĩ

₀(s,t)) using an inverse point spread function (IPSF) for producing the image data for the virtual image (Ī

₀ (s,t)), from the input image data of the individual images ((I₁(s,t), I₂(s,t), I_K(s,t), I_N (s,t)).

17. System as claimed in claim 16, wherein the system comprises:

a means for shifting for a number of depths (z₁, z₂, . . . ) the individual input images provided by the image capturing devices in proportion to the distance between the position of the particular image capturing devices and the virtual camera position (u₀,v₀) and in dependence on the specific depth (z₁, z₂, . . . ), to align the individual input images to the virtual camera position (u₀,v₀) and

b. means for adding the image data for the shifted images at each depth together to form image data of a synthetically focused images at the number of depths (I_0z1 (s,t)),

c means for adding the number of synthetically focussed images to form the integrated blurred image (Ĩ

₀(s,t))

d. means for generating a point spread function (PSF) by integrating expected blur kernels for a pixel

e. means for deconvolving the integrated blurred image(Ĩ

₀(s,t)), using the inverse point spread function (IPSF) of the point spread function (PSF), for producing the image data for the virtual output image (Ī

₀ (s,t)) at the virtual camera position (u₀,v₀) provided by the input virtual camera position signal (C₀).

18. System as claimed in claim 15, wherein the means for adding and inverse filtering comprises:

a. means for generating a projected path (S) of the image shift with respect to the virtual camera position (u₀,v₀) for each image capturing device of the array, the projected path S being the sum of translating impulse response functions, for aligning the image of a image capturing device with the virtual camera position (u₀,v₀) at a number of depths

b. means for convolving the individual images (I₁, I₂, . . . ) with their respective projected path (S) to form convolved images I_conv(s,t)

c. means for adding the convolved images I_conv(s,t) to form an integrated blurred image (Ĩ

₀(s,t))

d. means for generating a point spread function (PSF) by integration of the projected paths

e. means for deconvolving the integrated image (Ĩ

₀(s,t)), using the inverse point spread function (IPSF) of the point spread function (PSF), for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camera position signal (C₀).

19. System as claimed in claim 15 wherein the means for adding and inverse filtering comprises:

a. means for generating a projected path (S) of the image shift for each image capturing devices of the array with respect to the virtual camera position (u₀,v₀), the projected path (S) being the sum of translating impulse response functions for aligning the image with the virtual camera position at a number of depths

b. means for generating a point spread function (PSF) by integration of the projected paths (S)

c. means for convolving for each image the respective projected path (S) with the inverse point spread function (IPSF)

d. means for convolving each image (I₁(s,t), I₂(s,t)) with the convolution of path and inverse point spread function (S*IPSF) providing a convolved image

e. means for adding the convolved images for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camarea position signal (C₀).

20. System of as claimed in claim 15, wherein the means for adding and inverse filtering comprises:

a. means for generating a projected path (S) of the image shift for each image capturing device of the array (C₁, C₂, C_K, C_N), the path being determined with respect to the virtual camera position (u₀,v₀), the projected path (S) being the sum of translating impulse response functions for aligning the image of a image capturing device with the virtual camera position (u₀,v₀) at a number of depths

b. means for convolving the individual camera images (I₁, I₂, . . . ) with their respective projected path (S)

c. means for convolving the image of each image capturing device with a highpass filter (HP) perpendicular to the projected path,

d. means for adding the convolved images for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camarea position signal (C₀).

21. System as claimed in claim 15 wherein the means for adding and inverse filtering comprises:

a. means for generating a projected path (S) of the image shift for each image capturing device of the array, the path (S) being determined with respect to the virtual camera position (u₀,v₀), the projected path (S) being the sum of translating impulse response functions for aligning the image of a image capturing device with the virtual camera position (u₀,v₀) at a number of depths

b. means for generating a convolution filter for each image by convolving its respective projected path (S) with a highpass filter (S*HP) perpendicular to it

c. means for convolving the individual input images (I₁(s,t), I₂(s,t)) with their respective filter as generated in b

d. means for adding the convolved images for producing the image data for the virtual output image (Ī

₀ (s,t)) at the vertical camera position (u₀,v₀) provided by the input virtual camarea position signal (C₀).

22-24. -24. (canceled)