Method and apparatus for multiview image generation using depth map information

• US 8,773,427 B2
• Filed: 12/22/2010
• Issued: 07/08/2014
• Est. Priority Date: 12/22/2010
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Bae, Soo Hyun, Berestov, Alexander, Baqai, Farhan, Inaba, Seijiro, Kosakai, Ryota, Katoh, Naoya
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US12/976,944
• Publication Number:
  US 20120162193A1
• Time in Patent Office: 1,294 Days
• Field of Search: 345/419, 345/420, 345/422, 382/285
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307

Abstract

A method and apparatus for multiview image generation using depth map information is described. In one embodiment, a computer-implemented method comprises converting a input image and an input depth map into a projected image and a projected depth map using values from physical pixel locations that map to projected pixel locations, wherein the projected image and the projected depth map are associated with a particular view of the input image, inpainting the projected image and the projected depth map and producing an output image in a direction of the particular view using the inpainted projected image and the inpainted projected depth map.

Claims

1. A computer-implemented method for multi-view image generation using depth map information, the method being executed by one or more processors and comprising:

transforming an input image and an input depth map into a new view that is distinct from a view of the input image and the input depth map;

converting a input image and an input depth map into a projected image and a projected depth map using values from physical pixel locations that map to projected pixel locations, wherein the projected image and the projected depth map are associated with the new view;

identifying one or more occluded regions in the input image and the input depth map;

inpainting the projected image and the projected depth map and inpainting the one or more occluded regions with pixels associated with locations to the right of one or more occluded regions; and

producing an output image in a direction of the particular view using the inpainted projected image and the inpainted projected depth map.

2. method of claim 1, wherein converting the input image and the input depth map further comprising:

performing a forward projection from the transformed input image and the transformed input depth map onto the projected image and the projected depth map.

3. The method of claim 1, wherein converting the input image and the input depth map further comprising:

for each pixel in the input image;

determining a physical location in the input image, and

mapping the physical location to a projected location in the projected image.

4. The method of claim 3 further comprising sorting the projected pixel locations in the projected image.

5. The method of claim 3 further comprising for at least one pixel in the projected image, computing a value using at least two projected locations in the projected image, wherein the value comprises at least one of an intensity or a depth.

6. The method of claim 5, wherein computing the value further comprises interpolating at least two values associated with at least two pixels in the input image that map to the at least two projected locations.

7. The method of claim 6, wherein the at least two values comprise at least one of a value associated with a closest projected location, a value associated with a projected location having a lower depth or a value associated with a projected location having a minimum depth within a horizontal display area having a size of at least one pixel.

8. The method of claim 1, wherein inpainting the projected image and the projected depth map further comprises for each occluded region in the projected depth map, copying at least one value from at least one source region within the input depth map, wherein the at least one source region maps to at least one portion of the each occluded region.

9. The method of claim 8, wherein the at least one source region maps to a right most portion of the each occluded region.

10. The method of claim 1, wherein inpainting the projected image and the projected depth map further comprises for each occluded region in the projected image, processing at least one value from at least source region within the input image, wherein the at least one source region maps to at least a portion of the each occluded region.

11. The method of claim 1, wherein producing the output image further comprises blending at least two inpainted images.

12. The method of claim 1, wherein converting the input image and the input depth map further comprises identifying at least one occluded region within the projected image and the projected depth map.

13. An apparatus for multi-view generation of an image using depth map information, comprising:

a lens module for capturing an input image; and

a processor having modules comprising;

depth map storage comprising an input depth map; and

an image processor for transforming an input image and an input depth map into a new view that is distinct from a view of the input image and the input depth map, converting the input image and the input depth map into a projected image and a projected depth map using values from physical pixel locations that map to projected pixel locations, wherein the projected image and the projected depth map are associated with the new view, identifying one or more occluded regions in the input image and the input depth map, inpainting the projected image and the projected depth map and inpainting the one or more occluded regions with pixels associated with locations to the right of one or more occluded regions and producing an output image that forms the view using the inpainted projected image and the inpainted projected depth map.

14. The apparatus of claim 13, wherein the image processor further comprises:

a depth map inpainting module for copying values from at least one source region within the input depth map for each target region in the projected depth map, wherein the at least one source region corresponds with the projected pixel locations within the transformed input depth map for the each target region;

an image inpainting module for copying values from at least one source region within the transformed input image for each target region in a copy of the projected image, wherein each source region maps to pixels that are located right of the each target region and inpainting occluded regions within another copy of the projected image using the inpainted transformed input depth map and texture information; and

an output module for blending the inpainted copy of the projected image and the other inpainted copy of the projected image.

15. The apparatus of claim 13, wherein the convertor, for each pixel in the input image, determines a physical location and maps the physical location to a projected location in the projected image.

16. The apparatus of claim 15, wherein the convertor, for each pixel in the projected image, computes a depth value and an intensity value using at least two projected locations for at least two pixels in the input image.

17. The apparatus of claim 16, wherein the convertor interpolates at least two values associated with the at least two pixels, wherein the at least two pixels comprise at least one of a pixel associated with a closest projected location, a pixel associated with a projected location having a lowest depth value amongst the at least two pixels or a pixel associated with a projected location having a lowest depth within a horizontal display area having a size of at least one pixel.

18. A non-transitory computer readable storage medium comprising one or more processor executable instructions that, when executed by at least one processor, causes the at least one processor to perform a method comprising:

transforming an input image and an input depth map into a new view that is distinct from a view of the input image and the input depth map;

converting a input image and an input depth map into a projected image and a projected depth map using values from physical pixel locations that map to projected pixel locations, wherein the projected image and the projected depth map are associated with the new view;

identifying one or more occluded regions in the input image and the input depth map;

inpainting the projected image and the projected depth map and inpaintinq the one or more occluded regions with pixels associated with locations to the right of one or more occluded regions; and

producing an output image in a direction of the particular view using the inpainted projected image and the inpainted projected depth map.

19. The computer-readable-storage medium of claim 18, wherein the one or more processor executable instructions perform the method further comprising:

performing a forward projection from the transformed input image and the transformed input depth map onto the projected image and the projected depth map.

20. The computer-readable-storage medium of claim 18, wherein the one or more processor executable instructions perform the method further comprising:

for each pixel in the input image;

determining a physical location, and

mapping the physical location to a projected location in the projected image.