Apparatus and method for providing three dimensional media content

• US 9,674,506 B2
• Filed: 09/03/2015
• Issued: 06/06/2017
• Est. Priority Date: 08/12/2010
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Zhang, Yifu, Tian, Chao, Vaishampayan, Vinay Anant
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/844,412
• Publication Number:
  US 20150381966A1
• Time in Patent Office: 642 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A system that incorporates teachings of the exemplary embodiments may include, for example, means for generating a disparity map based on a depth map, means for determining accuracy of pixels in the depth map where the determining means identifies the pixels as either accurate or inaccurate based on a confidence map and the disparity map, and means for providing an adjusted depth map where the providing means adjusts inaccurate pixels of the depth map using a cost function associated with the inaccurate pixels. Other embodiments are disclosed.

Claims

1. A server comprising:

a processor; and

a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations comprising;

receiving from a camera system a depth map based on video data captured by the camera system, the video data comprising a plurality of pixels;

generating a disparity map based on the depth map;

receiving from the camera system a confidence map, wherein the confidence map is based on a confidence in an accuracy of the depth map for each pixel of the plurality of pixels; and

determining an accuracy of the pixels in the depth map by identifying pixels as either accurate or inaccurate based on the confidence map and the disparity map.

2. The server of claim 1, wherein inaccurate pixels are identified by measuring a disparity variance for a pre-determined region surrounding each pixel of the disparity map.

3. The server of claim 1, wherein the operations further comprise:

adjusting inaccurate pixels of the depth map to generate an adjusted depth map; and

generating three-dimensional media content based on the video data and the adjusted depth map.

4. The server of claim 3, wherein the inaccurate pixels in the depth map are adjusted according to a graph-cut analysis.

5. The server of claim 3, wherein the inaccurate pixels in the depth map are adjusted according to a belief-propagation analysis.

6. The server of claim 3, wherein the inaccurate pixels in the depth map are adjusted according to a cost function.

7. The server of claim 1,

wherein the camera system comprises a range camera and a vision camera, and

wherein the depth map and the confidence map are received from the range camera.

8. The server of claim 7, wherein the range camera has a resolution lower than that of the vision camera, and wherein the operations further comprise:

receiving a depth image from the range camera;

receiving a vision image from the vision camera; and

mapping the depth image to the vision image.

9. The server of claim 7, wherein the operations further comprise:

receiving an intensity map from the range camera; and

performing a calibration procedure for the camera system based on the intensity map.

10. The server of claim 7, wherein the video data comprises color frames received from the vision camera, and grey-scale frames received from the range camera.

11. A method comprising:

receiving, by a server comprising a processor, from a camera system a depth map based on video data captured by the camera system, the video data comprising a plurality of pixels;

generating, by the server, a disparity map based on the depth map;

receiving, by the server, from the camera system a confidence map, wherein the confidence map is based on a confidence in an accuracy of the depth map for each pixel of the plurality of pixels;

determining, by the server, an accuracy of the pixels in the depth map by identifying pixels as either accurate or inaccurate based on the confidence map and the disparity map; and

adjusting, by the server, inaccurate pixels of the depth map to generate an adjusted depth map.

12. The method of claim 11, further comprising generating three-dimensional media content based on the video data and the adjusted depth map.

13. The method of claim 11,

wherein the camera system comprises a range camera and a vision camera, and

wherein the depth map and the confidence map are received from the range camera.

14. The method of claim 11, wherein the inaccurate pixels are identified by measuring a disparity variance for a pre-determined region surrounding each pixel of the disparity map.

15. The method of claim 11, wherein the inaccurate pixels in the depth map are adjusted according to a cost function.

16. A device, comprising:

a processor; and

a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising;

receiving from a camera system a depth map based on video data captured by the camera system, the video data comprising a plurality of pixels, wherein the camera system comprises a range camera and a vision camera;

generating a disparity map based on the depth map;

receiving from the camera system a confidence map, wherein the confidence map is based on a confidence in an accuracy of the depth map for each pixel of the plurality of pixels, wherein the depth map and the confidence map are received from the range camera; and

determining an accuracy of the pixels in the depth map by identifying pixels as either accurate or inaccurate based on the confidence map and the disparity map.

17. The device of claim 16, wherein the operations further comprise:

adjusting inaccurate pixels of the depth map to generate an adjusted depth map; and

generating three-dimensional media content based on the video data and the adjusted depth map.

18. The device of claim 16, wherein inaccurate pixels are identified by measuring a disparity variance for a pre-determined region surrounding each pixel of the disparity map.

19. The device of claim 16, wherein inaccurate pixels in the depth map are adjusted according to a cost function.

20. The device of claim 16, wherein the video data comprises color frames received from the vision camera, and grey-scale frames received from the range camera.