2D to 3D image conversion

• US 20070279415A1
• Filed: 06/01/2006
• Published: 12/06/2007
• Est. Priority Date: 06/01/2006
• Status: Active Grant

• Current Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Original Assignee: Industrial Light & Magic (The Walt Disney Company)
• Inventors: Sullivan, Steve, Trombla, Alan D., Callari, Francesco G.
• Granted Patent: US 7,573,475 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A method of creating a complementary stereoscopic image pair is described. The method includes receiving a first 2D image comprising image data, where the first 2D image is captured from a first camera location. The method also includes projecting at least a portion of the first 2D image onto computer-generated geometry. The image data has depth values associated with the computer-generated geometry. The system includes rendering, using the computer-generated geometry and a second camera location that differs from the first camera location, a second 2-D image that is stereoscopically complementary to the first 2-D image, and infilling image data that is absent from the second 2-D image.

Claims

1-28. -28. (canceled)

29. A method of identifying an absence of image information when generating a complementary stereoscopic image pair, the method comprising:

projecting at least a portion of a first 2D image comprising image information onto computer-generated geometry from a first camera having a first viewing frustum at a first camera location;

designating a second camera having a second viewing frustum at a second camera location, wherein the second viewing frustum includes at least a portion of the computer- generated geometry;

determining depth values associated with the image information in the first viewing frustum and in the second viewing frustum, wherein each of the depth values indicates a distance between one of the cameras and a point in the projected first 2D image in the viewing frustum for the associated camera; and

identifying an absence of valid image information in a stereoscopic image corresponding to the first 2D image by comparing the depth values for corresponding points of the projected first 2D image in the first and second viewing frustums.

30. The method of claim 29, wherein the image information comprises one or more pixels or a portion of a continuous color field specified by coordinates.

31. The method of claim 29, further comprising identifying an absence of valid image information if a point in the second viewing frustum does not have a corresponding image information in the first viewing frustum.

32. The method of claim 29, further comprising rendering, using the computer- generated geometry, the second camera location, and the second viewing frustum, a second 2D image that is stereoscopically complementary to the first 2D image.

33. The method of claim 29, wherein the comparison of the depth values comprises determining if a difference between a depth value associated with image information from the first viewing frustum and a depth value associated with corresponding image information from the second viewing frustum exceeds a threshold.

34. The method of claim 33, wherein the absence of valid image information is identified if the threshold is exceeded.

35. The method of claim 29, further comprising receiving the computer-generated geometry.

36. The method of claim 29, further comprising generating the computer-generated geometry based on the received first 2D image.

37. The method of claim 29, further comprising generating the computer-generated geometry based on physical models of objects depicted in the first 2D image.

38. The method of claim 29, wherein the first 2D image comprises a final image of a motion picture.

39. The method of claim 29, further comprising deriving the first camera location from first camera information.

40. The method of claim 39, further comprising receiving the first camera information selected from a group consisting of aspect information, field-of view information, and information associated with objects included in the first 2D image.

41. The method of claim 29, wherein the second camera location is horizontally offset from the first camera location.

42. The method of claim 29, wherein the depth values further comprise background identifiers that indicate associated image information has a depth value greater than a background threshold.

43. The method of claim 29, further comprising receiving non-rendered elements.

44. The method of claim 43, further comprising receiving a compositing script comprising instructions used to combine non-rendered elements and the geometry in the rendering.

45. A method comprising:

projecting at least a portion of a first 2D image comprising image information onto computer-generated geometry from a first camera having a first viewing frustum at a first camera location; and

generating, at a second camera location, a second camera having a second viewing frustum that includes at least a portion of the computer-generated geometry, wherein optical axes of the first and second cameras are substantially parallel.

46. The method of claim 45, further compromising generating a stereoscopic pair comprising the first 2D image and a second 2D image based on image information included in the second viewing frustum of the second camera.

47. The method of claim 45, further comprising generating a third camera having a third viewing frustum at a third camera location, wherein the third viewing frustum includes at least a portion of the computer-generated geometry, wherein optical axes of the first and third camera are not substantially parallel.

48. The method of claim 47, wherein the second camera is generated based on the third camera.

49. The method of claim 47, further comprising computing homography between image planes associated with the second and third cameras.

50. The method of claim 49, further comprising substituting the third camera with the second camera for use in rendering a second 2D image.

51. The method of claim 50, further comprising using the homography to map offsets associated with the second camera that were generated in the rendering to offsets associated with the third camera.

52. A system for identifying an absence of valid pixel data in a stereoscopically complementary image comprising:

an interface to receive a first 2D image captured from a first camera position;

a projector to project a portion of the first 2D image onto computer-generated geometry that corresponds to the portion;

a rendering module to render from a second camera position a second 2D image that is stereoscopically complementary to the first 2D image using the projected portion and the computer-generated geometry; and

a depth calculator to determine depths of image information in the first and second images for use in identifying an absence of valid image infonnation in the second 2D image.

53. A computer program product tangibly embodied in an information carrier, the computer program product including instructions that, when executed, perform operations comprising:

aligning computer-generated geometry with a corresponding portion of a first 2D image captured from a first camera location;

projecting the portion of the first 2D image onto the geometry;

designating a second camera location that differs from the first camera location;

using the geometry and the designated second camera location to render a second 2D image that is stereoscopically complementary to the first 2D image; and

identifying an absence of valid image information in the second 2D image by comparing depth values of image information from the first and second 2D images.

54. A method of identifying an absence of image infonnation when generating a complementary stereoscopic image pair, the method comprising:

illuminating at least a portion of computer-generated geometry using a simulated light source positioned at a first camera location,

designating a second camera at a second camera location having a viewing frustum that includes the at least a portion of the computer-generated geometry; and

identifying an absence of valid image information at locations having shadows produced by the simulated light when the locations are viewed from the second camera.

55. The method of claim 54, further comprising identifying an absence of valid image information at points where there is an absence of any computer-generated geometry.

56. The method of claim 54, wherein the shadows indicate the associated locations are obscured when viewed from a first camera at the first camera location.

57. The method of claim 54, further comprising determining a location is external to a view from a first camera at the first camera location if the location is not illuminated by the simulated light.

58. The method of claim 57, further comprising assigning image information associated with the location external to the view from the first camera with offset values that specify the image information is categorized as outside.

59. A method of creating a complementary stereoscopic image pair, the method comprising:

receiving a first 2D image comprising image data, the first 2D image captured from a first camera location;

projecting at least a portion of the first 2D image onto computer-generated geometry, the image data having depth values associated with the computer-generated geometry; and

rendering, using the computer-generated geometry and a second camera location that differs from the first camera location, a second 2-D image that is stereoscopically complementary to the first 2-D image.

60. The method of claim 59, further comprising infilling image data that is absent from the second 2-D image.

61. The method of claim 59, wherein the image data comprises one or more pixels or a portion of a continuous color field specified by coordinates.

62. The method of claim 59, further comprising receiving the computer-generated geometry.

63. The method of claim 59, further comprising generating the computer-generated geometry based on the received first 2D image.

64. The method of claim 59, further comprising generating the computer-generated geometry based on physical models of objects depicted in the first 2D image.

65. The method of claim 59, wherein the first 2D image comprises a final image of a motion picture.

66. The method of claim 59, further comprising deriving the first camera location from first camera information associated with the image data of the first 2D image.