Data processing for three-dimensional displays

• US 7,525,541 B2
• Filed: 03/16/2005
• Issued: 04/28/2009
• Est. Priority Date: 04/05/2004
• Status: Active Grant

• Current Assignee: Gula Consulting Limited Liability Company (Intellectual Ventures LLC)
• Original Assignee: Actuality Systems, Inc. (Ellis Amalgamated LLC)
• Inventors: Cossairt, Oliver Strider, Chun, Won
• Primary Examiner(s): Sajous, Wesner
• Assistant Examiner(s): MATHEWS, CRYSTAL ANN
• Application Number:
  US11/082,169
• Publication Number:
  US 20050230641A1
• Time in Patent Office: 1,504 Days
• Field of Search: 345/419, 345 30- 32, 345/108, 345/110
• US Class Current: 345/419
• CPC Class Codes: G03H 2210/454 into planes
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/351 for displaying simultaneously
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/365 using digital micromirror devices [DMD]

Abstract

Generating a three-dimensional image of a three-dimensional scene by generating projection data and projecting light rays based on the projection data to generate the three-dimensional image. Sampling of the data is based on at least one physical parameter associated with the projection of light rays by a projection system used to project the light rays.

Claims

1. An image data processing apparatus comprising:

a storage to store a set of image data representing a three-dimensional scene;

a data processor to sample data points from the set of image data representing the three-dimensional scene to generate projection data, the sampling based on the position of an image surface relative to the position of a presentation surface; and

a projection device to project modulated light rays from the presentation surface onto the image surface based on the projection data and generating a three-dimensional image that is representative of the three-dimensional scene;

wherein the data processor samples different data points from the same set of image data responsive to changes in the position of the image surface relative to the presentation surface to generate projection data that can be used to generate a three-dimensional image that is representative of the three-dimensional scene.

2. The image data processing apparatus of claim 1, in which the projection device projects light rays that are modulated by a spatial light modulator, and the light rays are focused on the positions from which the light rays appear to be projected.

3. The image data processing apparatus of claim 2, in which the spatial light modulator modulates the light rays based on the projection data.

4. The image data processing apparatus of claim 1, in which the image surface diffuses the light rays only along a specified direction.

5. The image data processing apparatus of claim 1, in which the three-dimensional scene comprises at least one of a collection of points, a collection of lines, a collection of surfaces, and a collection of volumes.

6. The image data processing apparatus of claim 1, in which the data processor generates projection data that specify at least one of color and brightness of light rays that propagate along specified paths relative to the image surface.

7. The method image data processing apparatus of claim 1, in which the data representing the three-dimensional scene comprises information about observations of the three-dimensional scene from different positions.

8. The image data processing apparatus of claim 7, in which the data representing the three-dimensional scene comprises images of the three-dimensional scene captured by at least one camera from different positions.

9. The image data processing apparatus of claim 8, in which the camera has a pixel resolution that is different from a pixel resolution of a spatial light modulator used to modulate the projected light rays.

10. The image data processing apparatus of claim 8, in which the data processor samples data to compensate for distortions in the images.

11. The image data processing apparatus of claim 7, in which the data representing the three-dimensional scene comprises synthesized images of the three-dimensional scene as viewed from different positions.

12. The image data processing apparatus of claim 11, in which the synthesized images each has a pixel resolution that is different from a pixel resolution of a spatial light modulator used to modulate the projected light rays.

13. The image data processing apparatus of claim 1, in which the projection device projects a first set of modulated light rays from a first position during a first interval, and projects a second set of modulated light rays from a second position during a second interval.

14. The image data processing apparatus of claim 1, in which the three-dimensional image comprises an image of at least one of an accurate copy, an approximation, a closer view, and a farther view of the three-dimensional scene.

15. The image data processing apparatus of claim 1, in which the data representing the three-dimensional scene comprises information about at least one of color and brightness of light rays emanating from the three-dimensional scene.

16. The image data processing apparatus of claim 1 in which the data processor generates the projection data by sampling data from image data representing two-dimensional images of the three-dimensional scene as viewed from various locations, each set of projection data corresponding to one of multiple projection positions, each set of projection data comprising data sampled from multiple two-dimensional images, the sets of projection data having information representing light rays that, when projected from respective multiple projection positions, generate the three-dimensional image representative of the three-dimensional scene.

17. The image data processing apparatus of claim 1 in which the data processor samples a view volume that comprises data representing images of the three-dimensional scene as viewed from various locations, to generate sets of projection data, each set of projection data corresponding to one of multiple projection positions, each set of projection data comprising data sampled from at least two images.

18. The image data processing apparatus of claim 1 in which the data processor generates a view volume that comprises data representing observations of a three-dimensional scene from different positions and

samples the view volume to generate a first set of projection data, the sampling based on at least a position of a first image surface relative to a position of a first presentation surface;

a spatial light modulator modulates light rays based on the first set of projection data to generate a first set of presentation images at the first presentation surface;

the projection device projects the first set of presentation images from the first presentation surface toward the first image surface to reproduce a first three-dimensional image that is representative of the three-dimensional scene;

the data processor samples the same view volume to generate a second set of projection data, the sampling based on at least a position of a second image surface relative to a position of a second presentation surface;

the spatial light modulator modulates light rays based on the second set of projection data to generate presentation images at the second presentation surface; and

the projection device projects the presentation images from the second presentation surface toward the second image surface to reproduce a second three-dimensional image that is representative of the three-dimensional scene;

wherein at least one of (i) the first image surface being different from the second image surface, and (ii) the first presentation surface being different from the second presentation surface.

19. The image data processing apparatus of claim 1 in which the data processor sends at least one physical parameter of the image data processing apparatus to a server having data representing a three-dimensional scene;

the data processor receives projection data from the server, the projection data derived by sampling the data representing the three-dimensional scene based on the at least one physical parameter of the image data processing apparatus, wherein for different image data processing apparatuses having different physical parameters, the server samples different data points from the same data representing the three-dimensional scene based on the physical parameters of the image data processing apparatuses to generate different projection data

the projection device projects light rays based on the projection data to generate a three-dimensional image of the three-dimensional scene.

20. The image data processing apparatus of claim 19,

in which the data processor sends at least one physical parameter of the image data processing apparatus to a second server having data representing a second three-dimensional scene;

the data processor receives a second set of projection data from the second server, the second set of projection data derived by sampling the data representing the second three-dimensional scene based on the at least one physical parameter; and

the projection device projects light rays based on the projection data to generate a three-dimensional image of the second three-dimensional scene.

21. The image data processing apparatus of claim 1 in which the data processor receives at least one physical parameter of a first projection system;

the data processor generates a first set of projection data by sampling the data representing the three-dimensional scene based on the at least one physical parameter of the first projection system;

the data processor transmits the first set of projection data to the first projection system to enable the first projection system to project light rays based on the first set of projection data to generate a three-dimensional image of the three-dimensional scene;

the data processor receives at least one physical parameter of a second projection system;

the data processor generates a second set of projection data by sampling the same data representing the three-dimensional scene based on the at least one physical parameter of the second projection system; and

the data processor transmits the second set of projection data to the second projection system to enable the second projection system to project light rays based on the projection data to generate a three-dimensional image of the three-dimensional scene.

22. The image data processing apparatus of claim 1 in which the data processor generates a stream of data representing observations of a three-dimensional scene from a set of positions;

at a given time, the data processor stores a subset of the stream of data in the storage, the subset representing observations of the three-dimensional scene from a subset of the positions;

the data processor generates a stream of projection data by sampling the stored subset of the stream of data; and

the projection device projects light rays based on the stream of projection data to generate a three-dimensional image of scene.

23. The image data processing apparatus of claim 22, in which the projection device defines the image surface onto which the light rays are projected.

24. The image data processing apparatus of claim 23, in which the data processor samples the stored subset of the stream of data based on at least one of:

(a) a position of the image surface relative to positions from which the light rays appear to be projected,

(b) a position of the image surface relative to intended observation positions for observing the three-dimensional image,

(c) a size of the image surface,

(d) an extent of positions from which the light rays appear to be projected, and

(e) a shape of the image surface.

25. The image data processing apparatus of claim 22, in which the stream of data representing observations of the scene ends after data representing observations of the scene from all of the set of positions have been generated.

26. The image data processing apparatus of claim 22, in which the stream of data representing observations of the scene repeats itself after data representing observations of the scene from all of the set of positions have been generated.

27. The image data processing apparatus of claim 1, in which

the data processor generates sets of projection data by sampling data from image data representing two-dimensional images of a three-dimensional scene as viewed from various locations, each set of projection data corresponding to one of multiple projection positions, each set of projection data comprising data sampled from multiple two-dimensional images;

and

the projection device projects light rays based on the sets of projection data from respective multiple projection positions to generate a three-dimensional image that is representative of the three-dimensional scene.

28. The apparatus of claim 1, in which the projection device defines the image surface onto which the light rays are projected.

29. The apparatus of claim 1, in which the data processor samples the data also based on at least one of (a) a size of the image surface, (b) a shape of the image surface, and (c) a position of the image surface relative to intended observation positions for observing the three-dimensional image.

30. The apparatus of claim 1, in which the projection device comprises a spatial light modulator that modulates light rays according to the projection data.

31. The apparatus of claim 30, in which the projection device defines projection positions from which light rays modulated by the spatial light modulator appear to be projected.

32. The apparatus of claim 31, in which the data processor samples the data based on at least one of (a) a position of the image surface relative to the projection positions, and (b) a farthest distance between any two projection positions.

33. The apparatus of claim 31, in which the spatial light modulator modulates light rays to generate different presentation images at different times according to the projection data, and the projection device projects the presentation images from different projection positions in a time-division-multiplexed manner.

34. The apparatus of claim 1 in which the set of image data is represented by a view volume in a ray space having a coordinate system in which coordinates on a first axis represent coordinates on an image surface and coordinates on a second axis represent coordinates on a capture surface, the view volume comprising image slices representing two-dimensional images of the three-dimensional scene as viewed from various locations on the capture surface.

35. The apparatus of claim 34 in which the data processor samples data points from the set of image data based on one or more calibration surfaces that intersect the view volume.

36. The apparatus of claim 35 in which the calibration surfaces are skewed with respect to the image slices by an amount dependent on at least one of (i) relative positions of the image surface and the presentation surface, and (ii) relative sizes of the image surface and an intended observation surface.

37. The apparatus of claim 35 in which the calibration surfaces are curved.

38. The server of claim 34 wherein the data processor samples data points from the set of image data based on one or more calibration surfaces that intersect the view volume.

39. An image data processing server comprising:

a storage to store a set of image data representing a three-dimensional scene;

a data processor to

receive information about geometries of presentation surfaces and image surfaces of various client projection devices,

for each client projection device, generate a set of projection data by sampling data points from the set of image data representing the three-dimensional scene based on the information about the geometry of the presentation surface and the image surface of the client projection device,

for different client project devices having different geometries of presentation and image surfaces, sample different data points from the same set of image data, and

transmit each set of projection data to the corresponding client projection device to enable each client projection device to project light rays based on the corresponding set of projection data to generate a three-dimensional image of the three-dimensional scene.

40. The server of claim 39 wherein the set of image data is represented by a view volume in a ray space having a coordinate system in which coordinates on a first axis represent coordinates on an image surface and coordinates on a second axis represent coordinates on a capture surface, the view volume comprising image slices representing two-dimensional images of the three-dimensional scene as viewed from various locations on the capture surface.