True three-dimensional imaging and display system

• US 5,675,377 A
• Filed: 06/30/1995
• Issued: 10/07/1997
• Est. Priority Date: 06/30/1995
• Status: Expired due to Term

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Telefonaktiebolaget LM Ericsson
• Inventors: Gibas, Michael Patrick
• Primary Examiner(s): AU, AMELIA MEGAN
• Application Number:
  US08/497,643
• Time in Patent Office: 830 Days
• Field of Search: 348/42, 348/43, 348/47, 348/48, 348/46, 348/51, 348/54, 348/59, 382/154, 382/285, 359/458, 359/462, 359/463, 359/470, 359/472
• US Class Current: 348/47
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/334 using spectral multiplexing
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A scene is imaged from a plurality of points of view to generate plural images each including an image value and distance for each picture element. The images are converted to a principle point of view to generate a non-standard image that accounts for hidden objects in the scene from the perspective of the principle point of view by including multiple image values and distances for each picture element. The non-standard image is then converted to generate a plurality of display images, each having a different display point of view. The display images are individually displayed at corresponding display elements arrayed in a display panel for viewing. A lens at each display element fans out the display image and thus restricts the amount of the image visible at any given perception point to produce true three-dimensional images. An application of the system is disclosed for effectuating a virtual conferencing facility. Furthermore, disclosure is made of the generation or modification of three-dimensional images in the non-standard image format.

Claims

1. A true three-dimensional imaging and display system, comprising:

an image acquisition device including;

means for taking images of a scene from a plurality of spaced apart taken points of view, each taken image including image information for objects in the scene that are visible from the taken point of view of that image; and

means for converging the taken images to generate a non-standard image of the scene from a principle point of view, the generated non-standard image including image information for the objects imaged in the scene from any of the taken points of view but not necessarily visible from the principle point of view; and

an image display device connected to receive the non-standard image from the image acquisition device and including;

means for processing the received non-standard image to generate a plurality of output images of the scene, each generated output image being from a slightly different display point of view, including points of view different from the taken and principle points of view, and showing the imaged objects in the scene visible from only that display point of view; and

a display panel including an arrayed plurality of image displays, wherein each image display is associated with a different one of the display points of view and is operable to display the generated output image corresponding to that display point of view, and wherein each image display further includes a lens for fanning out the output image displayed by that image display.

2. The system of claim 1 wherein the means for taking comprises a plurality of stereo cameras located at the plurality of spaced apart taken points of view.

3. The system of claim 2 wherein each stereo camera comprises:

a left imaging camera for outputting a left image of the scene;

a right imaging camera for outputting a right image of the scene; and

means for processing the left and right images output by the imaging cameras to generate the taken image of the scene from the taken point of view.

4. The system of claim 3 wherein the means for processing comprises:

means for comparing the left and right images; and

means for determining from the comparison of the images a distance from the stereo camera to each object imaged in the left and right images of the scene.

5. The system of claim 1 wherein each taken image of the scene comprises a plurality of picture elements, each picture element defined by an image value for the object imaged in that picture element and a distance from the taken point of view to that imaged object.

6. The system of claim 5 wherein the means for converging comprises:

means for converting each picture element in the taken images to a corresponding picture element in the non-standard image;

means for converting the distances in the taken image to distances to the imaged object from the principle point of view; and

means for associating the converted distances to the corresponding picture elements.

7. The system as in claim 6 wherein the means for converting each picture element comprises means for identifying the picture element in the non-standard image in which the object imaged from the taken point of view would be imaged from the principle point of view.

8. The system of claim 1 wherein the non-standard image of the scene comprises a plurality of picture elements with each picture element being associated with a particular orientation of a scan ray emanating from the principle point of view, and wherein each picture element is defined by an image value and a distance from the principle point of view for each imaged object in the scene located along the scan ray associated with that picture element.

9. The system of claim 8 wherein the means for processing the received non-standard image comprises:

means for converting the distances in the non-standard image to distances to the imaged objects from each of the display points of view;

means for converting each picture element in the non-standard image to a corresponding picture element in each of the output images; and

means for associating the converted distances to the corresponding picture elements, the closest converted distances for each picture element identifying the objects in the scene visible from the display point of view.

10. The system of claim 1 wherein each output image comprises a plurality of picture elements, each picture element defined by an image value for an object displayed by that picture element.

11. The system as in claim 10 wherein the image display comprises a plurality of laser sources corresponding to the plurality of picture elements in the output image, the laser sources emitting a display ray having the image value, the emitted display ray refracted by the lens.

12. The system as in claim 11 wherein each laser source comprises a sub-array of mini-laser sources.

13. The system as in claim 11 wherein a surface of the lens further includes a plurality of mini-lenses thereon for spreading the fanned display rays emitted from the laser sources.

14. The system as in claim 1 wherein the fanning out of the output image by the lens restricts the amount of the output image that is visible at each display element to a viewer of the display panel from a particular point of view with respect to the display panel, the slightly different output images displayed at the plurality of display elements inducing viewer parallax and three-dimensional perception of the displayed image of the scene.

15. The system as in claim 1 further including an image storage device for storing images output by the image acquisition device.

16. The system as in claim 15 wherein the image storage device further includes an image processing unit for manipulating images output by the image acquisition device.

17. The system as in claim 1 wherein the means for taking images comprises means for scanning a three-dimensional space comprising the scene to acquire image information for the included objects.

18. A three-dimensional image processing system, comprising:

means for generating a true three-dimensional non-standard image of a scene comprising a plurality of picture elements with each picture element defined by an image value and a distance from a principle point of view to each object in the scene located along a scan ray associated with that picture element;

means for processing the image values and distances in the generated non-standard image of the scene to generate a plurality of output images of the scene from slightly different display points of view, each output image displaying the scene and the objects visible from the display point of view of the output image; and

means for displaying the plurality of output images for true three-dimensional perception of the scene.

19. The system of claim 18 wherein the means for generating comprises means for scanning in raster fashion a three-dimensional space comprising the scene.

20. The system as in claim 18 wherein the means for generating the non-standard image comprises an image acquisition device, comprising:

means for taking images of a scene from a plurality of spaced apart taken points of view; and

means for composing the taken images to the principle point of view to produce the non-standard image of the scene from the principle point of view.

21. The system as in claim 20 wherein the means for taking comprises a plurality of spaced apart stereo cameras.

22. The system as in claim 20 wherein the means for composing comprises:

means for converting image values and distances to objects from the taken points of view to image values and distances to objects from the principle point of view; and

means for associating the converted image values and distances for objects in the scene located along the scan rays with the picture element in the non-standard image associated with that scan ray.

23. The system as in claim 18 wherein the means for generating a non-standard image comprises an image generation device for creating and outputting the non-standard image of the scene from the principle point of view.

24. The system as in claim 23 wherein the means for generating further comprises means for manipulating previously acquired images to generate and output the non-standard image.

25. The system as in claim 18 wherein each output image generated by the means for processing comprises a plurality of picture elements with each picture element defined by an image value and a distance from the display point of view to the nearest object in the scene.

26. The system as in claim 19 wherein the means for processing comprises:

means for converting the image values and distances from the principle point of view to image values and distances from each of the display points of view; and

means for associating the converted image values and distances for objects in the scene with picture elements in the output images.

27. The system as in claim 18 wherein the means for displaying comprises:

a display panel; and

an arrayed plurality of display elements on the display panel, each display element displaying one of the plurality of output images.

28. The system as in claim 27 wherein each display element comprises:

a display device for displaying an output image; and

a lens for fanning out the displayed output image, the lens accordingly restricting the amount of the output image displayed by the display device that is visible to a viewer from a given viewing point of view with respect to the display panel.

29. The system as in claim 28 wherein the display device comprises an arrayed plurality of laser sources, one laser source for each picture element in the output image received by the display device, each laser source emitting a display ray having the image value of that picture element.

30. The system as in claim 29 wherein each laser source comprises an arrayed plurality of mini-laser sources.

31. The system as in claim 28 wherein each lenses includes a plurality of mini-lenses positioned on a surface of the lens.

32. A method for true three-dimensional imaging of a scene including a plurality of objects, comprising the steps of:

taking a plurality of images of the scene and its included objects from a plurality of spaced apart taken points of view;

converting the plurality of taken images to a principle point of view; and

combining the converted images to generate a non-standard image of the scene and its included objects from the principle point of view, the non-standard image including image information for the imaged objects in the scene including those objects not necessarily visible from the principle point of view.

33. The method of claim 32 wherein the step of taking comprises the step of stereo imaging the scene from each of the taken points of view to determine image information for the objects in the scene including distances from the taken points of view to each imaged object.

34. The method as in claim 33 wherein the step of converting comprises the step of converting the distances from the taken points of view to each imaged object to distances from the principle point of view to each imaged object.

35. The method as in claim 34 wherein the step of combining comprises the step of associating, for each imaged object, the imaged object and the converted distance to that object from the principle point of view with a picture element in the non-standard image.

36. The method as in claim 35 wherein each picture element in the non-standard image is associated with a particular orientation of a scan ray emanating from the principle point of view, and the step of associating comprises the step of associating the image information for each imaged object located along the scan ray with the picture element associated with that scan ray.

37. A method for true three-dimensional display of an image of a scene including a plurality of objects, the image being from a principle point of view and including image information for the objects both visible and hidden from the principle point of view, comprising the steps of:

converting the image from the principle point of view to generate a plurality of output images of the scene, each output image being from a slightly different display point of view;

simultaneously displaying the plurality of output images of the scene; and

restricting viewer perception of portions of each of the displayed plurality of output images to induce parallax in the displayed scene and true three-dimensional perception of the objects therein.

38. The method as in claim 37 wherein the step of converting comprises the step of converting the image information from the principle point of view to image information from each of the display points of view.

39. The method as in claim 37 wherein the step of simultaneously displaying comprises the step of displaying the output images in an arrayed fashion.

40. The method as in claim 37 wherein the step of restricting comprises the step of fanning out through a lens each of the displayed output images.

41. A true three-dimension image acquisition device, comprising:

a set of cameras for taking images of a scene from a plurality of spaced apart taken points of view, each taken image including image information for objects in the scene that are visible from the taken point of view of that image; and

an image processor for converging the taken images to generate a non-standard image of the scene from a principle point of view, the generated non-standard image including image information for the objects imaged in the scene from any of the taken points of view but not necessarily visible from the principle point of view.

42. The device of claim 41 wherein the set of cameras comprise a plurality of stereoscopic cameras for taking stereo images of the scene from the taken points of view.

43. The device of claim 42 wherein each stereoscopic camera comprises:

a left camera;

a right camera; and

means for processing images obtained from the left and right cameras to determine distances to each object viewed by the left and right cameras and output the stereo image of the scene comprising a plurality of picture elements, each picture element defined by an image value and the determined distance.

44. The device of claim 41 wherein the generated non-standard image output by the image processor comprises a plurality of picture elements, each picture element defined by an image value and a distance value to each object in the scene located along a scan ray emanating from the principle point of view whose orientation corresponds to that picture element.

45. The device of claim 41 wherein the taken image comprises a plurality of picture elements, each picture element defined by an image value and distance value to a closest object in the scene visible from the taken point of view, and wherein the image processor includes:

means for converting the picture element of the taken image to a corresponding picture element of the non-standard image;

means for converting the distance value from the taken point of view to a new distance value from the principle point of view; and

means for associating the image value of new distance value for picture element of the taken image to its corresponding picture element of the non-standard image.

46. A true three-dimension image display device, comprising:

an image processor for converting a non-standard image of a scene including a plurality of picture elements wherein each picture element includes image information for objects within the scene that are both visible and not visible from a principle point of view to generate a plurality of output images of the scene, each generated output image being from a slightly different display point of view and including a plurality of picture elements wherein each picture element includes image information for objects in the scene visible from only that display point of view; and

a display panel including an arrayed plurality of image displays, wherein each image display is associated with a different display point of view and is operable to display the generated output image corresponding to that display point of view, and wherein each image display further includes a lens for fanning out the output image displayed by that image display.

47. The device of claim 46 wherein each image display comprises a plurality of laser sources corresponding to the picture elements in the output image, the laser sources outputting a display ray in accordance with the image information of the corresponding picture element of the output image.

48. The device of claim 47 wherein each laser source comprises an arrayed plurality of mini-laser sources.

49. The device of claim 46 wherein the fanning out of the output image by the lens restricts the amount of the output image that is visible at each display element to a viewer of the display panel from a particular point of view with respect to the display panel, the slightly different output images displayed at the plurality of display elements inducing viewer parallax and three-dimensional perception of the displayed image of the scene.

50. The device of claim 46 wherein the lens further comprises a plurality of mini-lenses on a surface of the lens.