System and method for 3D photography and/or analysis of 3D images and/or display of 3D images

• US 20050053274A1
• Filed: 04/19/2004
• Published: 03/10/2005
• Est. Priority Date: 04/21/2003
• Status: Abandoned Application

• Current Assignee: Haim Gadassi, Yaron Mayer
• Original Assignee: Haim Gadassi, Yaron Mayer
• Inventors: Mayer, Yaron, Gadassi, Haim
• Application Number:
  US10/827,912
• Publication Number:
  US 20050053274A1
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G03B 21/18 Fire preventing or extinguishing
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

When 3D viewing means become much more available and common, it will be very sad that the many great movies that exist today will be able to be viewed in 3D only through limited and partial software attempts to recreate the 3D info. Films today are not filmed in 3D due to various problems, and mainly since a normal stereo camera could be very problematic when filming modern films, since for example it does not behave properly when zooming in or out is used, and it can cause many problems when filming for example smaller scale models for some special effects. For example, a larger zoom requires a correspondingly larger distance between the lenses, so that for example if a car is photographed at a zoom factor of 1:10, the correct right-left disparity will be achieved only if the lenses move to an inter-ocular distance of for example 65 cm instead of the normal 6.5 cm. The present invention tries to solve the above problems by using a 3D camera which can automatically adjust in a way that solves the zoom problem, and provides a solution also for filming smaller models. The angle between the two lenses is preferably changed according to the distance and position of the object that is at the center of focus, and changing the zoom affects automatically both the distance between the lenses and their angle, since changing merely the distance without changing the convergence angle would cause the two cameras to see completely different parts of the image. The patent also shows that similar methods can be used for example for a much better stereoscopic telescope with or without a varying zoom factor. In addition, the patent shows various ways to generate efficiently a 3D knowledge of the surrounding space, which can be used also for example in robots for various purposes, and also describes a few possible improvements in 3d viewing.

Claims

1. A system for obtaining 3D images, using at least two cameras or camera parts or binoculars, which automatically takes care of achieving proper stereo separation according to distance and zoom, comprising at least one of:

a. A system for automatically increasing the separation between the two cameras or binocular lenses by the factor of the zoom, while at the same time changing the angle of conversion so that the cameras still converge correctly on the same frame of view.

b. A system for automatic computerized extrapolation of the proper parallax between the two views, so that for increasing the zoom the two cameras or binocular lenses are moved apart only part of the needed distance or not moved at all, and the computer uses the parallax information from the real two images in order to extrapolate the enlarged parallax that should be achieved, while taking into account the estimated distances.

c. A system for automatic computerized interpolation of the proper parallax between the two views, so that for reducing the zoom the two cameras or binocular lenses are kept at a larger separation, and the computer uses the parallax information from the real two images in order to interpolate the reduced parallax that should be achieved, while taking into account the estimated distances.

2. The system of claim 1 wherein said extrapolation takes into account also the calculated distances for calculating the proper occlusion, so that at least one of:

a. When there is overlap of positions closer pixels override farther pixels.

b. If moving a closer part sideways reveals a part of a farther object that was previously hidden, the newly exposed part is extrapolated by at least one of;

Copying the nearest exposed pixels of the farther object, and Taking into account also information from the movement of the cameras and/or of the objects.

3. The system of claim 1 wherein the two cameras or camera parts are moved sideways in relation to each other and at least one of the following features exists:

a. They are mounted on arms that rotate around a central point and the angels of conversion are automatically adjusted to take into account also the rotation caused by the rotation of the arms, so that at least one of the arms moves.

b. They move sideways on at least one rod and/or tracks and/or extension, so that the distance between them can be increased or decreased by moving one or both of them on the rods or tracks or extension.

c. The sideways movement is achieved by at least one of a step motor and a voice coil (linear motor).

4. The system of claim 1 wherein at least one of the following features exists:

a. The two cameras or camera parts are adapted to automatically adjust the angle between them according to the distance from the object in focus.

b. For very close images at least one of the following is done;

1. Vertical size distortions are automatically fixed by an interpolation that makes the sides of the close object smaller, and 2. The two lens converge only partially and the two image are brought closer by interpolation in away similar to the way the extrapolation is computed.

c. The system automatically finds the distance to the target object by at least one of laser, ultrasound, and other known means for finding distances, automatically adjusts the focus and the angle between the lenses according to the distance, and if zoom is used than automatically the distance between the lenses is changed and their angle is also changed again accordingly.

d. The system automatically finds the distance to the target object by at laser, and said laser is an infrared laser, so that it does not disturb the photographed people or animals and does not add a visible mark to the image itself, and at least one laser mark is used, and the two cameras or camera parts automatically also detect the at least one laser mark and use it to help the adjustment of convergence based on auto-feedback, while taking into account the expected parallax of the laser mark, based on the distance.

e. At least some additional digital comparison of the two images is done in order to further make sure that the convergence has been done correctly.

f. The zooming process is electronically controlled through discrete steps, so that each time that a new frame is taken, the zooming stops temporarily, the angle of convergence is automatically fixed, and only then the two images are taken, and then the process moves on to the next step.

g. A combination of extrapolation with actual displacement is used for increasing the zoom and at least one of;

1. First only the available physical displacement is used, and only if more displacement is needed than the automatic computerized displacement comes in-to action. 2. The extrapolation is activated at all the ranges except at minimum zoom, so that the user gets a smooth feeling of correlation between the physical movement of the two lenses and the actual zoom.

h. The interpolation or extrapolation are done at least one of;

1. While capturing the images by one or more processors coupled to the cameras, and 2. While displaying them, and parameters such as the zoom factor are saved together with the images for the later processing.

i. The extrapolation and/or the interpolation take into consideration also the previous frames, so that a new calculation is done only for pixels that have changed from the previous frames.

j. At least two mirrors and/or prisms are moved sideways and/or change their angles instead of moving the cameras.

k. For filming small models at least one of the following is done;

1. A set of miniature lenses is used that can be brought together manually to a smaller distance that represents the scale. 2. The lenses remain with the normal separation or with a separation that is only partially smaller than normal, and interpolation is used for generating the image with smaller separation.

l. When CGI (Computer generated Images) are used for special effects, two sets of images with the appropriate angle disparities according to depth are automatically created by the computer and fitted each with the appropriate set of filmed frames.

5. The system of claim 1 wherein for a screen that uses a different focal point for each pixel also the original two (or more) images for each frame are used, so that the appropriate side-views are available.

8. The system of claim 1 wherein for improved autostereoscopic 3D viewing at least one of:

a. Elongated complex lenses are coupled to a display screen, so that they direct the light from each pixel-column into intermittent expanding stripes of light-dark more efficiently, so that the light in the blocked areas is not wasted but is added to the light in the lit areas.

b. Elongated miniature triangles, more than one per each pixel column, are used, with techniques like in optic fibers, where the light is reflected internally by a core and a cladding that have in different optical refraction indexes, so that each pixel column is concentrated into the desired expanding on-off stripes of light-dark.

c. Light-emitting nano-elements are used that come out of each pixel in many directions.

d. Head tracking is used for determining if the user is in the correct right-left position, and if not then the image itself is instantly corrected by the computer by at least one of;

Switching between all the left and right pixels, and Moving the entire display left or right one pixel-column.

e. When the user is in an in-between position where each eye would view a mix of left and right images, the image can be moved along with the user also in half-pixel steps or other fractions of a pixel,

f. When the user is in an in-between-state, the elongated lenses can be moved and/or rotated a little in order to shift a little the position of the border between the right-left expanding stripes.

g. Pre-distortions are automatically added to the images, so that at least parts of the image that appear to jump out of the screen and/or images that appear to be far away will look more sharp when in fact the user focuses his eyes on the illusory position of the object.

h. Pre-distortions can be automatically added to the images on the fly, according to eye tracking that determines where the user is currently trying to focus his eyes, so that at least parts of the image that appear to jump out of the screen and/or images that appear to be far away will look more sharp when in fact the user focuses his eyes on the illusory position of the object.

9. The system of claim 8 wherein said elongated lenses are at least one of:

a. Wavy shaped elongated lenses.

b. Fresnel lenses with the desired parameters.

10. The system of claim 3 wherein at least one of the following features exist:

a. The cameras or camera parts are mounted on jibs, so that two arms are used, one for each camera.

b. The cameras or camera parts are mounted on the same jib, so that at the end of the jib there is an extension on which the cameras can move sideways.

c. The cameras or camera parts are mounted on a crane, so that at one camera is connected directly to the crane'"'"'s arm, and the other camera is connected to a sideways extension with which the cameras can be moved sideways, with or without an additional crane arm for the second camera.

d. The camera operator is shown through binoculars the correct 3D image, as transmitted by the computer.

e. Each camera has a small slit or uses other means to have a good focus at a large range of distances, so at least most of the image or the central part of the image is in focus all the time, so that the user will have less motivation to try to change the focus with his eyes when viewing the filmed scenes.

6. (Canceled).

7. (Canceled).

11. A method for obtaining 3D images, using at least two cameras or camera parts or binoculars, which automatically takes care of achieving proper stereo separation according to distance and zoom, comprising at least one of the following steps:

a. Using a system for automatically increasing the separation between the two cameras or binocular lenses by the factor of the zoom, while at the same time changing the angle of conversion so that the cameras still converge correctly on the same frame of view.

b. Using a system for automatic computerized extrapolation of the proper parallax between the two views, so that for increasing the zoom the two cameras or binocular lenses are moved apart only part of the needed distance or not moved at all, and the computer uses the parallax information from the real two images in order to extrapolate the enlarged parallax that should be achieved, while taking into account the estimated distances.

c. Using a system for automatic computerized interpolation of the proper parallax between the two views, so that for reducing the zoom the two cameras or binocular lenses are kept at a larger separation, and the computer uses the parallax information from the real two images in order to interpolate the reduced parallax that should be achieved, while taking into account the estimated distances.

12. The method of claim 11 wherein said extrapolation takes into account also the calculated distances for calculating the proper occlusion, so that at least one of:

a. When there is overlap of positions closer pixels override farther pixels.

b. If moving a closer part sideways reveals a part of a farther object that was previously hidden, the newly exposed part is extrapolated by at least one of;

Copying the nearest exposed pixels of the farther object, and Taking into account also information from the movement of the cameras and/or of the objects.

13. The method of claim 11 wherein the two cameras or camera parts are moved sideways in relation to each other and at least one of the following features exists:

a. They are mounted on arms that rotate around a central point and the angels of conversion are automatically adjusted to take into account also the rotation caused by the rotation of the arms, so that at least one of the arms moves.

b. They move sideways on at least one rod and/or tracks and/or extension, so that the distance between them can be increased or decreased by moving one or both of them on the rods or tracks or extension.

c. The sideways movement is achieved by at least one of a step motor and a voice coil (linear motor).

14. The method of claim 11 wherein at least one of the following features exists:

a. The two cameras or camera parts are adapted to automatically adjust the angle between them according to the distance from the object in focus.

b. For very close images at least one of the following is done;

1. Vertical size distortions are automatically fixed by an interpolation that makes the sides of the close object smaller, and 2. The two lens converge only partially and the two image are brought closer by interpolation in away similar to the way the extrapolation is computed.

c. The system automatically finds the distance to the target object by at least one of laser, ultrasound, and other known means for finding distances, automatically adjusts the focus and the angle between the lenses according to the distance, and if zoom is used than automatically the distance between the lenses is changed and their angle is also changed again accordingly.

d. The system automatically finds the distance to the target object by at laser, and said laser is an infrared laser, so that it does not disturb the photographed people or animals and does not add a visible mark to the image itself, and at least one laser mark is used, and the two cameras or camera parts automatically also detect the at least one laser mark and use it to help the adjustment of convergence based on auto-feedback, while taking into account the expected parallax of the laser mark, based on the distance.

e. At least some additional digital comparison of the two images is done in order to further make sure that the convergence has been done correctly.

f. The zooming process is electronically controlled through discrete steps, so that each time that a new frame is taken, the zooming stops temporarily, the angle of convergence is automatically fixed, and only then the two images are taken, and then the process moves on to the next step.

g. A combination of extrapolation with actual displacement is used for increasing the zoom and at least one of;

1. First only the available physical displacement is used, and only if more displacement is needed than the automatic computerized displacement comes in-to action. 2. The extrapolation is activated at all the ranges except at minimum zoom, so that the user gets a smooth feeling of correlation between the physical movement of the two lenses and the actual zoom.

h. The interpolation or extrapolation are done at least one of;

1. While capturing the images by one or more processors coupled to the cameras, and 2. While displaying them, and parameters such as the zoom factor are saved together with the images for the later processing.

i. The extrapolation and/or the interpolation take into consideration also the previous frames, so that a new calculation is done only for pixels that have changed from the previous frames.

j. At least two mirrors and/or prisms are moved sideways and/or change their angles instead of moving the cameras.

k. For filming small models at least one of the following is done;

1. A set of miniature lenses is used that can be brought together manually to a smaller distance that represents the scale. 2. The lenses remain with the normal separation or with a separation that is only partially smaller than normal, and interpolation is used for generating the image with smaller separation.

l. When CGI (Computer generated Images) are used for special effects, two sets of images with the appropriate angle disparities according to depth are automatically created by the computer and fitted each with the appropriate set of filmed frames.

15. The method of claim 11 wherein for a screen that uses a different focal point for each pixel also the original two (or more) images for each frame are used, so that the appropriate side-views are available.

18. The method of claim 11 wherein for improved autostereoscopic 3D viewing at least one of:

a. Elongated complex lenses are coupled to a display screen, so that they direct the light from each pixel-column into intermittent expanding stripes of light-dark more efficiently, so that the light in the blocked areas is not wasted but is added to the light in the lit areas.

b. Elongated miniature triangles, more than one per each pixel column, are used, with techniques like in optic fibers, where the light is reflected internally by a core and a cladding that have a different optical refraction indexes, so that each pixel column is concentrated into the desired expanding on-off stripes of light-dark.

c. Light-emitting nano-elements are used that come out of each pixel in many directions.

d. Head tracking is used for determining if the user is in the correct right-left position, and if not then the image itself is instantly corrected by the computer by at least one of Switching between all the left and right pixels, and Moving the entire display left or right one pixel-column.

e. When the user is in an in-between position where each eye would view a mix of left and right images, the image can be moved along with the user also in half-pixel steps or other fractions of a pixel,

f. When the user is in an in-between-state, the elongated lenses can be moved and/or rotated a little in order to shift a little the position of the border between the right-left expanding stripes.

g. Pre-distortions are automatically added to the images, so that at least parts of the image that appear to jump out of the screen and/or images that appear to be far away will look more sharp when in fact the user focuses his eyes on the illusory position of the object.

h. Pre-distortions can be automatically added to the images on the fly, according to eye tracking that determines where the user is currently trying to focus his eyes, so that at least parts of the image that appear to jump out of the screen and/or images that appear to be far away will look more sharp when in fact the user focuses his eyes on the illusory position of the object.

19. The method of claim 18 wherein said elongated lenses are at least one of:

a. Wavy shaped elongated lenses.

b. Fresnel lenses with the desired parameters.

20. The method of claim 13 wherein at least one of the following features exist:

a. The cameras or camera parts are mounted on jibs, so that two arms are used, one for each camera.

b. The cameras or camera parts are mounted on the same jib, so that at the end of the jib there is an extension on which the cameras can move sideways.

c. The cameras or camera parts are mounted on a crane, so that at one camera is connected directly to the crane'"'"'s arm, and the other camera is connected to a sideways extension with which the cameras can be moved sideways, with or without an additional crane arm for the second camera.

d. The camera operator is shown through binoculars the correct 3D image, as transmitted by the computer.

e. Each camera has a small slit or uses other means to have a good focus at a large range of distances, so at least most of the image or the central part of the image is in focus all the time, so that the user will have less motivation to try to change the focus with his eyes when viewing the filmed scenes.

22. The method of claim 11 wherein for increasing the color information and/or the number of capture-able color combinations during capturing of images, at least one of the following steps are used:

a. Using at least 4 or more different primary color CCDs during the capture of the images.

b. Coding the images during the capture with 4 or more primary color codes instead of the normal 3.

c. Using a video capture system wherein the range of wavelengths sensitivity of each type of CCD is substantially higher or lower than normal.

d. Using a video capture system wherein the wavelength difference between the different primary color CCDs is substantially larger or substantially smaller than normal.

16. (Canceled).

17. (Canceled).

21. A method for increasing the color information and/or the number of capture-able color combinations during capturing of images, comprising at least one of the following steps:

a. Using at least 4 or more different primary color CCDs during the capture of the images.

b. Coding the images during the capture with 4 or more primary color codes instead of the normal 3.

c. Using a video capture system wherein the range of wavelengths sensitivity of each type of CCD is substantially higher or lower than normal.

d. Using a video capture system wherein the wavelength difference between the different primary color CCDs is substantially larger or substantially smaller than normal.