Method for the generation of synthetic images

• US 6,031,538 A
• Filed: 07/09/1996
• Issued: 02/29/2000
• Est. Priority Date: 08/30/1994
• Status: Expired due to Fees

• Current Assignee: Thomson Broadband Systems
• Original Assignee: Thomson Broadband Systems
• Inventors: Chupeau, Bertrand, Blonde, Laurent, Bordes, Philippe, Dusseux, Jean-Christophe
• Primary Examiner(s): FETTING, ANTON W
• Application Number:
  US08/619,526
• Time in Patent Office: 1,330 Days
• Field of Search: 395/173-175, 395/119, 395/949-960
• US Class Current: 345/419
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format

Abstract

The generation of images is done by the computation of the motion vectors representing the evolution from one image 2D to the other, directly from the co-ordinates and coefficients of geometrical transformation, of associated points in the 3D scene. The applications relate to the generation of monocular sequences by temporal interpolation and of stereoscopic sequences by extrapolation.

Claims

1. Method for the generation of synthetic images from structural data modeling a 3D scene to be displayed and data representing an apparent relative evolution of the scene with respect to a viewpoint, the structural data enabling the computation of the co-ordinates of points of the 3D scene and the data defining the evolution of the scene enabling the computation of the coefficients of geometrical transformation associated with these points, at which there is generated, on the basis of the structural data and a rendition algorithm, a 2D image representing the scene seen from this viewpoint, characterized in that, with each point of the 2D image, there is associated a corresponding point of the 3D image and a motion vector representing the displacement of this point of the image due to the apparent displacement of the corresponding point of the 3D scene with respect to the viewpoint, said motion vectors of the pixels of the image generated having been computed on the basis of the co-ordinates and coefficients of the associated points of the 3D scene in order to use them in the generation of at least one other 2D image on the basis of the first image, and then the generated images are shown on a display screen wherein the data defining the evolution comprise data on the evolution of the viewpoint of observation of the 3D scene corresponding to an apparent displacement of the static scene defined at an instant t, and in that a second image is computed by spatial extrapolation on the basis of a reference image that is generated from the rendition algorithm and corresponds to a viewpoint of the scene at the instant t, and on the basis of the motion vectors called disparity vectors and calculated from the said data, this second image corresponding to a second viewpoint of the scene at the same instant t to form a stereoscopic view.

2. Method according to claim 1, characterised in that a binocular sequence is formed by a reference channel constituted by key images generated on the basis of the rendition algorithm and intermediate images formed by temporal interpolation, and a conjugated channel constituted by key images formed by extrapolation of the key images of the reference channel of the basis of their disparity vectors and intermediate images formed by the temporal interpolation of these key images for the achievement of the stereoscopic effect.

3. Method according to claim 2, characterised in that the velocity vectors of a key image of the conjugated channel are computed on the basis of the velocity vectors and disparity vectors of the corresponding key image of the reference channel.

4. Method according to claim 3, characterized in that the zones in the conjugated image not placed in correspondence by the disparity vectors are generated by the rendition algorithm.

5. Method according to claim 2, characterized in that the zones in the conjugated image not placed in correspondence by the disparity vectors are generated by the rendition algorithm.

6. Method according to claim 1, characterised in that the zones in the conjugated image not placed in correspondence by the disparity vectors are generated by the rendition algorithm.

7. Method for the generation of synthetic images from structural data modeling a 3D scene to be displayed and data representing an apparent relative evolution of the scene with respect to a viewpoint, the structural data enabling the computation of the co-ordinates of points of the 3D scene and the data defining the evolution of the scene enabling the computation of the coefficients of geometrical transformation associated with these points, at which there is generated, on the basis of the structural data and a rendition algorithm, a 2D image representing the scene seen from this viewpoint, characterized in that, with each point of the 2D image, there is associated a corresponding point of the 3D image and a motion vector representing the displacement of this point of the image due to the apparent displacement of the corresponding point of the 3D scene with respect to the viewpoint, said motion vectors of the pixels of the image generated having been computed on the basis of the co-ordinates and coefficients of the associated points of the 3D scene in order to use them in the generation of at least one other 2D image on the basis of the first image, and then the generated images are shown on a display screen wherein the data defining the evolution comprise data on the temporal evolution of the 3D scene, and in that an intermediate image is computed by temporal interpolation on the basis of a reference image that is generated by the use of the rendition algorithm and that corresponds to the 3D scene at the instant t, and on the basis of the motion vectors called velocity vector and calculated from the said data, said intermediate image corresponding to a different instant and wherein a sequence of monocular images is constituted by key images generated on the basis of the rendition algorithm of the 3D scene and of intermediate images achieved by temporal interpolation, on the basis of these key images and of the computed velocity vectors.

8. Method according to claim 7, characterised in that it attributes, to each key image K(t), a first field of apparent velocity vectors V^- whose reverse vectors define, for each pixel of the key image K(t), its displacement from this image to the next key image K(t+N), a second field of apparent velocity vectors V⁺ defining the displacement of these pixels from the previous key image K(t-N) to the key image K(t), these vectors being computed on the basis of the data relating to the structure and evolution of the 3D scene, a distance map associating, with each pixel of the key image K(t), the distance from the corresponding visible point of the 3D scene to the point from which the scene is seen, and in that the interpolation between two key images K(t-N) and K(t) is done by reverse projection of the vectors V⁺ of the key image K(t) on the intermediate image and the key image K(t-N) and by reverse projection of the vectors V^- of the key image K(t-N) on the intermediate image and the key image K(t).

9. Method according to claim 8 characterised in that, during the temporal interpolation of an intermediate image I(t-n) between two key images K(t-N) and K(t), a conflict of projection between vectors V⁺ of the image K(t) or between vectors V^- of the image K(t-N) on the intermediate image I(t-n), i.e. the assignment of several vectors to one and the same pixel of the intermediate image, each vector being assigned to the closest pixel of its projection on the intermediate image, is resolved by choosing, as the chosen vector, that one whose origin is the pixel of the key image having the shortest distance, and in that the eliminated vectors are not taken into account for the temporal interpolation of this intermediate image.

10. Method according to claim 9, characterized in that, during the temporal interpolation of an intermediate image I(t-n) between two key images K(t-N) and K(t), a first step defines "appearing" and "vanishing" zones for each key image, the appearing zones on k(t), corresponding to zones or pixels not aimed at by the inverted apparent velocity vectors V^- of the image K(t-N), the vanishing zones on K(t-N) corresponding to zones or pixels not aimed at by the inverted apparent velocity vectors V⁺ reverse of the image K(t), the vectors V⁺ that reach the appearing zones and the vectors V^- that reach the vanishing zones being labelled as a function of these zones, a second step defines appearing and vanishing zones of intermediate image as corresponding respectively to the projections, on this intermediate image, of the vectors V⁺ and V^- labelled as being appearing and vanishing, a third step uses these zones in the temporal interpolation of this intermediate image.

11. Method according to claim 10, characterized in that the appearing and vanishing zones of the intermediate image I(t-n) are generated respectively and exclusively on the basis of the corresponding appearing zones of the key image K(t) and of the corresponding vanishing zones of the key image K(t-n), the other so-called "normal" zones being interpolated by taking account of the zones corresponding both to K(t-N) and K(t).

12. Method according to claim 9, characterized in that the management of interpolation errors is done by means of an estimated mask of the interpolation errors corresponding to the zones of the intermediate images for which the vectors V⁺ and V^- are not consistent.

13. Method according to claim 8 characterised in that, during the temporal interpolation of an intermediate image I(t-n) between two key images K(t-N) and K(t), a first step defines "appearing" and "vanishing" zones for each key image, the appearing zones on k(t) corresponding to zones or pixels not aimed at by the inverted apparent velocity vectors V^- of the image K(t-N), the vanishing zones on K(t-N) corresponding to zones or pixels not aimed at by the inverted apparent velocity vectors V⁺ reverse of the image K(t), the vectors V⁺ that reach the appearing zones and the vectors V^- that reach the vanishing zones being labelled as a function of these zones, a second step defines appearing and vanishing zones of intermediate image as corresponding respectively to the projections, on this intermediate image, of the vectors V⁺ and V^- labelled as being appearing and vanishing, a third step uses these zones in the temporal interpolation of this intermediate image.

14. Method according to claim 13, characterised in that the appearing and vanishing zones of the intermediate image I(t-n) are generated respectively and exclusively on the basis of the corresponding appearing zones of the key image K(t) and of the corresponding vanishing zones of the key image K(t-n), the other so-called "normal" zones being interpolated by taking account of the zones corresponding both to K(t-N) and K(t).

15. Method according to claim 13, characterized in that the management of interpolation errors is done by means of an estimated mask of the interpolation errors corresponding to the zones of the intermediate images for which the vectors V⁺ and V^- are not consistent.

16. Method according to claim 8, characterised in that the management of interpolation errors is done by means of an estimated mask of the interpolation errors corresponding to the zones of the intermediate images for which the vectors V⁺ and V^- are not consistent.