Temporal smoothing of scene analysis data for image sequence generation

• US 6,297,825 B1
• Filed: 04/06/1998
• Issued: 10/02/2001
• Est. Priority Date: 04/06/1998
• Status: Expired due to Fees

• Current Assignee: Synapix Incorporated
• Original Assignee: Synapix Incorporated
• Inventors: Eyring, Kenneth J., Cobb, Wesley K., Schott, Jean-Pierre, Madden, Paul B., Askey, David
• Primary Examiner(s): Vo, Cliff N.
• Application Number:
  US09/055,843
• Time in Patent Office: 1,275 Days
• Field of Search: 345/418, 345/419, 345/420, 345/426, 345/431, 345/432
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

A technique for temporal smoothing of results from a scene analysis process which creates a sequence of visually pleasing and acceptable images generated in whole or part from such results. The technique applies temporal smoothing across time-related sets of scene analysis results. The spatial smoothing can be applied at various steps in the process: to images in the original sequence, to intermediate or final results of scene analysis in either a pixel-oriented or geometric domain, or to the images generated in whole or part from the scene analysis results. In a preferred embodiment, different levels of smoothing are applied to different parts of the intermediate or final results. The differentiation can be done using image masks (for pixel-oriented results) or geometry selection techniques (for geometric results). This allows a higher level of smoothing in certain areas where the generated visual images will benefit from such additional smoothing, while avoiding over-smoothing in other areas where the smoothing would obscure critical details within the generated images.

Claims

1. A method for processing a sequence of source images of a visual scene comprising the steps of:

(a) analyzing the sequence of images to develop a data model of selected characteristics of the scene;

(b) selecting at least two time-based series of data sets from the data model of selected characteristics provided by the scene analysis process;

(c) comparing corresponding parametric values of a given time series with corresponding parametric values of another times series data sets, and

(d) applying a smoothing filter and scaling adjustments to the parametric values to modify the data sets, to obtain visually pleasing results in a design visualization application.

2. A method as in claim 1 wherein the step of selecting a time-based series of data sets selects data from the original source image sequence.

3. A method as in claim 1 wherein the step of selecting a time-based series of data sets selects data from intermediate results of the scene analysis selected from the group consisting of camera parameters, captured parametric data, mark up data related to the images, or adjustments to parametric data.

4. A method as in claim 1 wherein the step of selecting a time-based series of data sets selects data from results of the scene analysis.

5. A method as in claim 4 wherein the results are a depth map.

6. A method as in claim 4 wherein the results are a surface mesh.

7. A method as in claim 1 wherein the step of selecting at least two time-based series of data sets additionally comprises calculating an optical flow field.

8. A method as in claim 7 additionally comprising obtaining a parallax fit.

9. A method as in claim 7 additionally comprising deriving a gamma coefficient.

10. A method as in claim 1 further comprising, prior to the step of applying a smoothing filter, deriving a gamma coefficient for each of a multiplicity of pixels, the gamma coefficients being inversely proportional to the pixel depth of the pixels.

11. A method as in claim 10 wherein the step of applying a smoothing filter includes applying the smoothing filter to the gamma coefficients.

12. A method as in claim 11 further comprising, after the step of applying a smoothing filter, converting each gamma coefficient to an estimated pixel depth.

13. A method as in claim 12 further comprising applying the smoothing filter to the estimated pixel depth.

14. A method as in claim 10 further comprising converting each gamma coefficient to an estimated pixel depth.

15. A method as in claim 14 wherein the step of applying a smoothing filter includes applying the smoothing filter to the estimated pixel depth.

16. A method for processing a sequence of source images of a visual scene comprising the steps of:

(a) analyzing the sequence of images to develop a data model of selected characteristics of the scene;

(b) selecting at least two time-based series of data sets from the data model of selected characteristics provided by the scene analysis process;

(c) comparing corresponding parametric values of a given time series with corresponding parametric values of another times series data sets;

(d) deriving a gamma coefficient for each of a multiplicity of pixels, the gamma coefficients being inversely proportional to the pixel depth of the pixels; and

(e) modifying the parametric values and the gamma coefficients, to obtain visually pleasing results in a design visualization application.

17. A method as in claim 16, wherein the step of modifying includes applying a smoothing filter to the gamma coefficients.

18. A method as in claim 17, further comprising converting each gamma coefficient to an estimated pixel depth.

19. A method as in claim 18, further comprising applying the smoothing filter to the estimated pixel depth.