Adaptive modeling and segmentation of visual image streams

• US 6,124,864 A
• Filed: 10/10/1997
• Issued: 09/26/2000
• Est. Priority Date: 04/07/1997
• Status: Expired due to Fees

• Current Assignee: Synapix Incorporated
• Original Assignee: Synapix Incorporated
• Inventors: Robotham, John S., Moorby, Philip R., Schott, Jean-Pierre, Madden, Paul B.
• Primary Examiner(s): Zimmerman, Mark K.
• Assistant Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US08/948,721
• Time in Patent Office: 1,082 Days
• Field of Search: 345/473, 345/420, 345/435, 382/284, 382/285, 382/100, 382/154
• US Class Current: 345/473
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A technique for converging upon a computer-based model of a real world or synthetic scene. The computer model makes use of abstraction-based data objects as well as image-based data objects. A correlation mesh provides links between related image-based and abstraction-based objects. An initial step in a process analyzes an input image stream and user inputs to derive initial image-based objects and abstraction-based objects for the scene model. Subsequent steps in the process allow user inputs to refine the image-based objects, abstraction-based objects and/or the correlation mesh. As a result, refinements to the image-based object model of the scene can improve the abstraction-based model of the scene, and refinements to the abstraction-based object model can improve the image-based model.

Claims

1. A method for developing a scene model from a visual image sequence containing a sequence of visual image frames, the visual image sequence including a visual representation of one or more visual objects, the method comprising the steps of:

(a) analyzing portions of the visual image sequence in accordance with input parameters using a machine vision process by performing the steps of;

(i) defining an image-based data object in an image-based model of the visual image sequence containing a pixel representation corresponding to a portion of at least one frame of the visual image sequence;

(ii) defining an abstraction-based data object in an abstraction-based model of the visual objects containing an abstract representation of at least a portion of one of the visual objects represented by the visual image sequence; and

(iii) storing a link data object in a correlation mesh data structure indicating a correspondence between the image-based data object and the abstraction-based data object;

(b) refining an abstraction-based model of the visual imago sequence by performing the steps of;

(i) accepting input parameters to define in the abstraction-based model of the visual objects a new abstraction-based data object containing a new abstract representation of at least a portion of one of the visual objects contained in the visual image sequence that differs from the abstract representations contained in abstraction-based objects defined in the analysis step (a); and

(ii) adding a link object in the correlation mesh data structure indicating a correspondence between the new abstraction-based data object and another data object defined in the scene model; and

(c) iteratively improving the scene model by performing certain selected ones of steps (a) through (b) in an order as specified by user input until a desired level of refinement is obtained in the scene model such that selected link objects in the correlation mesh data structure added in iterations of the refining step (b) are used to provide additional input parameters to subsequent iterations thereby allowing the scene model to converge.

2. A method as in claim 1 wherein the abstraction-based objects comprise motion path models of the motion of a visual object from frame to frame.

3. A method as in claim 1 wherein the abstraction-based objects comprise camera motion path models of the motion of a camera from frame to frame.

4. A method as in claim 1 wherein the scene model is developed from a plurality of visual image sequences and certain links in the correlation mesh indicate correspondences between data objects representative of a given visual object as derived from different visual image sequences.

5. A method as in claim 1 wherein the visual object in the input visual image comprises a physical object.

6. A method as in claim 1 wherein the visual object in the input visual image comprises a synthetic object.

7. A method as in claim 1 wherein the step of defining an abstraction-based model additionally comprises the step of accepting user input to direct the operation of the machine vision process.

8. A method as in claim 1 wherein the step of defining an image-based data object comprises the step of operating a machine vision process.

9. A method as in claim 1 wherein the step of defining an abstraction-based model additionally comprises the step of accepting user input to define the abstract model.

10. A method as in claim 1 wherein the step of analyzing portions of the visual image sequence additionally comprises the step of:

using an accelerator to increase the speed of the analysis process.

11. A method as in claim 1 additionally comprising the step of presenting a view of the links in the correlation mesh to a user.

12. A method as in claim 1 wherein step (b)(ii) of adding a link to the correlation mesh data structure comprises the step of:

accepting user input to define a link between data objects in the scene model.

13. A method as in claim 1 additionally comprising the step of:

accessing a data object in the scene model by reference through the correlation mesh to another data object in the scene model.

14. A method as in claim 1 additionally comprising the step of:

accessing an image-based data object by reference to another data object in the scene model through a corresponding link in the correlation mesh.

15. A method as in claim 14 wherein at least one other data object is an abstract model data object.

16. A method as in claim 1 additionally comprising the step of:

accessing an abstraction-based data object by reference to another data object in the scene model through the corresponding link in the correlation mesh.

17. A method as in claim 16 wherein the other data object is a pixel representation data object.

18. A method as in claim 1 additionally comprising the step of:

storing a camera model data object in the scene model wherein the camera model data object defines at least one parameter of a camera that was used to generate the visual image sequence.

19. A method as in claim 18 wherein the camera model data object defines at least one camera parameter taken from the group consisting of camera position, camera movement, camera resolution, camera field of view, and camera orientation.

20. A method as in claim 1 wherein the visual image frames are stored as image-based data objects in the scene model.

21. A method as in claim 1 additionally comprising the step of:

storing a light model data object in the scene model wherein the light model data object defines at least one parameter of a light that was used to generate the visual image sequence.

22. A method as in claim 21 wherein the light model defines at least one light parameter taken from the group consisting of light position, light movement, light intensity, light color, and light orientation.

23. A method as in claim 1 additionally comprising the step of:

storing an acoustic model data object in the scene model wherein the acoustic model data object defines at least one parameter of an acoustic attribute of the scene model.

24. A method as in claim 1 wherein the step of refining an image-based model additionally comprises the steps of:

creating multiple pixel representation versions of a given visual image frame, the multiple pixel representation versions being at different levels of image resolution; and

using different ones of the pixel representation versions in given iterations.

25. A method as in claim 24 additionally comprising the step of:

presenting an interactive display of intermediate results of storing the scene model from a given iteration whereby a user specifies which one of the pixel representation versions is to be used in a subsequent iteration.

26. A method as in claim 24 wherein the different levels of image resolution are for different spatial resolutions.

27. A method as in claim 24 wherein the different levels of image resolution are for different color space resolutions.

28. A method as in claim 1 wherein the step of refining an abstraction-based model additionally comprises the steps of:

creating multiple abstraction-based model versions of visual object, the abstraction-based model versions being at different levels of modeling detail; and

using different ones of the abstraction-based model versions in given iterations.

29. A method as in claim 28 additionally comprising the step of:

presenting an interactive display of an intermediate scene model from a given iteration whereby a user specifies which one of the abstraction-based model versions is to be used in a subsequent iteration.

30. A method as in claim 28 wherein the different levels of modeling detail are different levels of geometry detail.

31. A method as in claim 28 wherein the different levels of modeling detail are different representations of surface textures.

41. A method as in claim 1 additionally comprising the step of:

(d) refining an image-based model of the visual image sequence by performing the step Of;

(i) accepting input parameters to define in the image-based model a new imago-based data object containing a pixel representation corresponding to a portion of at least one frame of the visual image sequence.

42. A method as in claim 41 wherein step (d) additionally comprises the step of:

(ii) adding a link object in the correlation mesh data structure to indicate a correspondence between the new image-based data object and another data object in the scene model; and

wherein step (c) additionally comprises the step of performing certain selected ones of steps (a), (b), and (d) in an order as specified by user input such that selected link objects in the correlation mesh added in iterations of refining step (d) are also used as additional input parameters to subsequent iterations.

43. A method as in claim 1 in which an additional refining step accepts input parameters which modify or delete a link object in the correlation mesh data structure.

32. A method for developing a computerized scene model from a digital image, the method comprising the steps of:

(a) running an automated image analysis algorithm according to input parameters to provide image analysis results data;

(b) storing portion of the image analysis results data as an image-based data object in the scene model;

(c) storing other portions of the image analysis results data as an abstraction-based data object in the scene model;

(d) annotating a correlation mesh data object in the scene model with data indicating a link between the image-based data object and the abstraction-based data object;

(e) refining the scene model by accepting input parameters concerning at least one new abstract representation of the digital image, the new abstract object containing an abstract representation differing from the abstraction-based object stored in step (c);

(f) annotating a correlation mesh data object in the scene model with data indicating a link between the new abstraction-based data object and other data objects in the scene model; and

(g) iterating selected ones of steps (a) through (f) in response to user input until a desired level of refinement is obtained in the scene model such that selected link objects in the correlation mesh data structure added in iterations of the refining step (e) are used to provide additional input parameters to subsequent iterations, thereby allowing the scene model to converge.

33. A method as in claim 32 wherein the image processing algorithm is selected from the group consisting of feature tracking, image segmentation, and optical flow.

34. A method as in claim 32 wherein the abstract representations include geometric models.

35. A method as in claim 32 wherein the abstract representations include volumetric models.

36. A method as in claim 32 additionally comprising the step of:

specifying a spatial reference model defining a three-dimensional stage coordinate system in the scene model.

37. A method as in claim 36 wherein a projective transformation between the image plane of an image-based data object and the coordinate system of the abstraction-based data object is defined with respect to the spatial reference model.

38. A method as in claim 32 wherein the step of accepting input data concerning abstract representations accepts input data from a human user.

39. A method as in claim 32 wherein the step of accepting input data concerning abstract representations accepts input data from an image processing algorithm.

40. A method as in claim 32 additionally comprising the step of, before step (f), displaying a visual representation rendered from the scene model.