Modelization of objects in images

• US 9,070,207 B2
• Filed: 09/07/2008
• Issued: 06/30/2015
• Est. Priority Date: 09/06/2007
• Status: Active Grant

• Current Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Original Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Inventors: Yomdin, Yosef, Dinkin, Gregory, Sarig, Niv, Yomdin, Avital
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Wilson, Nicholas R
• Application Number:
  US12/676,363
• Publication Number:
  US 20100259546A1
• Time in Patent Office: 2,487 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models

Abstract

A system includes an aligner to align an initial position of a partially kinematically, parameterized model with an object in an image, and a modelizer to adjust parameters of the model to match the model to contours of the object, given the initial alignment. An animation system includes a modelizer to hierarchically match a hierarchically rigid model to an object in an image, and a cutter to cut the object from the image and to associate it with the model. A method for animation includes hierarchically matching a hierarchically rigid model to an object in an image, and cutting the object from the image to associate it with the model.

Claims

1. A system comprising:

an aligner to align an initial position of a hierarchically rigid model with an object in an image, said hierarchically rigid model represented by a plurality of dots associated with alignment points, initial and secondary anatomic points, and by contour points; and

wherein contour points define contour lines;

a modelizer to adjust parameters of said model to match said model to contours of said object, given said initial alignment; and

means to enable a user to operate at least one of said aligner and said modelizer interactively;

wherein said initial position alignment comprises said aligner positioning said alignment points at locations associated with extremities of said object, and

wherein said parameters adjustment comprises

said modelizer matching said rigid model in stages to correct for noise in each stage, said stages comprising;

determining a position of said model by positioning said initial anatomic points relative to said alignment points;

correcting anatomic positioning of said model by positioning said secondary anatomic points relative to said initial anatomic points;

defining geometrically shaped active regions in said model associated with segments of said contour lines;

matching said contour lines to contour edges in said object.

2. The system according to claim 1 and also comprising means to perform model guided interactive matching.

3. The system according to claim 1 and also comprising a cutter to cut said object from said image and to associate it with said model.

4. The system according to claim 3 and also comprising an animation adapter to animate said cut object according to an animation associated with said model.

5. The system according to claim 4 and wherein said animation adapter comprises means to provide small movement animation.

6. The system according to claim 5 and wherein said animation adapter comprises:

an enlarger to enlarge a part of said object; and

an animator to animate said enlarged part of said object over a white space area surrounded by a background margin, wherein;

said white space area is defined by an original position of said part of said object, and

said background margin is defined as a difference in area of said image covered by said enlarged part of said object and said white space area.

7. The system according to claim 4 and wherein said animation adapter comprises:

an extractor to extract said cut object from said image; and

an animator to animate said cut object on a background different than that of said image.

8. The system according to claim 7 and wherein said animation adapter comprises a reconstructor to perform an approximate reconstruction of occluded parts of said object as per at least one of the following:

an associated non-occluded part of said object, a matching part of a model matched to said object, and interpolation of non-occluded colors in said object.

9. The system according to claim 4 and wherein said animation is associated with at least one of the following:

a greeting card, a game, a game avatar and a digital camera implementation.

10. The system according to claim 1 and wherein said model is of a human standing enface, wherein an appropriate said model is determined based on a matrix of qualifications including at least one of sex, age, body build, athletic skill and pose.

11. The system according to claim 1 and wherein said model is of one of the following:

an animal and a virtual animated object.

12. The system according to claim 1 and wherein said aligner comprises means to receive marking points from a user marking said alignment points on said image.

13. The system according to claim 12 and wherein said aligner comprises means to adjust said model to align said marking points with said alignment points.

14. The system according to claim 1 and wherein said at least one of said initial and secondary points are for performing coarse scale model positioning in a multi-step process.

15. The system according to claim 1 and wherein said points are organized into groups, and each group has a set of forbidden values based on anatomic restrictions.

16. The system according to claim 15 and comprising a model distortion calculator to calculate overall model distortion as a sum of:

penalty functions for each group, a global penalty function, and a stretching function.

17. The system according to claim 16 and comprising means to stop processing when said overall model distortion exceeds a pre-defined threshold.

18. The system according to claim 16 and comprising a penalty function calculator to calculate said penalty function for each said group, wherein a value of said penalty function approaches infinity as said control points of said group approach said forbidden values.

19. The system according to claim 16 and comprising a global penalty calculator to calculate a global penalty function defined by a total distance of said control parameters from a globally anatomically forbidden region.

20. The system according to claim 16 and comprising a stretching function calculator to calculate said stretching function as a measure of total deviation of all said parameters from their associated reference points.

21. The system according to claim 1 and wherein said aligner comprises means to enable a user to create said image with said object in generally the same pose as said model.

22. The system according to claim 1 and wherein said aligner comprises means to cunningly adjust said model to coarsely align said model with said object in said image.

23. The system according to claim 1 and also comprising:

a video frame provider, to provide said aligner with one frame at a time from a video sequence; and

an action generator to generate an action sequence from a collection of models generated by said modelizer in response to said frames of said video sequence.

24. The system according to claim 23 and wherein said video sequence is from at least one of a movie, media clip and broadcast.

25. The system according to claim 23 and wherein said video sequence is of movement of a user of a game and said action sequence is provided to an avatar of said user.

26. The system according to claim 1 and wherein said modelizer comprises:

an edge detector to generate an edge image of said image;

an anatomic points matcher to adjust said anatomic points of said model to coarsely match a portion of a contour of said model to said edges of said object that are generally parallel to said contour and are in a vicinity of said model and to mark said edges;

a contour aligner to adjust said contour of said model to more tightly match said edges of said object that are generally parallel to said contour and are in a vicinity of said model and to mark said edges; and

a contour refiner to adjust said contour of said model to said selected edges.

27. The system according to claim 26 and wherein said matcher comprises means to calculate a series of matching functions between said edges and portions of said contours, wherein an overall contour matching quality may be expressed as a maximum of calculated values for said matching functions over all matching groups.

28. The system according to claim 26 and wherein said modelizer comprises:

a geometric combination analyzer to perform a model-guided geometric search by analyzing said edge images and to provide scanning regions for processing to said anatomic points matcher.

29. The system according to claim 28 and wherein said geometric combination analyzer comprises means to perform said model-guided geometric search on progressively finer scales with progressively more detailed said models until the results of a said search no longer improve upon the results of a previous iteration of said search.

30. The system according to claim 28 and wherein said geometric combination analyzer comprises:

means to output a set of possible preliminary positions of said model on said image with a likelihood for each said preliminary position; and

means to form a scanning region in a Model Parameter Space (MPS) around each of said preliminary model positions.

31. The system according to claim 30 and wherein said geometric combination analyzer comprises:

a filter to perform model guided geometric filtering to construct chains of edges with a shape approximately corresponding to a shape of pre-specified characteristic regions of a relevant said model;

means to form groups of lines corresponding to pre-specified characteristic regions on said model;

a validator to validate said groups of lines by comparing each of said groups to other said groups and as per any a priori known restrictions of said model; and

a pivot elements detector to identify pivot elements as per known restrictions of said model.

32. The system according to claim 1 wherein said modelizer matches said contour lines to said contour edges in said object by perpendicularly displacing any one of said initial anatomical points, said secondary anatomical points, and said control points, relative to said segments of said contour lines.

33. An animation system comprising:

a rigid model and animation library comprising at least one rigid model and animations associated with said at least one rigid model;

a modelizer to hierarchically match a rigid model selected from said rigid model and animation library to an object in an image;

a cutter to cut said object from said image along object contour lines determined by said modelizer;

an animation adapter to animate said cut object according to animations in said rigid model and animation library associated with said selected rigid model; and

means to allow a user to select said rigid model and said associated animations from said rigid model and animation library,

wherein said hierarchical matching comprises matching said rigid model in stages to correct for noise in each stage, said stages comprising;

determining a position of said model by positioning initial anatomic points in said model relative to alignment points in said model, wherein said alignment points are associated with extremities of said object;

correcting anatomic positioning of said model by positioning secondary anatomic points in said model relative to said initial anatomic points;

defining geometrically shaped active regions in said model associated with segments of contour lines in said model;

matching said contour lines to contour edges in said object.

34. A method for animation comprising:

selecting, using user selection means, a rigid model from a rigid model and animation library comprising at least one rigid model and animations associated with said at least one rigid model;

hierarchically matching said selected rigid model to an object in an image using a modelizer;

cutting said object from said image along object contour lines determined by said modelizer using a cutter;

selecting, using said user selection means, animations from said rigid model and animations library associated with said selected rigid model; and

animating said cut object according to said selected associated animations,

wherein said hierarchical matching comprises

matching said rigid model in stages to correct for noise in each stage, said

stages comprising;

determining a position of said model by positioning initial anatomic points in said model relative to alignment points in said model, wherein said alignment points are associated with extremities of said object;

correcting anatomic positioning of said model by positioning secondary anatomic points in said model relative to said initial anatomic points;

defining geometrically shaped active regions in said model associated with segments of said contour lines;

matching said contour lines to contour edges in said object.

35. The method according to claim 34 and also comprising enabling said user to perform said matching interactively.

36. The method according to claim 34 and also comprising enabling said user to perform model guided interactive matching.

37. The method according to claim 34 and wherein said animating is small movement animation.

38. The method according to claim 37 and wherein said small movement animation comprises:

enlarging a part of said object; and

animating said enlarged part of said object over a white space area surrounded by a background margin, wherein;

said white space area is defined by an original position of said part of said object, and

said background margin is defined as a difference in area of said image covered by said enlarged part of said object and said white space area.

39. The method according to claim 34 and wherein said animating comprises:

extracting said cut object from said image; and

animating said cut object on a different background.

40. The method according to claim 34 and also comprising performing an approximate reconstruction of occluded parts of said object as per at least one of the following:

an associated non-occluded part of said object, a matching part of a model matched to said object, and interpolation of non-occluded colors in said object.

41. The method according to claim 34 and wherein said animating is associated with at least one of a greeting card, game, game avatar and a digital camera implementation.

42. The method according to claim 34 and wherein said model is of a human standing enface and determining an appropriate said model based on a matrix of qualifications including at least one of sex, age, body build, athletic skill and pose.

43. The method according to claim 34 and wherein said model is of one of the following:

an animal and a virtual animated object.

44. The method according to claim 34 and comprising receiving marking points from a user marking said alignment points on said image.

45. The method according to claim 44 and wherein said alignment points are control points organized in a hierarchically rigid manner to define a specific multi-scale organization of control parameters for fitting them to said object.

46. The method according to claim 45 and wherein said control points are at least one of:

anatomic parameters for coarse scale model positioning;

geometrically shaped active regions of certain combinations of segments of a contour of a model; and

contour control points for finely fitting edges to said contour within a proscribed neighborhood of said model.

47. The method according to claim 46 and wherein said anatomic parameters are at least one of said initial and secondary points for performing said coarse scale model positioning in a multi-step process.

48. The method according to claim 45 and comprising organizing said control points into groups, and defining a forbidden set of values based on anatomic restrictions for each group.

49. The method according to claim 48 and comprising calculating overall model distortion as a sum of:

penalty functions for each group, a global penalty function, and a stretching function.

50. The method according to claim 49 and comprising stopping processing when said overall model distortion exceeds a pre-defined threshold.

51. The method according to claim 49 and comprising calculating said penalty function for each said group, wherein a value of said penalty function approaches infinity as said control points of said group approach said forbidden set.

52. The method according to claim 49 and comprising calculating a global penalty function defined as a total distance of said control parameters from a globally anatomically forbidden region.

53. The method according to claim 49 and comprising calculating said stretching function as a measure of total deviation of all said control parameters from their associated reference points.

54. The method according to claim 44 and comprising adjusting said model to align said marking points with said alignment points.

55. The method according to claim 34 and comprising enabling a user to create said image with said object in a generally same pose as said model.

56. The method according to claim 34 and comprising cunningly adjusting said model to coarsely align said model with said object in said image.

57. The method according to claim 34 and also comprising:

providing one frame at a time from a video sequence for aligning an initial position of said model with said object; and

generating an action sequence from a collection of said models in response to said frames of said video sequence.

58. The method according to claim 57 and wherein said video sequence is from at least one of a movie, media clip and broadcast.

59. The method according to claim 57 and comprising providing said action sequence to an avatar of said user, wherein said video sequence is of the movement of a user of a game.

60. The method according to claim 34 and wherein said matching comprises:

generating an edge image of said image;

adjusting anatomic points of said model to coarsely match a portion of a contour of said model to edges of said object that are generally parallel to said contour and are in a vicinity of said model and to mark said edges;

adjusting said contour of said model to more tightly match edges of said object that are generally parallel to said contour and are in a vicinity of said model and to mark said edges; and

adjusting said contour of said model to said selected edges.

61. The method according to claim 60 and wherein said matching also comprises calculating a series of matching functions between said edges and portions of said contours, wherein an overall contour matching quality may be expressed as a maximum of calculated values for said matching functions over all matching groups.

62. The method according to claim 60 and also comprising:

performing a model-guided geometric search by analyzing said edge images; and

providing scanning regions for processing to said anatomic points matcher.

63. The method according to claim 62 and wherein said performing comprises performing said model-guided geometric search on progressively finer scales with progressively more detailed said models until the results of a said search no longer improve upon the results of a previous iteration of said search.

64. The method according to claim 62 and wherein said performing comprises:

outputting a set of possible preliminary positions of said model on said image with a likelihood for each said preliminary position; and

forming a scanning region in a MPS around each of said preliminary model positions.

65. The method according to claim 64 and also comprising:

performing model guided geometric filtering to construct chains of edges with a shape approximately corresponding to a shape of pre-specified characteristic regions of a relevant said model;

forming groups of lines corresponding to pre-specified characteristic regions on said model;

validating said groups of lines by comparing each of said groups to other said groups and as per any a priori known restrictions of said model; and

identifying pivot elements as per known restrictions of said model.

66. The animation system according to claim 34 wherein said rigid model further comprises free parameters with which to change a shape of said model and an external contour of said model prevents said modelizer from following noisy or false object contour lines during said hierarchical matching.