Visual motion analysis method for detecting arbitrary numbers of moving objects in image sequences

• US 20030219146A1
• Filed: 05/23/2002
• Published: 11/27/2003
• Est. Priority Date: 05/23/2002
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Black, Michael J., Jepson, Allan D., Fleet, David J.
• Granted Patent: US 6,954,544 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

A visual motion analysis method that uses multiple layered global motion models to both detect and reliably track an arbitrary number of moving objects appearing in image sequences. Each global model includes a background layer and one or more foreground “polybones”, each foreground polybone including a parametric shape model, an appearance model, and a motion model describing an associated moving object. Each polybone includes an exclusive spatial support region and a probabilistic boundary region, and is assigned an explicit depth ordering. Multiple global models having different numbers of layers, depth orderings, motions, etc., corresponding to detected objects are generated, refined using, for example, an EM algorithm, and then ranked/compared. Initial guesses for the model parameters are drawn from a proposal distribution over the set of potential (likely) models. Bayesian model selection is used to compare/rank the different models, and models having relatively high posterior probability are retained for subsequent analysis.

Claims

1. A visual motion analysis method for analyzing an image sequence depicting a three-dimensional event including a plurality of objects moving relative to a background scene, the image sequence being recorded in a series of frames, each frame including image data forming a two-dimensional representation including a plurality of image regions depicting the moving objects and the background scene at an associated point in time, the method comprising:

identifying a first moving object of the plurality of moving objects by comparing a plurality of frames of the image sequence and identifying a first image region of the image sequence including the first moving object, wherein the first image region includes a central portion surrounded by an outer edge; and

generating a layered global model including a background layer and a foreground component, wherein the foreground component includes exclusive spatial support region including image data located in the central portion of the first image region, and a probabilistic boundary region surrounding the exclusive spatial support region and including image data associated with the outer edge of the first image region.

2. The visual motion analysis method according to claim 1, wherein generating the layered global model comprises defining the exclusive spatial support region using image data associated with a closed-polygon shaped region of the central portion of the first image region.

3. The visual motion analysis method according to claim 2, wherein the closed polygonal shape is an octagon.

4. The visual motion analysis method according to claim 2, wherein the exclusive spatial support region is defined by size, orientation, and position parameters defining the closed-polygon shaped region.

5. The visual motion analysis method according to claim 1, wherein generating the layered global model comprises defining a width of the probabilistic boundary region according to image characteristics associated with the outer edge, wherein the width of the probabilistic boundary region is relatively wide when the outer edge is relatively blurred, and wherein the width of the probabilistic boundary region is relatively narrow when the edge region is relatively sharp.

6. The visual motion analysis method according to claim 1, wherein generating the layered global model comprises generating a plurality of foreground components, wherein each foreground component is assigned an explicit depth ordering relative to the other foreground components.

7. The visual motion analysis method according to claim 1, further comprising determining the visibility of each foreground component at each pixel of an image frame, wherein the visibility is determined by the depth orderings assigned to the plurality of foreground components.

8. The visual motion analysis method according to claim 7, wherein the visibility of a first foreground component at a first pixel is considered fully occluded when the exclusive spatial support region of a second foreground component having higher depth ordering includes the first pixel.

9. The visual motion analysis method according to claim 7,

wherein each foreground component is assigned an opacity value, and

wherein the visibility of a first foreground component at a first pixel is considered partially occluded when the exclusive spatial support region of a second foreground component having higher depth ordering includes the first pixel, and when the opacity value of the second foreground component indicates that the object assigned to the second foreground component is partial transparency.

10. The visual motion analysis method according to claim 1, wherein generating the foreground component further comprises forming an appearance model including image data from the first image region, and forming a motion model by calculating movement of the first image region over two or more frames of the image sequence.

11. The visual motion analysis method according to claim 10, wherein the appearance model is a WSL appearance model.

12. The visual motion analysis method according to claim 1, wherein the background layer includes a background appearance model including image data associated with regions of the image sequence depicting the background scene.

13. The visual motion analysis method according to claim 12, wherein the background layer includes parameters associated with motion of the background appearance model in the image sequence.

14. The visual motion analysis method according to claim 1, wherein generating the layered global model comprises defining the exclusive spatial support region using image data associated with one of:

a spline shaped portion of the first image region, a portion of the first image region having a shape defined by harmonic bases, and a portion of the first image region defined by level-sets of implicit polynomic functions.

15. A visual motion analysis method for analyzing an image sequence depicting a three-dimensional event including a plurality of objects moving relative to a background scene, the image sequence being recorded in a series of frames, each frame including image data forming a two-dimensional representation including a plurality of moving image regions, each moving image region depicting one of the moving objects at an associated point in time, each frame also including image data associated with the background scene at the associated point in time, the method comprising:

generating a plurality of layered global models utilizing image data from the image sequence, each layered global model including a background layer and at least one foreground component, wherein each foreground component includes exclusive spatial support region including image data from a central portion of an associated moving image region, and a probabilistic boundary region surrounding the exclusive spatial support region and including image data including an outer edge of the associated moving image region;

refining each foreground component of each layered global model such the exclusive spatial support region of each foreground component is optimized to the image data of the moving image region associated with said each foreground component; and

ranking the plurality of layered global models and identifying a layered global model that most accurately models the image data of the image sequence.

16. The visual motion analysis method according to claim 15, further comprising identifying the plurality of moving objects by comparing image data from a plurality of frames of the image sequence and identifying outliers.

17. The visual motion analysis method according to claim 15, wherein generating the plurality of layered global models comprises selecting a moving object from the plurality of identified moving objects, and generating a layered global model having a foreground component assigned to the selected moving object.

18. The visual motion analysis method according to claim 15, wherein generating the plurality of layered global models comprises generating a seed model having a seed background layer, and spawning a next-generation layered global model by identifying a selected moving image region in the seed background layer, assigning a new foreground component to the selected moving image region, and forming the next-generation layered global model by combining the seed model and the new foreground component.

19. The visual motion analysis method according to claim 18, wherein a size of the new foreground component is initially limited to a predefined initial amount of image data.

20. The visual motion analysis method according to claim 19, wherein subsequent refining of the new foreground component to fit the image data associated with the selected moving object region is restricted such that an increase from the initial amount is limited to a predetermined rate for a given time period.

21. The visual motion analysis method according to claim 15, wherein generating the plurality of layered global models comprises generating, for a selected global model, a plurality of foreground components, wherein each foreground component is assigned an explicit depth ordering relative to the other foreground components.

22. The visual motion analysis method according to claim 15, wherein refining each foreground component of each layered global model comprises using a “

hill climbing”

method to optimize parameters of each foreground component to the image data of the associated moving image region.

23. The visual motion analysis method according to claim 22, wherein refining each foreground component comprises differentiating a likelihood function with respect to pose parameters of said each foreground component to determine how much a change in the pose parameter will affect a fitness value associated with the pose parameter, wherein the pose parameters include size, position, and orientation of the exclusive spatial support region.

24. The visual motion analysis method according to claim 22, wherein refining each foreground component further comprises utilizing a gradient-based search technique to determine whether a particular parameter change improves the fitness value.

25. The visual motion analysis method according to claim 22, wherein refining each foreground component is constrained such that changes to the parameter are limited to a predetermined amount per frame.

26. The visual motion analysis method according to claim 15, wherein ranking comprises utilizing a Bayesian model selection criterion that determines the fit of each foreground component parameter to the image data of the associated moving object region.

27. The visual motion analysis method according to claim 15, wherein ranking comprises utilizing a likelihood function that is biased to penalize complexity such that a first layered global model having a relatively low number of foreground components is ranked higher than a second layered global models having a relatively high number of foreground components when the fitness of the first and second layered global models is otherwise equal.

28. The visual motion analysis method according to claim 15, further comprising eliminating low ranking global models from the plurality of layered global models after ranking is performed.

29. The visual motion analysis method according to claim 28, wherein eliminating low ranking global models comprises eliminating all but one layered global models for each group of layered global models having an equal number of foreground components.

30. The visual motion analysis method according to claim 28, further comprising repeating the generating, refining, and ranking procedures for each sequential selected frame of the image sequence.

31. A visual motion analysis method for analyzing an image sequence depicting a three-dimensional event including a plurality of objects moving relative to a background scene, the image sequence being recorded in a series of frames, each frame including image data forming a two-dimensional representation including a plurality of moving image regions, each moving image region depicting one of the moving objects at an associated point in time, each frame also including image data associated with the background scene at the associated point in time, the method comprising:

generating a plurality of layered global models utilizing image data from the image sequence, each layered global model including a background layer and at least one foreground component, wherein each foreground component includes exclusive spatial support region including image data from a central portion of an associated moving image region, and a probabilistic boundary region surrounding the exclusive spatial support region and including image data including an outer edge of the associated moving image region;

ranking the plurality of layered global models such that layered global models that relatively accurately model the image data of the image sequence are ranked relatively high, and layered global models that relatively inaccurately model the image data of the image sequence are ranked relatively low; and

eliminating low ranking global models from the plurality of layered global models.

32. The visual motion analysis method according to claim 31, further comprising identifying the plurality of moving objects by comparing image data from a plurality of frames of the image sequence and identifying outliers.

33. The visual motion analysis method according to claim 31, wherein generating the plurality of layered global models comprises selecting a moving object from the plurality of identified moving objects, and generating a layered global model having a foreground component assigned to the selected moving object.

34. The visual motion analysis method according to claim 31, wherein generating the plurality of layered global models comprises generating a seed model having a seed background layer, and spawning a next-generation layered global model by identifying a selected moving image region in the seed background layer, assigning a new foreground component to the selected moving image region, and forming the next-generation layered global model by combining the seed model and the new foreground component.

35. The visual motion analysis method according to claim 34, wherein a size of the new foreground component is initially limited to a predefined initial amount of image data.

36. The visual motion analysis method according to claim 35, wherein subsequent refining of the new foreground component to fit the image data associated with the selected moving object region is restricted such that an increase from the initial amount is limited to a predetermined rate for a given time period.

37. The visual motion analysis method according to claim 31, wherein generating the plurality of layered global models comprises generating, for a selected global model, a plurality of foreground components, wherein each foreground component is assigned an explicit depth ordering relative to the other foreground components.

38. The visual motion analysis method according to claim 31, further comprising refining each foreground component of each layered global model such the exclusive spatial support region of each foreground component is optimized to the image data of the moving image region associated with said each foreground component.

39. The visual motion analysis method according to claim 38, wherein refining each foreground component of each layered global model comprises using a “

hill climbing”

method to optimize parameters of each foreground component to the image data of the associated moving image region.

40. The visual motion analysis method according to claim 39, wherein refining each foreground component comprises differentiating a likelihood function with respect to pose parameters of said each foreground component to determine how much a change in the pose parameter will affect a fitness value associated with the pose parameter, wherein the pose parameters include size, position, and orientation of the exclusive spatial support region.

41. The visual motion analysis method according to claim 39, wherein refining each foreground component further comprises utilizing a gradient-based search technique to determine whether a particular parameter change improves the fitness value.

42. The visual motion analysis method according to claim 39, wherein refining each foreground component is constrained such that changes to the parameter are limited to a predetermined amount per frame.

43. The visual motion analysis method according to claim 31, wherein ranking comprises utilizing a Bayesian model selection criterion that determines the fit of each foreground component parameter to the image data of the associated moving object region.

44. The visual motion analysis method according to claim 31, wherein ranking comprises utilizing a likelihood function that is biased to penalize complexity such that a first layered global model having a relatively low number of foreground components is ranked higher than a second layered global models having a relatively high number of foreground components when the fitness of the first and second layered global models is otherwise equal.

45. The visual motion analysis method according to claim 31, wherein eliminating low ranking global models comprises eliminating all but one layered global models for each group of layered global models having an equal number of foreground components.

46. The visual motion analysis method according to claim 31, further comprising repeating the generating, ranking, and eliminating procedures for each sequential selected frame of the image sequence.