Method and system for multiple object detection by sequential Monte Carlo and hierarchical detection network

• US 8,605,969 B2
• Filed: 04/06/2011
• Issued: 12/10/2013
• Est. Priority Date: 04/06/2010
• Status: Active Grant

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Corp. (Siemens AG)
• Inventors: Sofka, Michal, Zhang, Jingdan, Zhou, Shaohua Kevin, Comaniciu, Dorin
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US13/080,964
• Publication Number:
  US 20110243386A1
• Time in Patent Office: 979 Days
• Field of Search: 382/128, 382/181
• US Class Current: 382/128
• CPC Class Codes: A61B 8/0816 using echo-encephalography
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30016 Brain
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30056 Liver; Hepatic
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 2201/031 of internal organs

Abstract

A method and system for detecting multiple objects in an image is disclosed. A plurality of objects in an image is sequentially detected in an order specified by a trained hierarchical detection network. In the training of the hierarchical detection network, the order for object detection is automatically determined. The detection of each object in the image is performed by obtaining a plurality of sample poses for the object from a proposal distribution, weighting each of the plurality of sample poses based on an importance ratio, and estimating a posterior distribution for the object based on the weighted sample poses.

Claims

1. A method for detecting multiple objects in an image, comprising:

sequentially detecting a plurality of objects in an image in an order specified by a trained hierarchical detection network, wherein the detection of each object in the image comprises;

obtaining a plurality of sample poses for the object from a proposal distribution of object poses for the object,

weighting each of the plurality of sample poses based on an importance ratio calculated for each sample pose, and

estimating a posterior distribution for the object based on the weighted sample poses.

2. The method of claim 1, wherein the step of obtaining a plurality of sample poses for the object from a proposal distribution of object poses for the object comprises:

determining the proposal distribution for the object based on one or more detected poses for a previously detected one of the plurality of objects.

3. The method of claim 2, wherein the step of determining the proposal distribution for the based one or more detected poses for a previously detected one of the plurality of objects comprises:

applying a Gaussian distribution transition kernel to each of the one or more detected poses of the previously detected one of the plurality of objects, wherein the Gaussian distribution transition kernel is estimated based on training data.

4. The method of claim 1, wherein the step of obtaining a plurality of sample poses for the object from a proposal distribution of object poses for the object comprises:

obtaining the plurality of sample poses using k-means sampling.

5. The method of claim 1, wherein the step of weighting each of the plurality of sample poses based on an importance ratio calculated for each sample comprises:

weighting each of the plurality of sample poses based on an importance ratio calculated for each sample pose using a probability determined for each sample pose by a trained object detector.

6. The method of claim 1, wherein the step of estimating a posterior distribution for the object based on the weighted sample poses comprises:

estimating a posterior distribution for the object and all previously detected ones of the plurality of objects based on the weighted sample poses.

7. The method of claim 6, wherein the step of estimating a posterior distribution for the object and all previously detected ones of the plurality of objects based on the weighted sample poses comprises:

resampling the weighted sample poses using the importance ratios the weighted sample poses to calculate an unweighted approximation of the posterior distribution for the object and all previously detected ones of the plurality of objects.

8. The method of claim 1, wherein the step of estimating a posterior distribution for the object based on the weighted sample poses comprises:

averaging a certain number of highest weighted sample poses.

9. The method of claim 1, wherein the trained hierarchical detection network specifies a hierarchy of scales for detecting the plurality of objects.

10. The method of claim 1, wherein the step of sequentially detecting a plurality of objects in an image in an order specified by a trained hierarchical detection network comprises:

sequentially detecting the plurality of objects in the image in an order specified by the trained hierarchical detection network using Monte Carlo sampling.

11. The method of claim 1, wherein the order for detecting the plurality of objects is one of set automatically in training of the hierarchical detection network, set manually by a user, and set semi-automatically where a partial order is set manually by a user and the remaining order is set automatically in training of the hierarchical detection network.

12. A method of training a hierarchical detection network for detecting a plurality of objects in an image, comprising:

individually training a plurality of object detectors, each corresponding to one of the plurality of objects, using a first set of annotated training data; and

automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors.

13. The method of claim 12, wherein the step of automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors comprises:

selecting a first one of the trained plurality of object detectors to add to a hierarchical detection network, the first one of the trained plurality of object detectors having a highest score for detecting the corresponding one of the plurality of objects in the second set of annotated training data;

recursively adding a next one of the trained plurality of detectors to connect to a precursor in the hierarchical detection network, by selecting the next one of the trained plurality of detectors and the precursor to maximize an expected value of a score for detecting the corresponding objects in the second set of annotated training data.

14. The method of claim 12, wherein the step of automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors comprises:

automatically determining a detection order and a scale hierarchy for detecting the plurality of objects.

15. An apparatus for detecting multiple objects in an image, comprising:

means for sequentially detecting a plurality of objects in an image in an order specified by a trained hierarchical detection network, wherein the means for sequentially detection comprises;

means for obtaining a plurality of sample poses for an object from a proposal distribution of object poses for the object,

means for weighting each of the plurality of sample poses based on an importance ratio calculated for each sample pose, and

means for estimating a posterior distribution for the object based on the weighted sample poses.

16. The apparatus of claim 15, wherein the means for obtaining a plurality of sample poses for the object from a proposal distribution of object poses for the object comprises:

means for determining the proposal distribution for the object based on one or more detected poses for a previously detected one of the plurality of objects.

17. The apparatus of claim 16, wherein the means for determining the proposal distribution for the based one or more detected poses for a previously detected one of the plurality of objects comprises:

means for applying a Gaussian distribution transition kernel to each of the one or more detected poses of the previously detected one of the plurality of objects, wherein the Gaussian distribution transition kernel is estimated based on training data.

18. The apparatus of claim 15, wherein the means for weighting each of the plurality of sample poses based on an importance ratio calculated for each sample comprises:

means for weighting each of the plurality of sample poses based on an importance ratio calculated for each sample pose using a probability determined for each sample pose by a trained object detector.

19. The apparatus of claim 15, wherein the means for estimating a posterior distribution for the object based on the weighted sample poses comprises:

means for estimating a posterior distribution for the object and all previously detected ones of the plurality of objects based on the weighted sample poses.

20. The apparatus of claim 19, wherein the means for estimating a posterior distribution for the object and all previously detected ones of the plurality of objects based on the weighted sample poses comprises:

means for resampling the weighted sample poses using the importance ratios the weighted sample poses to calculate an unweighted approximation of the posterior distribution for the object and all previously detected ones of the plurality of objects.

21. The apparatus of claim 15, wherein the trained hierarchical detection network specifies a hierarchy of scales for detecting the plurality of objects.

22. An apparatus for training a hierarchical detection network for detecting a plurality of objects in an image, comprising:

means for individually training a plurality of object detectors, each corresponding to one of the plurality of objects, using a first set of annotated training data; and

means for automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors.

23. The apparatus of claim 22, wherein the means for automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors comprises:

means for selecting a first one of the trained plurality of object detectors to add to a hierarchical detection network, the first one of the trained plurality of object detectors having a highest score for detecting the corresponding one of the plurality of objects in the second set of annotated training data;

means for recursively adding a next one of the trained plurality of detectors to connect to a precursor in the hierarchical detection network, by selecting the next one of the trained plurality of detectors and the precursor to maximize an expected value of a score for detecting the corresponding objects in the second set of annotated training data.

24. The apparatus of claim 22, wherein the means for automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors comprises:

means for automatically determining a detection order and a scale hierarchy for detecting the plurality of objects.

25. A non-transitory computer readable medium encoded with computer executable instructions for detecting multiple objects in an image, the computer executable instructions defining steps comprising:

sequentially detecting a plurality of objects in an image in an order specified by a trained hierarchical detection network, wherein the detection of each object in the image comprises;

obtaining a plurality of sample poses for the object from a proposal distribution of object poses for the object,

weighting each of the plurality of sample poses based on an importance ratio calculated for each sample pose, and

estimating a posterior distribution for the object based on the weighted sample poses.

26. The computer readable medium of claim 25, wherein the computer executable instructions defining the step of obtaining a plurality of sample poses for the object from a proposal distribution of object poses for the object comprise computer executable instructions defining the step of:

determining the proposal distribution for the object based on one or more detected poses for a previously detected one of the plurality of objects.

27. The computer readable medium of claim 26, wherein the computer executable instructions defining the step of determining the proposal distribution for the based one or more detected poses for a previously detected one of the plurality of objects comprise computer executable instructions defining the step of:

applying a Gaussian distribution transition kernel to each of the one or more detected poses of the previously detected one of the plurality of objects, wherein the Gaussian distribution transition kernel is estimated based on training data.

28. The computer readable medium of claim 25, wherein the computer executable instructions defining the step of weighting each of the plurality of sample poses based on an importance ratio calculated for each sample comprise computer executable instructions defining the step of:

weighting each of the plurality of sample poses based on an importance ratio calculated for each sample pose using a probability determined for each sample pose by a trained object detector.

29. The computer readable medium of claim 25, wherein the computer executable instructions defining the step of estimating a posterior distribution for the object based on the weighted sample poses comprise computer executable instructions defining the step of:

estimating a posterior distribution for the object and all previously detected ones of the plurality of objects based on the weighted sample poses.

30. The computer readable medium of claim 29, wherein the computer executable instructions defining the step of estimating a posterior distribution for the object and all previously detected ones of the plurality of objects based on the weighted sample poses comprise computer executable instructions defining the step of:

resampling the weighted sample poses using the importance ratios the weighted sample poses to calculate an unweighted approximation of the posterior distribution for the object and all previously detected ones of the plurality of objects.

31. The computer readable medium of claim 25, wherein the trained hierarchical detection network specifies a hierarchy of scales for detecting the plurality of objects.

32. A non-transitory computer readable medium encoded with computer executable instructions for training a hierarchical detection network for detecting a plurality of objects in an image, the computer executable instructions defining steps comprising:

individually training a plurality of object detectors, each corresponding to one of the plurality of objects, using a first set of annotated training data; and

automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors.

33. The computer readable medium of claim 32, wherein the computer executable instructions defining the step of automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors comprise computer executable instructions defining the steps of:

selecting a first one of the trained plurality of object detectors to add to a hierarchical detection network, the first one of the trained plurality of object detectors having a highest score for detecting the corresponding one of the plurality of objects in the second set of annotated training data;

recursively adding a next one of the trained plurality of detectors to connect to a precursor in the hierarchical detection network, by selecting the next one of the trained plurality of detectors and the precursor to maximize an expected value of a score for detecting the corresponding objects in the second set of annotated training data.

34. The computer readable medium of claim 32, wherein the computer executable instructions defining the step of automatically determining a detection order for detecting the plurality of objects using a second set of annotated training data and the trained plurality of object detectors comprise computer executable instructions defining the step of:

automatically determining a detection order and a scale hierarchy for detecting the plurality of objects.