Method and apparatus for spawning specialist belief propagation networks

• US 8,644,599 B2
• Filed: 05/20/2013
• Issued: 02/04/2014
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: Golden Edge Holding Corporation
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Harandi, Siamak
• Application Number:
  US13/897,470
• Publication Number:
  US 20130259311A1
• Time in Patent Office: 260 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A method and apparatus for processing image data is provided. The method includes the steps of employing a main processing network for classifying one or more features of the image data, employing a monitor processing network for determining one or more confusing classifications of the image data, and spawning a specialist processing network to process image data associated with the one or more confusing classifications.

Claims

1. A method for processing image data, comprising the steps of:

employing a main belief propagation network for classifying one or more features of the image data;

employing a monitor belief propagation network for determining one or more classifications of the image data; and

spawning a specialist belief propagation network to process image data associated with the one or more determined classifications.

2. The method of claim 1, wherein the main belief propagation network, monitor belief propagation network, and specialist belief propagation network comprise restricted Boltzmann Machines.

3. The method of claim 1, wherein the specialist belief propagation network is spawned by the monitor belief propagation network.

4. The method of claim 1, further comprising the steps of:

employing the monitor belief propagation network for determining one or more classifications of the image data by the specialist belief propagation network; and

spawning a second specialist belief propagation network to process image data associated with the one or more determined classifications of the image data processed by the specialist belief propagation network.

5. The method of claim 4 wherein the second specialist belief propagation network further spawns a third or more specialist belief propagation networks.

6. The method of claim 1, wherein image data processed by the specialist belief propagation-network is provided to the main belief propagation network directly or via analysis and updates by the monitor belief propagation network.

7. The method of claim 1, wherein the classifications comprise one or more background objects.

8. The method of claim 1, wherein a single (x,y) pixel location may comprise classifications of multiple depth (z) objects.

9. The method of claim 7, wherein one or more of a foreground object, an occluded object and a background object may occupy a single (x,y) pixel location.

10. The method of claim 1, wherein upon implementation of the specialist subnet, error rates associated with the image data associated with the one or more classifications are reduced.

11. The method of claim 1, wherein the classifications comprise one or more scene objects, one or more salient scene features being used to train one or more of the main belief propagation networks and specialist belief propagation network.

12. The method of claim 1, wherein the step of classifying the one or more features of the image data further comprises the steps of:

classifying one or more scene exposure settings by the main belief propagation network according to one of scene exposure and scene quality;

spawning one or more specialist belief propagation networks, each adapted to process image data related to one or more of the one or more scene exposure setting classifications;

wherein each of the one or more specialist belief propagation networks is adapted to modify one or more exposure settings associated with the associated classification of exposure settings, thereby improving the exposure conditions thereof.

13. The method of claim 12, wherein the specialist belief propagation network is also adapted to modify one or more of chroma settings, saturation settings, color relevant settings and texture relevant settings.

14. The method of claim 1, wherein the classifications allow for the performance of disparity space decomposition via disparity decomposition metrics that change thresholds adaptively with disparity values.

15. The method of claim 14 wherein the performance of disparity space decomposition further comprises the step of extraction of “

energy nodules”

that are used in training one or more of the specialist belief propagation networks on various scene settings.

16. The method of claim 1, further comprising the step of computing a volatility index associated with the computation of a segment'"'"'s disparity in accordance with training one or more specialist belief propagation networks in accordance with one or more scene analysis and stability metrics.

17. The method of claim 1, wherein a more comprehensive topology is employed comprising a plurality of subnets branching from the main subnet to deal with various modalities including one or more of gesture and facial expression.

18. The method of claim 17, wherein one or more additional specialist subnets are spawned in the presence of an altogether different modality, unrelated to either facial or gesture expression recognition.

19. The method of claim 1, wherein the image data comprises data from a pair of stereo cameras sensitive to visible light, and data from a pair of stereo cameras sensitive to infrared light.

20. The method of claim 19, further comprising the step of determining whether to employ image data from the pair of stereo cameras sensitive to visible light or the pair of stereo cameras sensitive to infrared light in accordance with ambient lighting conditions.

21. The method of claim 1, further comprising the steps of:

extracting a feature set from each of a plurality of candidate regions;

inputting this extracted feature set to a facial expression recognition system; and

providing an output from the facial recognition system to a monitor system, the monitor system assessing the accuracy of face detection.

22. The method of claim 1, further comprising the steps of:

spawning a first specialist belief propagation network that is responsible for gesture recognition;

spawning a second specialist belief propagation network that is responsible for facial expression detection and recognition; and

spawning a third type of belief propagation network specializing in yet unclassified modalities;

the monitor belief propagation network managing the a first specialist belief propagation network, the second specialist belief propagation network and the third type of belief propagation network.

23. The method of claim 1, further comprising the step of deleting the spawned specialist belief propagation network should its presence in the network become unnecessary, as determined by the monitor subnet.

24. A system for processing image data, comprising:

an image acquisition apparatus for acquiring image data;

a processor employing a main belief propagation network for classifying one or more features of the image data, employing a monitor belief propagation network for determining one or more classifications of the image data, spawning a specialist belief propagation network to process image data associated with the one or more determined classifications, and spawning new belief propagation networks to classify new modalities and scenarios.

25. The system of claim 24, wherein the main belief propagation network, monitor belief propagation network, and specialist belief propagation network comprise restricted Boltzmann Machines.

26. The system of claim 24, wherein the specialist belief propagation network is spawned in accordance with the monitor belief propagation network.

27. The system of claim 24, wherein the processor further employs the monitor belief propagation network for determining one or more classifications of the image data by the specialist belief propagation network, and spawns a second specialist belief propagation network to process image data associated with the one or more classifications of the image data processed by the specialist belief propagation network.

28. The system of claim 24, wherein the image acquisition apparatus further comprises a pair of stereo cameras sensitive to visible light, and data from a pair of stereo cameras sensitive to infrared light.

29. The system of claim 24, wherein the processor further determines whether to employ image data from the pair of stereo cameras sensitive to visible light or the pair of stereo cameras sensitive to infrared light in accordance with ambient lighting conditions.

30. The system of claim 24, wherein the processor further extracts a feature set from each of a plurality of candidate regions, inputs this extracted feature set to a facial expression recognition system, and provides an output from the facial recognition system to a monitor system, the monitor system assessing the accuracy of face detection.

31. The system of claim 24, wherein the processor further spawns a first specialist belief propagation network that is responsible for gesture recognition, and spawns a second specialist belief propagation network that is responsible for facial expression detection and recognition, wherein the monitor belief propagation network manages the a first specialist belief propagation network and the second specialist belief propagation network.

32. A computer program stored to a non-transitory computer readable storage medium, the computer program, when operated, having instructions for causing a multi-purpose computer to perform the steps of:

employing a main belief propagation network for classifying one or more features of the image data;

employing a monitor belief propagation network for determining one or more classifications of the image data; and

spawning a specialist belief propagation network to process image data associated with the one or more determined classifications.

33. The computer program of claim 32, wherein the main belief propagation network, monitor belief propagation network, and specialist belief propagation network comprise restricted Boltzmann Machines.

34. The computer program of claim 32, wherein the specialist belief propagation network is spawned by the monitor belief propagation network.

35. The computer program of claim 32, further comprising instructions to cause the multi-purpose computer to perform the steps of:

employing the monitor belief propagation network for determining one or more classifications of the image data by the specialist belief propagation network; and

spawning a second specialist belief propagation network to process image data associated with the one or more determined classifications of the image data processed by the specialist belief propagation network.

36. The computer program of claim 32, wherein the image data comprises data from a pair of stereo cameras sensitive to visible light, and data from a pair of stereo cameras sensitive to infrared light.

37. The computer program of claim 36, further comprising the step of determining whether to employ image data from the pair of stereo cameras sensitive to visible light or the pair of stereo cameras sensitive to infrared light in accordance with ambient lighting conditions.

38. The computer program of claim 32, further comprising the steps of:

extracting a feature set from each of a plurality of candidate regions;

inputting this extracted feature set to a facial expression recognition system; and

providing an output from the facial recognition system to a monitor system, the monitor system assessing the accuracy of face detection.

39. The computer program of claim 32, further comprising the steps of:

spawning a first specialist belief propagation network that is responsible for gesture recognition;

spawning a second specialist belief propagation network that is responsible for facial expression detection and recognition; and

spawning a third type of specialist belief propagation network specializing in a yet unclassified set of cases,

the monitor belief propagation network managing a first specialist belief propagation network and the second specialist belief propagation network.