Apparatus and method for hardware implementation of object recognition from an image stream using artificial neural network

• US 8,081,816 B1
• Filed: 06/06/2008
• Issued: 12/20/2011
• Est. Priority Date: 06/06/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Irick, Kevin Maurice, NARAYANAN, VIJAYKRISHNAN, Moon, Hankyu, Sharma, Rajeev, Jung, Namsoon
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US12/157,087
• Time in Patent Office: 1,292 Days
• Field of Search: 382/155, 382/156, 382/157, 382/158, 706/15, 706/27, 706/33
• US Class Current: 382/156
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 10/955 using specific electronic processors

Abstract

The present invention is an apparatus and method for object recognition from at least an image stream from at least an image frame utilizing at least an artificial neural network. The present invention further comprises means for generating multiple components of an image pyramid simultaneously from a single image stream, means for providing the active pixel and interlayer neuron data to at least a subwindow processor, means for multiplying and accumulating the product of a pixel data or interlayer data and a synapse weight, and means for performing the activation of an accumulation. The present invention allows the artificial neural networks to be reconfigurable, thus embracing a broad range of object recognition applications in a flexible way. The subwindow processor in the present invention also further comprises means for performing neuron computations for at least a neuron. An exemplary embodiment of the present invention is used for object recognition, including face detection and gender recognition, in hardware. The apparatus comprises a digital circuitry system or IC that embodies the components of the present invention.

Claims

1. An apparatus for object recognition from at least an image stream from at least an image frame utilizing at least an artificial neural network, comprising:

a) means for generating multiple subwindows of an image of different scales and offsets simultaneously from a single image stream,

b) at least a subwindow filter module that selects pixels belonging to a collection of subwindows that are assigned to the subwindow filter module for a predefined scale,

wherein a collection of subwindow filter modules select pixels for a plurality of scales simultaneously,

c) means for providing active pixel and interlayer neuron data to at least a subwindow processor,

d) means for multiplying and accumulating the product of a pixel data or interlayer neuron data and a synapse weight,

e) means for performing the activation of an accumulation, and

f) means for mapping tasks to multiple instances of subwindow processors,

wherein the artificial neural network is reconfigurable by programming at least a table,

wherein the subwindow processor comprises means for performing neuron computations for at least a neuron, and

wherein the number of instances of the subwindow processor is variable based on resource constraints of the object recognition.

2. The apparatus according to claim 1, wherein the apparatus further comprises means for supporting multiple artificial neural network configurations simultaneously,

wherein each of the multiple artificial neural network configurations is mapped to a subwindow processor, and

wherein a subwindow is declared as interesting or non-interesting as a result of information gained from an optional subwindow filtering process.

3. The apparatus according to claim 1, wherein the apparatus further comprises means for changing structure of the artificial neural network dynamically at run-time,

wherein the structure comprises synapse connections of the artificial neural network, and

wherein a subwindow logically encompasses neurons, synapse connections, and synapse weights, belonging to an instance of the artificial neural network.

4. The apparatus according to claim 1, wherein the apparatus further comprises means for reusing the neuron computation resources across different neurons of an artificial neural network to reduce resource utilization with respect to a direct implementation of the artificial neural network.

5. The apparatus according to claim 1, wherein the apparatus further comprises means for programming weight and threshold values to alter the computation performed by the artificial neural network.

6. The apparatus according to claim 1, wherein the apparatus further comprises means for augmenting available processing state memory with auxiliary memory to support a plurality of instances of an artificial neural network within a processing element.

7. The apparatus according to claim 1, wherein the apparatus further comprises means for providing application-specific region of interest information to reduce the number of subwindows and processing requirements.

8. The apparatus according to claim 1, wherein the apparatus further comprises means for maintaining computation state of at least a neuron comprising:

a) accumulation value,

b) the number of remaining accumulations, and

c) a target neuron to which the accumulation result is forwarded.

9. The apparatus according to claim 1, wherein the apparatus further comprises means for emulating the artificial neural network to classify rectangular subwindows from the image frame,

wherein a subwindow is the smallest 2D fixed sized array of pixels that can be processed by the artificial neural network, and

wherein the artificial neural network is iteratively applied to each subwindow of interest.

10. The apparatus according to claim 1, wherein the apparatus further comprises means for maintaining column, row, and subwindow counts for the image subwindows to which the subwindow filter module is assigned.

11. The apparatus according to claim 1, wherein the apparatus further comprises means for identifying which of the assigned subwindows are of interest and should subsequently be forwarded to a subwindow processor.

12. The apparatus according to claim 1, wherein the apparatus further comprises means for mapping the assigned subwindows to the subwindow processor assigned to receive pixel data belonging to the subwindow.

13. The apparatus according to claim 1, wherein the apparatus further comprises at least a subwindow filter module that forms a pixel data packet for each non-discarded pixel,

wherein the pixel data packet consists of a data field and a header field, and

wherein the data field consists of the pixel, subwindow identification, and source identification, and

the header field minimally consists of the subwindow processor identification which facilitates routing the packet to the proper subwindow processor via the interconnection network.

14. A method for object recognition from at least an image stream from at least an image frame utilizing at least an artificial neural network, comprising the steps of:

a) generating multiple subwindows of an image of different scales and offsets simultaneously from a single image stream,

b) utilizing at least a subwindow filter module that selects pixels belonging to a collection of subwindows that are assigned to the subwindow filter module for a predefined scale,

wherein a collection of subwindow filter modules select pixels for a plurality of scales simultaneously,

c) providing active pixel and interlayer neuron data to at least a subwindow processor,

d) multiplying and accumulating the product of a pixel data or interlayer data and a synapse weight,

e) performing the activation of an accumulation, and

f) mapping tasks to multiple instances of subwindow processors,

wherein the artificial neural networks is reconfigurable by programming at least a table,

wherein the subwindow processor comprises means for performing neuron computations for at least a neuron, and

wherein the number of instances of the subwindow processor is variable based on resource constraints of the object recognition.

15. The method according to claim 14, wherein the method further comprises a step of supporting multiple artificial neural network configurations simultaneously,

wherein each of the multiple artificial neural network configurations is mapped to a subwindow processor, and

wherein a subwindow is declared as interesting or non-interesting as a result of information gained from an optional subwindow filtering process.

16. The method according to claim 14, wherein the method further comprises a step of changing structure of the artificial neural network dynamically at run-time,

wherein the structure comprises synapse connections of the artificial neural network, and

wherein a subwindow logically encompasses neurons, synapse connections, and synapse weights, belonging to an instance of the artificial neural network.

17. The method according to claim 14, wherein the method further comprises a step of reusing the neuron computation resources across different neurons of an artificial neural network to reduce resource utilization with respect to a direct implementation of the artificial neural network.

18. The method according to claim 14, wherein the method further comprises a step of programming weight and threshold values to alter the computation performed by the artificial neural network.

19. The method according to claim 14, wherein the method further comprises a step of augmenting available processing state memory with auxiliary memory to support a plurality of instances of an artificial neural network within a processing element.

20. The method according to claim 14, wherein the method further comprises a step of providing application-specific region of interest information to reduce the number of subwindows and processing requirements.

21. The method according to claim 14, wherein the method further comprises a step of maintaining computation state of at least a neuron comprising:

a) accumulation value,

b) the number of remaining accumulations, and

c) a target neuron to which the accumulation result is forwarded.

22. The method according to claim 14, wherein the method further comprises a step of emulating the artificial neural network to classify rectangular subwindows from the image frame,

wherein a subwindow is the smallest 2D fixed sized array of pixels that can be processed by the artificial neural network, and

wherein the artificial neural network is iteratively applied to each subwindow of interest from the image pyramid.

23. The method according to claim 14, wherein the method further comprises a step of maintaining column, row, and subwindow counts for the image subwindows to which the subwindow filter module is assigned.

24. The method according to claim 14, wherein the method further comprises a step of identifying which of the assigned subwindows are of interest and should subsequently be forwarded to a subwindow processor.

25. The method according to claim 14, wherein the method further comprises a step of mapping the assigned subwindows to the subwindow processor assigned to receive pixel data belonging to the subwindow.

26. The method according to claim 14, wherein the method further comprises a step of utilizing at least a subwindow filter module that forms a pixel data packet for each non-discarded pixel,

wherein the pixel data packet consists of a data field and a header field, and

wherein the data field consists of the pixel, subwindow identification, and source identification, and

wherein the header field minimally consists of the subwindow processor identification which facilitates routing the packet to the proper subwindow processor via the interconnection network.