Apparatus and method for measuring audience data from image stream using dynamically-configurable hardware architecture

• US 8,165,386 B1
• Filed: 08/18/2009
• Issued: 04/24/2012
• Est. Priority Date: 08/18/2009
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Moon, Hankyu, Irick, Kevin Maurice, NARAYANAN, VIJAYKRISHNAN, Sharma, Rajeev, Jung, Namsoon
• Primary Examiner(s): Bali, Vikkram
• Application Number:
  US12/583,323
• Time in Patent Office: 980 Days
• Field of Search: 382/155, 382/156, 382/157, 382/158, 706/15, 706/16, 706/17
• US Class Current: 382/156
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks

Abstract

The present invention is an embedded audience measurement platform, which is called HAM. The HAM includes hardware, apparatus, and method for measuring audience data from image stream using dynamically-configurable hardware architecture. The HAM provides an end-to-end solution for audience measurement, wherein reconfigurable computational modules are used as engines per node to power the complete solution implemented in a flexible hardware architecture. The HAM is also a complete system for broad audience measurement, which has various components built into the system. Examples of the components comprise demographics classification, gaze estimation, emotion recognition, behavior analysis, and impression measurement.

Claims

1. An apparatus for measuring audience data from at least an image stream using dynamically-configurable hardware architecture, comprising:

a) means for aggregating computational modules using a directed graph,

wherein every vertex of the graph represents a computational module that implements a classifier or a computational component that constitutes a classifier, and every edge of the graph between a first computational module and a second computational module represents a decision of whether or not the processing goes from the first computational model to the second computational module and any data that need to be passed from the first computational module and the second computational module,

b) means for determining an audience measurement hardware architecture and the computation that the architecture carries out, based on the presence and absence of the edges between the vertices and the kind of computation that is represented by the edge in the graphical model,

c) a supervising control to change the hardware configuration in executing the computational modules in the system,

wherein the supervising control monitors the external or internal environment to make a decision whether or not it is necessary to reconfigure the system to adjust to the changes in environment,

wherein the supervising control then makes a decision about how to change the hardware configurations to adapt to the given environment, and

wherein the audience data comprises the behavioral attributes and demographics of the audience per node.

2. The apparatus according to claim 1, wherein the apparatus further comprises means for supporting multiple neural network configurations simultaneously.

3. The apparatus according to claim 1, wherein the apparatus further comprises means for measuring behavior analysis.

4. The apparatus according to claim 1, wherein the apparatus further comprises means for estimating the gaze of the audience.

5. The apparatus according to claim 1, wherein the apparatus further comprises means for recognizing the emotion of the audience.

6. The apparatus according to claim 1, wherein the apparatus further comprises means for classifying the demographics of the audience.

7. The apparatus according to claim 1, wherein the apparatus further comprises at least a part for at least a common computational element,

wherein the computational element is developed based on at least a classification algorithm including neural network and support vector machine, and

wherein the computational element is developed based on at least an image processing algorithm including image filtering.

8. The apparatus according to claim 1, wherein the apparatus further comprises means for reusing at least a computational module to carry out different tasks,

wherein the computational module comprises neural network module and support vector machine module.

9. The apparatus according to claim 1, wherein the apparatus further comprises modules according to a high-level topology for a set of computational modules and image filters,

wherein the attributes for the computational modules and image filters are set up according to at least a specification of at least a task, and

wherein the task comprises behavior measurement, traffic count, dwell time measurement, impression measurement, gaze estimation, emotion change detection, and demographic classification.

10. The apparatus according to claim 1, wherein the apparatus further comprises means for reconfiguring the topology for a set of computational modules and image filters.

11. The apparatus according to claim 1, wherein the apparatus further comprises means for setting different sets of parameters for different tasks for each computational module,

wherein the parameters comprise model parameters and weights according to changes in the architecture of the module.

12. The apparatus according to claim 1, wherein the apparatus further comprises means for using a subset of the computational modules depending on the task.

13. The apparatus according to claim 1, wherein the apparatus further comprises means for statically configuring the hardware platform for a different image processing algorithm comprising face detection and expression recognition,

wherein the static configuration enables spatial diversity where different instances of the same hardware implement a different algorithm.

14. The apparatus according to claim 1, wherein the apparatus further comprises means for configuring the hardware platform for a different algorithm at a different time,

wherein the temporal reconfiguration allows the apparatus a dynamic adaptation to environment.

15. The apparatus according to claim 1, wherein the apparatus further comprises means for dynamic adaptation of implemented algorithm, based on a dynamically-recognized scene.

16. The apparatus according to claim 1, wherein the apparatus further comprises means for dynamic adaptation to resource constraints,

wherein the algorithm implemented on one instance of the hardware is changed based on configuration of adjacent nodes.

17. The apparatus according to claim 1, wherein the apparatus further comprises means for providing a unified schema for inter-module capability and output format sharing,

wherein a support vector machine module configured on the platform publishes to other modules in the system a set of information in a standard scheme.

18. The apparatus according to claim 1, wherein the apparatus further comprises means for providing a boosting framework for any classifier that is resident in the system,

wherein the boosting is an environment in which the modules are instantiated, and

wherein the modules use the outputs of any other modules through the boosting interface to form a cascade of weak classifiers.

19. A method for measuring audience data from at least an image stream using a dynamically-configurable hardware architecture, comprising the steps of:

a) aggregating computational modules using a directed graph,

wherein every vertex of the graph represents a computational module that implements a classifier or a computational component that constitutes a classifier, and every edge of the graph between a first computational module and a second computational module represents a decision of whether or not the processing goes from the first computational module to the second computational module and any data that need to be passed from the first computational module and the second computational module,

b) determining an audience measurement hardware architecture and the computation that the architecture carries out, based on the presence and absence of the edges between the vertices and the kind of the computation that is represented by the edge in the graphical model,

c) utilizing a supervising control to change the hardware configuration in executing the computational modules in the system,

wherein the supervising control monitors the external or internal environment to make a decision of whether or not it is necessary to reconfigure the system to adjust to the changes in environment,

wherein the supervising control then makes a decision about how to change the hardware configurations to adapt to the given environment, and

wherein the audience data comprises behavioral attributes and demographics of the audience per node.

20. The method according to claim 19, wherein the method further comprises a step of constructing modules according to a high-level topology for a set of computational modules and image filters,

wherein the attributes for the computational modules and image filters are set up according to at least a specification of at least a task, and

wherein the task comprises behavior measurement, traffic count, dwell time measurement, impression measurement, gaze estimation, emotion change detection, and demographic classification.