System and method for processing image data

• US 9,213,781 B1
• Filed: 09/19/2013
• Issued: 12/15/2015
• Est. Priority Date: 09/19/2012
• Status: Active Grant

• Current Assignee: Placemeter, LLC
• Original Assignee: Placemeter, LLC
• Inventors: Winter, Alexandre
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/031,998
• Time in Patent Office: 817 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/90 Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The PLACEMETER PLATFORM APPARATUSES, METHODS AND SYSTEMS (“PM-PLATFORM”) transform sensor data and/or feedback via PM-PLATFORM components into notifications, updates, coupons, promotions, transactions and/or activitiesnotifications, updates, coupons, promotions, transactions and/or activities. In one implementation, the PM-PLATFORM comprises a sensor, a memory, and a processor disposed in communication with the sensor and memory, the memory storing processor-issuable instructions to receive raw environment data at a sensor for at least two discrete points in time, analyze the received raw environment data locally to determine an at least one occupancy metric, store the occupancy metric, receive further raw environment data for a further point in time, process the further raw environment data to determine a further occupancy metric, compare the further occupancy metric to at least one previous occupancy metric, and issue a notification based on the comparison.

Claims

1. A system for activity measurement, the system comprising:

a plurality of sensors distributed geographically and configured to communicate via one or more networks;

a network interface coupled to the one or more processor and configured to communicate via the one or more networks;

one or more storage devices coupled to the one or more processors;

a plurality of processing components coupled to the one or more processors and configured to execute an activity measurement process, the plurality of components comprising a global image comparison component configured to,

receive raw signal data from at least one of the plurality of sensors located in an environment at different points in time, wherein the raw signal data comprises,

one or more video streams;

electronic thermometer readings; and

one or more audio streams;

analyze the raw signal data received at each of the different points in time;

generate a plurality of metrics from the raw signal data for each of the different points in time;

analyze the plurality of metrics to determine changes in the environment over time, wherein the changes comprise changes in occupancy of a venue;

changes in wait time to go through a line;

changes in foot traffic in the environment;

historical information regarding the changes in the environment over time; and

predict future changes in the environment over time.

2. The system of claim 1, wherein the raw signal data comprises:

wired video streams;

wireless video streams;

wired audio streams;

wireless audio streams; and

radio frequency data.

3. The system of claim 1, wherein the raw signal data comprises one or more video streams, and wherein analyzing the plurality of metrics to determine changes in the environment over time comprises comparing similar metrics from two images of a same scene from two different points in time.

4. The system of claim 3, wherein the similar metrics comprises gray levels.

5. The system of claim 4, wherein the wherein analyzing the plurality of metrics to determine changes in the environment over time further comprises estimating a gray level probability density function for each of the two images.

6. The system of claim 3, wherein analyzing the plurality of metrics to determine changes in the environment over time further comprises using the probability density functions of the two images to estimate generate a measure of normalized mutual information.

7. The system of claim 1, wherein the raw signal data comprises one or more video streams, and wherein analyzing the plurality of metrics to determine changes in the environment over time comprises comparing similar metrics from two images of a same scene from two different points in time, and wherein the plurality of processing components further comprises a local image comparison map component configured to draw a difference map of difference between the two images.

8. The system of claim 7, wherein the plurality of processing components further comprises a difference map component configured to compute multiple difference maps over a sequence of images captured over time.

9. The system of claim 7, wherein the plurality of processing components further comprises a local analysis component configured to:

store a first image in a sequence as a background image;

detecting a current mask as a difference map between a current image and the background image; and

determining a next mask, comprising defining object tracks as consecutive masks representing a same object moving in a video.

10. The system of claim 9, wherein the local analysis component is further configured to store a list of current object tracks.

11. The system of claim 10, wherein the local analysis component is further configured to:

split a newly created mask into connected components;

compare each connected component to existing objects; and

associate a current connected component with the object that best matches it.

12. The system of claim 11, wherein the local analysis component is further configured to archive an object when the object has not been associated with a connected component of a mask for a predetermined period, wherein a trajectory of an archived object represents the trajectory of the object through an image over time.

13. The system of claim 12, wherein the local analysis component is further configured to analyze multiple archived object trajectories to determine what the archived object trajectories represent.

14. The system of claim 13, wherein the analysis of archived object trajectories yields one or more of the plurality of metrics.

15. The system of claim 14, wherein the local analysis component is further configured to store the plurality of metrics in a database and associate each of the plurality of metrics with data regarding its origination.

16. The system of claim 1, wherein the plurality of processing components is configured to generate an end user view that graphically illustrates information related to the environment, the information comprising,

the changes in the environment over time, wherein the changes comprise changes in occupancy of a venue;

changes in wait time to go through a line;

changes in foot traffic in the environment;

historical information regarding the changes in the environment over time; and

predicted future changes in the environment over time.

17. The system of claim 16, wherein generating the end user view comprises:

formatting the information for display on a plurality of end user devices communicatively coupled to the network interface; and

communicating with the plurality of end user devices to display the information and receive queries regarding the information.

18. A system for activity measurement, the system comprising:

a plurality of sensors distributed geographically and configured to communicate via one or more networks;

a network interface coupled to the one or more processor and configured to communicate via the one or more networks;

one or more storage devices coupled to the one or more processors;

a plurality of processing components coupled to the one or more processors and configured to execute an activity measurement process, the plurality of components comprising a global image comparison component configured to,

receive raw signal data from at least one of the plurality of sensors located in an environment at different points in time, wherein the raw signal data comprises,

wired video streams;

wireless video streams;

wired audio streams;

wireless audio streams;

radio frequency data;

one or more video streams;

electronic thermometer readings; and

one or more audio streams

analyze the raw signal data received at each of the different points in time;

generate a plurality of metrics from the raw signal data for each of the different points in time;

analyze the plurality of metrics to determine changes in the environment over time, wherein the changes comprise changes in occupancy of a venue;

changes in wait time to go through a line;

changes in foot traffic in the environment;

historical information regarding the changes in the environment over time; and

predict future changes in the environment over time.

19. The system of claim 18, wherein the raw signal data comprises one or more video streams, and wherein analyzing the plurality of metrics to determine changes in the environment over time comprises comparing similar metrics from two images of a same scene from two different points in time.

20. The system of claim 19, wherein the similar metrics comprises gray levels.

21. The system of claim 20, wherein the wherein analyzing the plurality of metrics to determine changes in the environment over time further comprises estimating a gray level probability density function for each of the two images.

22. The system of claim 19, wherein analyzing the plurality of metrics to determine changes in the environment over time further comprises using the probability density functions of the two images to estimate generate a measure of normalized mutual information.

23. The system of claim 18, wherein the raw signal data comprises one or more video streams, and wherein analyzing the plurality of metrics to determine changes in the environment over time comprises comparing similar metrics from two images of a same scene from two different points in time, and wherein the plurality of processing components further comprises a local image comparison map component configured to draw a difference map of difference between the two images.

24. The system of claim 23, wherein the plurality of processing components further comprises a difference map component configured to compute multiple difference maps over a sequence of images captured over time.

25. The system of claim 23, wherein the plurality of processing components further comprises a local analysis component configured to:

store a first image in a sequence as a background image;

detecting a current mask as a difference map between a current image and the background image; and

determining a next mask, comprising defining object tracks as consecutive masks representing a same object moving in a video.

26. The system of claim 25, wherein the local analysis component is further configured to store a list of current object tracks.

27. The system of claim 26, wherein the local analysis component is further configured to:

split a newly created mask into connected components;

compare each connected component to existing objects; and

associate a current connected component with the object that best matches it.

28. The system of claim 27, wherein the local analysis component is further configured to archive an object when the object has not been associated with a connected component of a mask for a predetermined period, wherein a trajectory of an archived object represents the trajectory of the object through an image over time.

29. The system of claim 28, wherein the local analysis component is further configured to analyze multiple archived object trajectories to determine what the archived object trajectories represent.

30. The system of claim 29, wherein the analysis of archived object trajectories yields one or more of the plurality of metrics.

31. The system of claim 30, wherein the local analysis component is further configured to store the plurality of metrics in a database and associate each of the plurality of metrics with data regarding its origination.

32. The system of claim 18, wherein the plurality of processing components is configured to generate an end user view that graphically illustrates information related to the environment, the information comprising,

the changes in the environment over time, wherein the changes comprise changes in occupancy of a venue;

changes in wait time to go through a line;

changes in foot traffic in the environment;

historical information regarding the changes in the environment over time; and

predicted future changes in the environment over time.

33. The system of claim 32, wherein generating the end user view comprises:

formatting the information for display on a plurality of end user devices communicatively coupled to the network interface; and

communicating with the plurality of end user devices to display the information and receive queries regarding the information.