Indirect monitoring of device usage and activities

• US 8,407,028 B2
• Filed: 03/16/2011
• Issued: 03/26/2013
• Est. Priority Date: 12/31/2003
• Status: Active Grant

• Current Assignee: Reality Analytics, Inc. (Renesas Electronics Corporation)
• Original Assignee: Jeffrey M Sieracki
• Inventors: Sieracki, Jeffrey M.
• Primary Examiner(s): Bui, Bryan
• Application Number:
  US13/049,867
• Publication Number:
  US 20110224950A1
• Time in Patent Office: 741 Days
• Field of Search: 702/160, 702/189, 600/301, 600/511, 607/59
• US Class Current: 702/189
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34

Abstract

Signal characteristics, or signatures, defined by one or more forms of energy being transferred during the commission of an activity are captured in dimensionally-reduced numerical sequences. Dimensionality reduction is achieved such that reduced data acquired during a detection phase can be directly compared with such reduced data produced during system training. Activities, events, human identities and so on can be identified through such direct comparison. Dimensionality reduction, such as through sparse approximation or simultaneous sparse approximation, may produce combinations of scaled prototype functions. Such combinations or their parametric representations compactly describe the signal characteristics for purposes of discovering new activity signatures, of extracting test signals from a set of measurements and of comparing sets for purposes of detection and classification.

Claims

1. A monitoring apparatus to monitor a region of interest for activity, the apparatus comprising:

a plurality of sensor channels to produce numerical sequences proportional to energy of at least two energy modalities transferred to respective sensors thereof during commission of an activity; and

a processor to determine whether a known activity is committed by matching dimensionally-reduced representations of the numerical sequences with dimensionally-reduced representations of known numerical sequences containing signal characteristics defined by energy transference in the commission of the known activity, and to provide an indication of the known activity upon positive determination that the match is within a similarity criterion.

2. The apparatus as recited in claim 1, wherein the processor generates the dimensionally-reduced representations of the numerical sequences as combinations of scaled prototype functions and constructs therefrom representation data structures to include an indication of the respective combinations.

3. The apparatus as recited in claim 2, further comprising:

a database of signature data structures, each including an indication of the combinations of scaled prototype functions representing the known numerical sequences; and

wherein

the processor determines a similarity measure between the combinations of scaled prototype functions indicated in the representation data structures and the combinations of scaled prototype functions indicated in the signature data structures such that the positive determination of the match is indicated by the similarity measure meeting the similarity criterion.

4. The apparatus as recited in claim 3, wherein the signature data structures include respective indications of multiple sets of the combinations of scaled prototype functions, each of the sets representing respective known numerical sequences defined by the energy transference during the known activity, the processor indicating the match upon the similarity measure between the combinations of scaled prototype functions indicated in the representation data structures and the sets of the combinations indicated in the signature data structures meeting the predetermined similarity criteria.

5. The apparatus as recited in claim 4, wherein the processor generates the combinations of scaled prototype functions as sets thereof, each of the combinations in the set being generated from the numerical sequences originating from the sensor channels of at least two distinct energy modalities associated with the commission of the activity.

6. The apparatus as recited in claim 5, wherein the processor generates at least one commonly-defined dimensionally-reduced representation from the numerical sequences originating from the sensor channels of the at least two distinct energy modalities.

7. The apparatus as recited in claim 6, wherein the processor executes a machine implementation of a simultaneous sparse approximator.

8. The apparatus as recited in claim 2, wherein the processor generates at least one commonly-defined dimensionally-reduced representation from the numerical sequences originating from a plurality of sensor channels of a like energy modality.

9. The apparatus as recited in claim 8, wherein the processor executes a machine implementation of a simultaneous sparse approximator.

10. The apparatus as recited in claim 3, wherein the processor revises the dimensionally-reduced representation of the known numerical sequences in the signature data structures with the signal characteristics contained in the dimensionally-reduced representations of the numerical sequences in the representation data structures upon predetermined revision criteria being met.

11. The apparatus as recited in claim 10, wherein the revision criteria include the positive determination of the match.

12. The apparatus as recited in claim 10, wherein the revision criteria include a positive determination that the similarity measure is within a predetermined range.

13. The apparatus as recited in claim 1, further comprising:

a housing to contain the sensor channels and the processor.

14. A machine-implemented method for monitoring a region of interest for activity, the method comprising:

forming at least one signature representation containing signature signal characteristics defined by energy transference in committing a known activity;

converting energy transferred during commission of an activity into signals having activity signal characteristics;

comparing at least one representation of the activity signal characteristics with the signature representations to obtain a similarity measure between the activity signal characteristics and the signature signal characteristics, where at least one of the signature representation and the activity signal representation is dimensionally-reduced from an original signal representation domain prior to the comparison; and

reporting the activity as the known activity upon a positive determination that the similarity measure meets a predetermined criterion.

15. The method as recited in claim 14, wherein the forming of the at least one signature representation includes:

establishing a dictionary of prototype functions; and

forming the signature representation as at least one combination of scaled prototype functions from the dictionary.

16. The method as recited in claim 15, wherein the comparing of the representations includes:

generating the dimensionally-reduced activity representation as at least another combination of scaled prototype functions from the dictionary; and

comparing directly the dimensionally-reduced signature representation and the dimensionally-reduced activity representation to obtain the similarity measure.

17. The method as recited in claim 16 further comprising:

storing the combination of scaled prototype functions and the other combination of scaled prototype functions as a data set including indexes into the dictionary of the prototype functions comprising the combination and respective scaling factors applied to the prototype functions of the combination; and

wherein the comparing of the dimensionally-reduced signature representation and the dimensionally-reduced activity representation includes;

comparing directly the indexes and the scaling factors of the data set representing the dimensionally-reduced signature representation and the dimensionally-reduced activity representation to obtain the similarity measure.

18. The method of claim 14 further comprising:

distributing sensors at locations in the region of interest to intercept energy of at least two energy modalities; and

converting the energy intercepted by the sensors into the signals having the activity signal characteristics.

19. The method as recited in claim 18 further comprising:

establishing numerical sequences as the signature representations of the signature signal characteristics of the at least two energy modalities;

forming an array of the numerical sequences;

generating a joint signature representation of the signature signal characteristics as a dimensionally-reduced representation of the array;

generating numerical sensor sequences from the sensor signals;

forming another array from the sensor sequences;

generating a joint activity representation of the activity signal characteristics from the sensor signals as a dimensionally-reduced representation of the other array; and

comparing directly the dimensionally-reduced array and the dimensionally-reduced other array to obtain the similarity measure.

20. The method as recited in claim 14, wherein the generating of the dimensionally-reduced signature signal characteristics and the generating of the dimensionally-reduced activity signal characteristics includes:

executing a machine implementation of a simultaneous sparse approximator to generate the dimensionally-reduced signature signal characteristics and the dimensionally-reduced activity signal characteristics.

21. A machine-implemented method of monitoring a region of interest for activity, the method comprising:

generating training signals to include signal characteristics indicative of commission of a known activity;

collecting sensor signals by each of a plurality of sensors in the region of interest, the sensors being constructed to respond to energy transference in a plurality of energy modalities;

generating representations of the sensor signals of the energy modalities by which similarities thereof to representations of the signal characteristics in the training signals are comparable;

determining whether the signal characteristics are present in the sensor signals from a similarity measure obtained from the representation of the sensor signals and the representation of the signal characteristics of the training signals corresponding to the energy modalities, where at least one of the representations of the signal characteristics of the training signals and of the sensor signals are dimensionally-reduced from that of an original signal representation domain thereof prior to the similarity measure being applied; and

indicating commission of the activity in the region of interest upon a positive determination that the similarity measure meets a similarity criterion.

22. The method as recited in claim 21 further comprising:

establishing at least one dictionary of prototype functions;

dimensionally-reducing the at least one of the representations of the training signals and the sensor signals through simultaneous sparse approximation based on the dictionary.

23. The method as recited in claim 22, wherein the establishing of the dictionary includes:

establishing a plurality of the dictionaries; and

the dimensionally-reducing the at least one of the representations of the training signals and the sensor signals includes;

dimensionally-reducing the representations of the training signals through simultaneous sparse approximation based on one of the dictionaries; and

generating the dimensionally-reduced numeric representations of the sensor signals through simultaneous sparse approximation based on another of the dictionaries such that the similarity measure meeting the similarity criterion is indicative of the commission of the known activity.

24. The method as recited in claim 21 further comprising:

dimensionally-reducing the signal characteristics of the training signals into a reduced representation domain; and

wherein the generating of the representations of the sensor signals includes;

forming a numerical signal vector from at least one of the sensor signals of at least one of the energy modalities; and

dimensionally-reducing the signal vector into the reduced representation domain.

25. The method as recited in claim 24, wherein the forming of the signal vector includes:

monitoring the sensor signals for a marker signal characteristic; and

forming the signal vector from the at least one sensor signal in accordance with a relationship between the commission of the known activity and the marker signal characteristic.

26. The method of claim 25 further comprising:

receiving a data stream containing numerical data corresponding to the sensor signals; and

forming the signal vector from the data stream responsive to the marker signal characteristics by;

establishing a window to encompass a segment of the data stream, the window having a length corresponding to a length of the signal vector; and

applying the window to the data stream to select the segment thereof as the signal vector.

27. The method as recited in claim 25, wherein the forming of the signal vector includes:

forming a plurality of the signal vectors from the plurality of sensor signals in accordance with respective relationships between the commission of the known activity and the marker signal characteristic; and

aligning the signal vectors in accordance with the respective relationships.

28. The method as recited in claim 27 further comprising:

establishing numerical sequences from the signal characteristics of the training signals for the respective energy modalities;

aligning the numerical sequences into an array;

generating a joint signature representation of the signal characteristics of the training signals as a dimensionally-reduced representation of the array;

aligning the signal vectors into another array;

generating a joint activity representation of the sensor signals as a dimensionally-reduced representation of the other array; and

the determining of whether the signal characteristics are present in the sensor signals includes;

comparing the dimensionally-reduced array and the dimensionally-reduced other array to obtain the similarity measure.

29. The method as recited in claim 28 further comprising:

establishing at least one dictionary of prototype functions;

generating the dimensionally-reduced array and the dimensionally-reduced other array through simultaneous sparse approximation based on the dictionary.

30. The method as recited in claim 29, wherein the establishing of the dictionary includes:

establishing a plurality of the dictionaries; and

the generation of the dimensionally-reduced array and the dimensionally-reduced other array includes;

generating the dimensionally-reduced array through simultaneous sparse approximation based on one of the dictionaries; and

generating the dimensionally-reduced other array through simultaneous sparse approximation based on another of the dictionaries.

31. The method as recited in claim 25 further comprising:

identifying the signal characteristics in the generated training signals;

identifying parameters that configure a matched filter in the original representation domain to generate an output signal having an amplitude responsive to a signal provided thereto the maximum of which is generated when such signal contains the signal characteristics therein.

32. The method as recited in claim 31, wherein the monitoring of the sensor signals includes:

providing at least one of the sensor signals in the original representation domain to the matched filter;

monitoring the amplitude of the output signal of the matched filter; and

generating the marker signal characteristic upon a positive determination that the amplitude of the output signal of the matched filter exceeds a threshold.

33. The method as recited in claim 31 further comprising:

storing the parameters of the matched filter as the dimensionally-reduced representations of the training signals.

34. The method as recited in claim 21, wherein the generating of the training signals includes:

distributing a plurality of training sensors in a spatial distribution;

committing the known activity such that the energy transferred thereby is intercepted by the training sensors; and

identifying the signal characteristics indicative of the commission of the known activity from signals generated by the training sensors; and

further comprising;

distributing the sensors in a spatial distribution in the region of interest;

refining the representation of the signal characteristics obtained from the distributed training sensors upon a positive determination that the similarity measure meets the similarity criterion.

35. The method as recited in claim 34, wherein the refining includes:

excluding a signal characteristic from at least one of the energy modalities upon a positive determination that the representation of the remaining signal characteristics is refined in accordance with a refinement criterion.

36. The method as recited in claim 35 further comprising:

removing a sensor corresponding to the excluded signal characteristic from the distribution of sensors in the region of interest.

37. The method as recited in claim 34, wherein the distributing of the training sensors includes:

including in the distribution at least two of the training sensors to intercept the energy of a like energy modality; and

the distribution of the sensors includes;

including in the distribution at least two of the sensors to intercept the energy of the like energy modality.

38. The method as recited in claim 21 further comprising:

establishing as the similarity criterion a Euclidean distance metric in a dimensionally-reduced signal representation domain into which the at least one of the representations of the training signals and the sensor signals are dimensionally-reduced.

39. The method as recited in claim 21, wherein the known activity includes that which identifies an actor in the region of interest.

40. The method as recited in claim 21, wherein the known activity includes that which quantifies a number of actors in the region of interest.

41. The method as recited in claim 21, wherein the signal characteristics in the training signals include that by which a location of the activity in the region of interest is identified from the activity signal characteristics.