Apparatus and method for removing non-discriminatory indices of an indexed dataset

• US 20070009160A1
• Filed: 12/18/2003
• Published: 01/11/2007
• Est. Priority Date: 12/19/2002
• Status: Active Grant

• Current Assignee: Drexel University
• Original Assignee: Drexel University
• Inventors: Hrebien, Leonid, Kam, Moshe, Loo, Lit-Hsin
• Granted Patent: US 8,010,296 B2
• Time in Patent Office: Days
• US Class Current: 382/225
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing
• CPC Class Codes: G06F 2218/04 Denoising
• CPC Class Codes: G16B 25/00 ICT specially adapted for hybridisation; ICT specially adapted for gene or protein expression
• CPC Class Codes: G16B 25/10 Gene or protein expression profiling; Expression-ratio estimation or normalisation
• CPC Class Codes: G16B 40/00 ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
• CPC Class Codes: G16B 40/20 Supervised data analysis
• CPC Class Codes: G16B 40/30 Unsupervised data analysis
• CPC Class Codes: Y10T 436/24 Nuclear magnetic resonance, electron spin resonance or other spin effects or mass spectrometry

Abstract

The present invention provides a device and method for removing non-discriminatory indices of an indexed dataset using ensemble statistics analysis. The device may include a data removal module (320) for removing non-discriminatory indices. For example, the data removal module (320) may comprise a common characteristic removal module and/or a noise removal module. In addition, the data analyzer (300) may comprise a normalization means (310) for normalizing the indexed data. The method of the present invention comprises the steps of identifying and removing portions of the set of data having insufficient discriminatory power based on ensemble statistics of the set of indexed data. For example, the method may include the steps of identifying and removing common characteristics and/or noise portions of the set of indexed data. In addition, the method may comprise the step of normalizing the indexed data either prior to or after the step of removing portions of the set of data.

Claims

1. A data analyzer for use with a pattern classifier to compress a set of indexed data, comprising a data removal module for identifying and removing portions of the set of indexed data having insufficient discriminatory power based on the ensemble statistics of the set of indexed data.

2. The data analyzer according to claim 1, wherein the data removal module comprises a common characteristic removal module comprising means for identifying and removing common characteristics of the set of indexed data based on the ensemble statistics of the set of indexed data.

3. The data analyzer according to claim 1, wherein the data removal module comprises a noise removal module comprising means for identifying and removing noise portions of the set of indexed data based on ensemble statistics of the set of indexed data.

4. The data analyzer according to claim 3, wherein the data removal module comprises a common characteristic removal module comprising means for identifying and removing common characteristics of the set of indexed data based on the ensemble statistics of the set of indexed data.

5. The data analyzer according to claim 4, comprising a normalization means for normalizing the indexed data.

6. The data analyzer according to claim 5, wherein the normalization means is configured to process the indexed data prior to processing by the common characteristic removal module.

7. The data analyzer according to claim 5, wherein the normalization means is configured to process the indexed data after processing by the common characteristic removal module.

8. The data analyzer according to claim 5, wherein the normalization means comprises means for normalizing a member of the set to the standard deviation of the member.

9. The data analyzer according to claim 5, wherein the normalization means comprises means for normalizing a member of the set to the maximum value of the member.

10. The data analyzer according to claim 4, wherein the set of indexed data comprises indexed control-data and the common characteristic removal module comprises means for analyzing the indexed control-data to identify the portions of the set of indexed data that contain common characteristics.

11. The data analyzer according to claim 4, wherein the common characteristic removal module comprises a threshold means for identifying the portions of the indexed data that contain common characteristics.

12. The data analyzer according to claim 11, wherein the threshold means calculates the threshold relative to an ensemble statistic of the set of indexed data.

13. The data analyzer according to claim 11, wherein the threshold means comprises means for removing an index from the indexed data having an ensemble variance higher than the threshold value.

14. The data analyzer according to claim 4, wherein the noise removal module comprises a threshold means for identifying the portions of the indexed data that contain noise.

15. The data analyzer according to claim 14, wherein the threshold means calculates the threshold relative to the ensemble variance of the set of indexed data.

16. The data analyzer according to claim 14, wherein the threshold means comprises means for removing an index from the indexed data having an ensemble variance lower than the threshold value.

17. The data analyzer according to claim 4, wherein the common characteristic removal module comprises means for decreasing the cardinality of the set of indexed data.

18. The data analyzer according to claim 17, wherein the means for reducing cardinality comprises means for removing a portion of the data from a member of the indexed set.

19. The data analyzer according to claim 4, wherein the noise removal module comprises means for decreasing the cardinality of the set of indexed data.

20. The data analyzer according to claim 19, wherein the means for reducing cardinality comprises means for removing a portion of the data from a member of the indexed set.

21. The data analyzer according to any one of claims 1-20, comprising:

a feature extraction module for extracting a feature portion from the compressed indexed data to provide a set of feature indexed data; and

a classification module for classifying the feature indexed data to provide pattern classification of the set of indexed data.

22. A data analyzer for use with a pattern classifier to compress a set of indexed data having common characteristics and noise, comprising:

a. means for determining a common characteristic threshold for the indexed data set;

b. means for removing indices having ensemble statistics higher than the common characteristic threshold value to provide a retained dataset;

c. means for calculating ensemble statistics of each retained index in the retained dataset;

d. means for determining a noise threshold;

e. means for removing indices from the retained dataset having an ensemble statistic lower than a noise threshold value; and

f. means for normalizing the indexed data.

23. The data analyzer according to claim 22, wherein the normalization means is configured to process the indexed data prior to processing by the common characteristic threshold means.

24. The data analyzer according to claim 22, wherein the normalization means is configured to process the indexed data after processing by the common characteristic threshold means.

25. The data analyzer according to claim 22, wherein the normalization means comprises means for normalizing a member of the set to the standard deviation of the member.

26. The data analyzer according to claim 22, wherein the normalization means comprises means for normalizing a member of the set to the maximum value of the member.

27. A method for filtering spectral data from a set of spectra to remove common characteristics and noise, comprising:

identifying and removing the common characteristics from each spectrum within the set of spectra based on the ensemble statistics of the set of spectra, and

identifying and removing the noise portions of each spectrum based on ensemble statistics of the set of spectra,

whereby a filtered spectra is provided in which the common characteristic and noise portions have been removed.

50. The method according to any one of claims 27-49, comprising the steps of:

extracting a feature portion from the compressed indexed data to provide a set of feature indexed data; and

classifying the feature indexed data to provide pattern classification of the set of indexed data.

28. A method for analyzing a set of indexed data to compress the set of data, comprising the steps of identifying and removing portions of the set of data having insufficient discriminatory power based on ensemble statistics of the set of indexed data, thereby providing a set of compressed indexed data.

29. The method according to claim 28, wherein the step of identifying and removing portions of the set of data comprises identifying and removing common characteristics of the set of data based on ensemble statistics of the set of indexed data.

30. The method according to claim 28, wherein the step of identifying and removing portions of the set of data comprises identifying and removing noise portions of the set of indexed data based on ensemble statistics of the set of indexed data.

31. The method according to claim 30, wherein the step of identifying and removing portions of the set of data comprises identifying and removing common characteristics of the set of data based on ensemble statistics of the set of indexed data.

32. The method according to claim 31, comprising the step of normalizing the indexed data.

33. The method according to claim 32, wherein the step of normalizing is performed prior to the step of identifying and removing common characteristics.

34. The method according to claim 32, wherein the step of normalizing is performed after the step of identifying and removing common characteristics.

35. The method according to claim 32, wherein the step of normalizing comprises normalizing a member of the set to the standard deviation of the member.

36. The method according to claim 32, wherein the step of normalizing comprises normalizing a member of the set to the maximum value of the member.

37. The method according to claim 32, wherein the set of indexed data comprises indexed control-data and the step of identifying and removing common characteristics comprises analyzing the indexed control-data to identify the portions of the set of indexed data that contain common characteristics.

38. The method according to claim 32, wherein the step of identifying and removing common characteristics comprises identifying the portions of the indexed data that contain common characteristics based on comparison to a threshold value.

39. The method according to claim 38, wherein the threshold is calculated based on an ensemble statistic of the set of indexed data.

40. The method according to claim 38, wherein the step of identifying and removing common characteristics comprises removing an index from the indexed data having an ensemble variance higher than the threshold value.

41. The method according to claim 31, wherein the step of identifying and removing the noise portions comprises identifying the portions of the indexed data that contain noise based on comparison to a threshold value.

42. The method according to claim 41, wherein the threshold is calculated based on an ensemble statistic of the set of indexed data.

43. The method according to claim 41, wherein the step of identifying and removing the noise portions comprises removing an index from the indexed data having an ensemble variance lower than the threshold value.

44. The method according to claim 31, wherein the step of identifying and removing common characteristics comprises decreasing the cardinality of the set of indexed data.

45. The method according to claim 44, wherein reducing cardinality comprises removing a portion of the data from a member of the indexed set.

46. The method according to claim 31, wherein the step of identifying and removing common characteristics comprises computing an ensemble variance of a set of control spectra.

47. The method according to claim 31, wherein the step of identifying and removing noise portions comprises computing an ensemble variance of a set of control spectra.

48. The method according to claim 31, wherein the step of identifying and removing the noise portions comprises decreasing the cardinality of the set of indexed data.

49. The method according to claim 48, wherein reducing cardinality comprises removing a portion of the data from a member of the indexed set.

51. A method for classifying a set of indexed data which include a set of control spectra, comprising the steps of:

a. calculating an ensemble statistic at each index in the control spectra;

b. identifying those indices at which the ensemble statistic exceeds a first selected threshold;

c. removing the identified indices from all spectra in the set of indexed data to provide a set of compressed indexed data;

d. calculating an ensemble statistic at each index of the compressed indexed data;

e. removing all indices from each compressed spectrum that have an ensemble statistic that is lower than a second selected threshold value to provide a set of reduced indexed data;

f. extracting a feature portion of each of the reduced indexed data to provide a set of feature spectra; and

g. classifying the set of feature spectra into clusters.

52. The method according to claim 51, wherein the step of calculating an ensemble statistic at each index in the control spectra comprises computing the ensemble variance of the control spectra.

53. The method according to claim 51, wherein the step of calculating an ensemble statistic at each index of the compressed indexed data comprises computing the ensemble variance of the compressed indexed data.