SYSTEM AND METHOD OF MANAGING LARGE DATA FILES

• US 20140164444A1
• Filed: 11/27/2013
• Published: 06/12/2014
• Est. Priority Date: 12/01/2012
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Bowen, Benjamin P., Ruebel, Oliver
• Granted Patent: US 10,860,526 B2
• Time in Patent Office: Days
• US Class Current: 707/821
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: H01J 49/0004 Imaging particle spectrometry

Abstract

Disclosed are systems and software that provide a high-performance, extensible file format and web API for remote data access and a visual interface for data viewing, query, and analysis. The described system supports can support storage of raw spectroscopic data such as neural recording data, MSI data, metadata, and derived analyses in a single, self-describing format that may be compatible by a large range of analysis software.

Claims

1. A system configured to present neurosensory and/or spectroscopic data to a user computer, the system comprising:

an interface to a data storage device configured to store neurosensory and/or spectroscopic data, analysis data and associated metadata;

a first storage module configured to store the neurosensory and/or the spectroscopic data, analysis data, and the associated metadata as one or more of individual spectrally aligned data chunks, image aligned data chunks, and hybrid data chunks, wherein the stored data chunks comprise sub-blocks of the spectroscopic data; and

a user interface module configured to retrieve image slices, spectral data, or arbitrary sub-cubes from the stored data chunks and graphically display the retrieved data and the associated metadata to the user computer.

2. The system of claim 1, wherein the neurosensory and/or the spectroscopic data comprises spectra generated by absorption, emission, energy, frequency, reflectance, resonance, molecular vibration, and/or secondary emission.

3. The system of claim 1, wherein the spectroscopic data comprises mass spectrometry imaging data.

4. The system of claim 1, wherein the neurosensory data comprises neural recordings data.

5. The system of claim 3, wherein the mass spectrometry imaging data comprises single spectrum or multiple spectra at a plurality of positions, the mass spectrometry imaging data further comprising spectra of intact molecules and fragments of molecules.

6. The system of claim 3, wherein the mass spectrometry imaging data comprises mass to charge ratio and ion-mobility data.

7. The system of claim 3, wherein mass spectrometry imaging data comprises 1-dimensional, 2-dimensional, or 3-dimensional spectral imaging data.

8. The system of claim 1, wherein the user interface module is configured to retrieve one or more of the spectrally aligned data chunks, image aligned data chunks, and/or hybrid data chunks independently of each other.

9. The system of claim 3, wherein the user interface module is further configured to receive and process API functions, the API functions including URL patterns for sending one or more of:

a request to retrieve the associated metadata,

a request to retrieve information about m/z data,

a request to retrieve ion image and/or ion image slices,

a request to retrieve one or more m/z spectrum, and

a request to retrieve arbitrary sub-cubes of the mass spectrometry imaging data.

10. The system of claim 3, wherein the associated metadata comprises mass spectrometry imaging instrument information, sample information, and/or imaging settings.

11. The system of claim 3, wherein the first storage module is further configured to store the data chunks in a self-describing file format comprising descriptions of hierarchy of the mass spectrometry imaging data and type of the mass spectrometry imaging data.

12. The system of claim 11, wherein the one or more file formats include the HDF5 file format.

13. The system of claim 1, wherein the user interface module comprises an array-based user interface configured to retrieve the neurosensory and/or the spectroscopic data stored in a 3-dimensional cube format and a reduced data format.

14. The system of claim 1, wherein the spectra aligned data chunks are configured to store a single full spectrum or a portion of a spectrum per data chunk.

15. The system of claim 1, wherein the image aligned data chunks are configured to store a single full ion-image or a portion of an ion-image per data chunk.

16. The system of claim 1, wherein the hybrid data chunks are configured to store at least a portion of a spectrum and a portion of an ion-image.

17. The system of claim 1, wherein the first storage module is further configured to compress the neurosensory and/or the spectroscopic data using compression filters.

18. The system of claim 1, further comprising one or more additional storage modules configured to replicate the neurosensory and/or the spectroscopic data and store two or more copies of the spectroscopic data as data chunks.

19. The system of claim 1, wherein the first storage module is configured to store the received the neurosensory and/or the spectroscopic data as a 2-dimensional dataset of spectra and one or more index datasets configured to record relationships between spatial locations and the spectra.

20. The system of claim 1, wherein the system further comprises a data analysis module configured to:

analyze the neurosensory and/or the neurosensory and/or the spectroscopic data;

track input parameters and dependencies; and

in response to receiving a request from the user computer, retracing the analysis performed on the neurosensory and/or the spectroscopic data for purposes of repeating an analysis.

21. The system of claim 20, wherein the data analysis module is further configured to process the stored data chunks, the processing comprising data reduction, dimension reduction, feature detection, and/or clustering on the stored data chunks.

22. The system of claim 20, wherein the data reduction includes one or more of maximum, minimum, average, standard deviation, and variance.

23. A non-transitory computer-readable storage medium comprising computer-executable instructions that when executed direct a computing system to:

store the neurosensory and/or the spectroscopic data, analysis data, and metadata as one or more of individual spectrally aligned data chunks, image aligned data chunks, and hybrid data chunks, and wherein the stored data chunks comprise sub-blocks of the neurosensory and/or the spectroscopic data; and

retrieve image slices, spectra data, or arbitrary sub-cubes from the stored data chunks and graphically display the retrieved data and the metadata to a user computer.

24. The non-transitory computer-readable storage medium of claim 23, wherein the neurosensory and/or the spectroscopic data comprises spectra generated by absorption, emission, energy, frequency, reflectance, resonance, molecular vibration, and/or secondary emission.

25. The non-transitory computer-readable storage medium of claim 23, wherein the spectroscopic data is mass spectral imaging data and in response to receiving instructions from a user computer, retrieving ion image slices, m/z spectra data, or arbitrary sub-cubes from the stored data chunks and graphically displaying the retrieved data and the associated metadata to the user computer.

26. The non-transitory computer-readable storage medium of claim 23, wherein the neurosensory and/or the spectroscopic data is neural recording data and in response to receiving instructions from a user computer, retrieving ion image slices, m/z spectra data, or arbitrary sub-cubes from the stored data chunks and graphically displaying the retrieved data and the associated metadata to the user computer.

27. The non-transitory computer-readable storage medium of claim 25, wherein the mass spectrometry imaging data comprises multiple spectra at a plurality of positions, the mass spectrometry imaging data further comprising spectra of intact molecules and fragments of molecules.

28. The non-transitory computer-readable storage medium of claim 25, wherein the associated metadata includes mass spectrometry imaging instrument information, sample information, and/or imaging settings.

29. The non-transitory computer-readable storage medium of claim 25, wherein the data chunks are further configured to be stored in a self-describing file format, the self-describing file format comprises descriptions of hierarchy of the neurosensory and/or the spectroscopic data and type of the neurosensory and/or the spectroscopic data.

30. A computer-implemented method in a computer having a processor, comprising:

obtaining neurosensory and/or spectroscopic data and associated metadata from an imaging device;

storing the neurosensory and/or the spectroscopic data and associated metadata as one or more of individual spectrally aligned data chunks, image aligned data chunks, and hybrid data chunks to a storage, wherein the stored data chunks comprise sub-blocks of the neurosensory and/or the spectroscopic data;

retrieving ion image slices, spectra data, or arbitrary sub-cubes from the stored data chunks in response to a user request; and

graphically displaying the retrieved the neurosensory and/or the spectroscopic data and metadata to a user'"'"'s computer.

31. The computer-implemented method of claim 30, wherein obtaining the neurosensory and/or the spectroscopic data comprises obtaining neurosensory and/or spectroscopic data generated by absorption, emission, energy, frequency, reflectance, resonance, molecular vibration, and/or secondary emission.

32. The computer-implemented method of claim 30, wherein obtaining the spectroscopic data comprises obtaining mass spectrometry imaging data from a mass spectrometer.

33. The computer-implemented method of claim 32, wherein obtaining the mass spectrometry imaging data comprises obtaining multiple spectra at a plurality of positions, the mass spectrometry imaging data further comprising spectra of intact molecules and fragments of molecules.

34. The computer-implemented method of claim 30, wherein the neurosensory data comprises neural recordings data.

35. The computer-implemented method of claim 30, wherein storing the neurosensory and/or the spectroscopic data comprises storing the neurosensory and/or the spectroscopic data in data chunks in a self-describing file format, the self-describing file format comprising descriptions of hierarchy of the neurosensory and/or the spectroscopic data and type of the neurosensory and/or the spectroscopic data.

36. The computer-implemented method of claim 30, wherein retrieving spectral data further comprises retrieving one or more of the spectrally aligned data chunks, image aligned data chunks, and/or hybrid data chunks independently.