Distributed data analytics

• US 10,706,970 B1
• Filed: 09/28/2017
• Issued: 07/07/2020
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Ukelson, Michal Ziv, Dach, Ran, Benshahar, Arnon
• Primary Examiner(s): Negin, Russell S
• Application Number:
  US15/719,231
• Time in Patent Office: 1,013 Days
• Field of Search: None
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06Q 50/22 Social work or social welfare, e.g. community support activities or counselling services
• CPC Class Codes: G16B 20/00 ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations
• CPC Class Codes: G16B 30/00 ICT specially adapted for sequence analysis involving nucleotides or amino acids
• CPC Class Codes: G16B 50/30 Data warehousing; Computing architectures
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: Y02A 90/10 Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

An apparatus in one embodiment comprises a distributed data processing system in which multiple processing devices communicate with one another over at least one network. The distributed data processing system is configured to obtain reads of biological samples of respective microbiomes, with each of the biological samples containing genomic material from a plurality of distinct microorganisms of its corresponding one of the microbiomes, and to perform distributed data analytics to detect a disease, infection or contamination that involves genomic material from multiple ones of the distinct microorganisms in one or more of the microbiomes. Performing distributed data analytics illustratively comprises performing local analytics in respective ones of a plurality of data zones, and performing global analytics utilizing results of the local analytics performed in the respective data zones. Each of the data zones may comprise, for example, one or more sequencing centers utilized to generate a corresponding subset of the reads within that data zone.

Claims

1. A method comprising:

obtaining reads of biological samples of respective microbiomes wherein each of the biological samples contains genomic material from a plurality of distinct microorganisms of its corresponding one of the microbiomes; and

performing distributed data analytics to detect a disease, infection or contamination that involves genomic material from multiple ones of the distinct microorganisms in one or more of the microbiomes;

wherein performing distributed data analytics comprises;

performing local analytics in respective ones of a plurality of data zones; and

performing global analytics utilizing results of the local analytics performed in the respective data zones;

wherein each of the data zones comprises one or more sequencing centers utilized to generate a corresponding subset of the reads within that data zone;

wherein the local analytics performed in a given one of the data zones utilize reads of one or more of the biological samples sequenced in the one or more sequencing centers of the given data zone;

wherein the local analytics performed in the given data zone comprise analyzing the reads of the one or more biological samples against a local set of known virulence factors; and

wherein the method is implemented by a distributed data processing system comprising a plurality of processing devices configured to communicate with one another over at least one network.

2. The method of claim 1 wherein the reads of the biological samples comprise respective sets of gene units sequenced from those biological samples in corresponding ones of a plurality of sequencing centers.

3. The method of claim 1 wherein a result of the global analytics characterizes the disease, infection or contamination as involving genomic material identified in different ones of the reads of the biological samples generated in different ones of the data zones by corresponding different ones of the sequencing centers.

4. The method of claim 3 wherein the result of the global analytics is obtained without reassembly of the genomic material identified in the different reads generated by the different sequencing centers into genomes of respective ones of the distinct microorganisms.

5. The method of claim 1 wherein the local analytics performed in a given one of the data zones are performed at least in part in parallel with the local analytics performed in one or more other ones of the data zones.

6. The method of claim 1 wherein a result of the global analytics is generated in a particular one of the plurality of data zones that performs local analytics.

7. The method of claim 1 wherein the local analytics performed in the given data zone comprise generating a virulence profile for the reads of the one or more biological samples and wherein the virulence profile provides at least a portion of the results of the local analytics of the given data zone utilized in performing the global analytics.

8. The method of claim 1 wherein a result of the global analytics is generated at least in part by performing one or more biclustering operations on results of the local analytics performed in the respective data zones.

9. The method of claim 8 wherein a given one of the biclustering operations is performed on a hit abundance score matrix that relates reads of different ones of a plurality of biological samples sequenced in different ones of the sequencing centers to respective different ones of a plurality of target genomic sequences.

10. The method of claim 1 wherein the data zones comprise respective geographically-distributed regional data centers each configured to perform local analytics utilizing locally accessible data resources of its corresponding data zone and wherein the results of the local analytics performed in the respective data zones preserve for each of the data zones at least one specified policy of that data zone relating to at least one of privacy, security, governance, risk and compliance.

11. The method of claim 1 wherein the distributed data processing system comprises a plurality of WWH nodes configured to control performance of at least portions of the distributed data analytics to detect the disease, infection or contamination.

12. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by a distributed data processing system comprising a plurality of processing devices configured to communicate with one another over at least one network causes said distributed data processing system:

to obtain reads of biological samples of respective microbiomes wherein each of the biological samples contains genomic material from a plurality of distinct microorganisms of its corresponding one of the microbiomes; and

to perform distributed data analytics to detect a disease, infection or contamination that involves genomic material from multiple ones of the distinct microorganisms in one or more of the microbiomes;

wherein performing distributed data analytics comprises;

performing local analytics in respective ones of a plurality of data zones; and

performing global analytics utilizing results of the local analytics performed in the respective data zones;

wherein each of the data zones comprises one or more sequencing centers utilized to generate a corresponding subset of the reads within that data zone;

wherein the local analytics performed in a given one of the data zones utilize reads of one or more of the biological samples sequenced in the one or more sequencing centers of the given data zone; and

wherein the local analytics performed in the given data zone comprise analyzing the reads of the one or more biological samples against a local set of known virulence factors.

13. The computer program product of claim 12 wherein a result of the global analytics is generated at least in part by performing one or more biclustering operations on results of the local analytics performed in the respective data zones.

14. An apparatus comprising:

a distributed data processing system comprising a plurality of processing devices configured to communicate with one another over at least one network;

wherein said distributed data processing system is configured;

to obtain reads of biological samples of respective microbiomes wherein each of the biological samples contains genomic material from a plurality of distinct microorganisms of its corresponding one of the microbiomes; and

to perform distributed data analytics to detect a disease, infection or contamination that involves genomic material from multiple ones of the distinct microorganisms in one or more of the microbiomes;

wherein performing distributed data analytics comprises;

performing local analytics in respective ones of a plurality of data zones; and

performing global analytics utilizing results of the local analytics performed in the respective data zones;

wherein each of the data zones comprises one or more sequencing centers utilized to generate a corresponding subset of the reads within that data zone;

wherein the local analytics performed in a given one of the data zones utilize reads of one or more of the biological samples sequenced in the one or more sequencing centers of the given data zone;

wherein the local analytics performed in the given data zone comprise analyzing the reads of the one or more biological samples against a local set of known virulence factors.

15. The apparatus of claim 14 wherein a result of the global analytics is generated at least in part by performing one or more biclustering operations on results of the local analytics performed in the respective data zones.

16. The apparatus of claim 14 wherein the local analytics performed in the given data zone comprise generating a virulence profile for the reads of the one or more biological samples and wherein the virulence profile provides at least a portion of the results of the local analytics of the given data zone utilized in performing the global analytics.

17. The apparatus of claim 14 wherein a result of the global analytics is generated at least in part by performing one or more biclustering operations on results of the local analytics performed in the respective data zones.

18. The apparatus of claim 17 wherein a given one of the biclustering operations is performed on a hit abundance score matrix that relates reads of different ones of a plurality of biological samples sequenced in different ones of the sequencing centers to respective different ones of a plurality of target genomic sequences.

19. The apparatus of claim 14 wherein the data zones comprise respective geographically-distributed regional data centers each configured to perform local analytics utilizing locally accessible data resources of its corresponding data zone and wherein the results of the local analytics performed in the respective data zones preserve for each of the data zones at least one specified policy of that data zone relating to at least one of privacy, security, governance, risk and compliance.

20. The apparatus of claim 14 wherein the distributed data processing system comprises a plurality of WWH nodes configured to control performance of at least portions of the distributed data analytics to detect the disease, infection or contamination.