Method and system for collecting and organizing data corresponding to an event

• US 10,146,810 B2
• Filed: 08/02/2013
• Issued: 12/04/2018
• Est. Priority Date: 02/01/2008
• Status: Active Grant

• Current Assignee: Musarubra US LLC (Musarubra US SellCo LLC)
• Original Assignee: FireEye, Inc. (Alphabet Inc.)
• Inventors: Shiffer, Jason, Frazier, Matthew, Cunningham, Sean, Hogsten, Scott, Helvey, Eric, Butler, James, Villadsen, Peter
• Primary Examiner(s): Richardson, James E
• Application Number:
  US13/957,702
• Publication Number:
  US 20130325872A1
• Time in Patent Office: 1,950 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence

Abstract

A system and method for analyzing data from a plurality of computer environments. The computer environments are authenticated and data is imported to a memory location. The data is converted into a uniform format to enable expedited searching by one or more authenticated users. The data may be marked so that a user may determine which computer environment provided the data. The system may also create one or more indexes of the data to assist one or more users in searching the data.

Claims

1. A method of collecting data comprising:

searching, of a plurality of computer environments communicatively coupled via a network, for data that matches a predetermined criterion;

copying matching data to a memory location;

creating two or more temporary indexes from the matching data;

adding the two or more temporary indexes to a temporary index queue; and

updating the two or more temporary indexes at a predetermined interval, the updating the two or more temporary indexes includes

selecting a first temporary index of the two or more temporary indexes in the temporary index queue,

merging the selected temporary index into a permanent index while the permanent index, after receiving at least some of the content of the selected temporary index, has failed to exceed a storage threshold,

discontinuing merging of the selected temporary index into the permanent index when the permanent index exceeds the storage threshold, and

converting a second temporary index of the two or more temporary indexes into a second permanent index by selecting the next temporary index in the temporary index queue as the second temporary index and changing the definition of the second temporary index from temporary to permanent without processing the contents of the second temporary index.

2. The method of claim 1 wherein the creating of the two or more temporary indexes occurs based on recently discovered data.

3. The method of claim 1 wherein the updating the two or more temporary indexes further comprises deleting one or more indexes based on recently deleted data.

4. The method of claim 1 wherein the creating of the two or more temporary indexes and the updating the two or more temporary indexes are conducted in accordance with an index management scheme, the index management scheme includes an analysis of search cost that control merging of the selected temporary index into the permanent index along with an analysis of index strategy in controlling a structure and process of the selected temporary index.

5. The method of claim 1 wherein the updating of the two or more temporary indexes comprises deleting one or more indexes based on one or more searches requested by a user.

6. The method of claim 5 wherein the deleting of the one or more indexes based on one or more searches requested by the user comprises deleting one or more indexes if the indexes have not been utilized by the system within a predetermined period of time.

7. The method of claim 1 wherein the updating of the two or more temporary indexes at the predetermined interval further comprises creating the selected temporary index for a new amount of data when the new amount of data has been detected during the searching of the plurality of computer environments.

8. The method of claim 7 wherein the merging of the selected temporary index into the permanent index occurs once the selected temporary index has reached a predetermined size.

9. The method of claim 1 being performed by a controller deployed in a computer system that is coupled to the plurality of computer environments over the network.

10. The method of claim 1 wherein the predetermined interval is a time interval.

11. The method of claim 1, further comprising

converting the second temporary index to the second permanent index when the first permanent index is full.

12. A method of searching for data in a computer environment comprising a number of storage devices, the method comprising:

presenting a list of storage devices to a user for selection;

selecting, by a user, one or more storage devices from the list;

searching for stored data on the selected one or more storage devices;

creating a plurality of indexes of the stored data, wherein the creating of the plurality of indexes includes initially defining each of the plurality of indexes as a temporary index that is convertible to a permanent index, and adding each of the plurality of indexes to a temporary index queue;

converting a first temporary index of the plurality of indexes into a first permanent index by at least

merging the first temporary index in the temporary index queue into a first permanent index while content of the first temporary index has failed to exceed a storage threshold of the first permanent index, and discontinuing the merging of the first temporary index into the first permanent index when the first permanent index exceeds the storage threshold, and

selecting a second temporary index in the temporary index queue and changing the definition of the second temporary index from temporary to permanent without processing the contents of the second temporary index; and

executing a search of the stored data based on the plurality of indexes.

13. The method of claim 12 further comprising converting the second temporary index in the temporary index queue into the second permanent index when the first permanent index exceeds the storage threshold.

14. The method of claim 12 further comprising copying the stored data to a single memory location.

15. The method of claim 12 wherein the plurality of indexes are tables which include a number of variables and files in the stored data that correspond to the variables.

16. The method of claim 15 wherein the variables comprise file creation dates or file types.

17. The method of claim 15 wherein the tables further comprise the location of the stored data on the selected one or more storage devices.

18. The method of claim 12 wherein the executing of the search is conducted by a search engine that is configured with to search across all temporary indexes and all permanent indexes.

19. The method of claim 12 wherein the plurality of indexes are saved to a storage device of the one or more storage devices.

20. The method of claim 12 wherein the plurality of indexes are periodically updated upon the discovery of a predetermined amount of unindexed data.

21. A system for collecting data comprising:

means for searching a plurality of computer environments communicatively coupled via a network for data that matches a predetermined criterion;

means for copying the matching data to a memory location;

means for creating two or more temporary indexes from the matching data;

means for adding the two or more temporary indexes to a temporary index queue; and

means for updating the two or more temporary indexes at a predetermined interval, the updating the two or more temporary includes

selecting a first temporary index of the two or more temporary indexes in the temporary index queue,

merging the selected temporary index into a permanent index while the permanent index, after receiving at least some of the content of the selected temporary index, has failed to exceed a storage threshold,

discontinuing merging of the selected temporary index into the permanent index when the permanent index exceeds the storage threshold, and

converting a second temporary index of the two or more temporary indexes into a second permanent index by selecting the next temporary index in the temporary index queue as the second temporary index and changing the definition of the second temporary index from temporary to permanent without processing the contents of the second temporary index.