Systems and methods for efficient data searching, storage and reduction
First Claim
1. A system for searching in a repository data for data that are similar to an input data, the repository data being divided into one or more repository chunks, the system comprising:
- means for, for each repository chunk, calculating a corresponding set of repository distinguishing characteristics (RDCs), each set of RDCs comprising a plurality of distinguishing characteristics, said means arranged to partition the respective data chunks into a plurality of seeds, each seed being a smaller part of the respective data chunk and ordered in a seed sequence and to apply a hash function to each of the seeds to generate a plurality of hash values wherein each seed yields one hash value;
means for maintaining an index associating each set of RDCs and the corresponding repository chunk;
means for comparing input distinguishing characteristics of an input chunk of input data to one or more sets of RDCs stored in the index to determine whether a similarity exists between the input chunk and the distinguishing repository chunk, characterized in that;
said comparing means is configured to determine a similarity exists if a similarity threshold (j) of a set of input distinguishing characteristics is found in a set of RDCs stored in the index; and
in that said calculating means is configured to select a subset (k) of the plurality of hash values;
to determine positions of the seeds within the seed sequence corresponding to the selected subset of hash values;
to apply a function to the determined positions to determine corresponding other positions within the seed sequence; and
to define the set of distinguishing characteristics as the hash values of the seeds at the determined other positions.
0 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods enabling search of a repository for the location of data that is similar to input data, using a defined measure of similarity, in a time that is independent of the size of the repository and linear in a size of the input data, and a space that is proportional to a small fraction of the size of the repository. The similar data segments thus located are further analyzed to determine their common (identical) data sections, regardless of the order and position of the common data sections in the repository and input, and in a time that is linear in the segment size and in constant space.
-
Citations
20 Claims
-
1. A system for searching in a repository data for data that are similar to an input data, the repository data being divided into one or more repository chunks, the system comprising:
-
means for, for each repository chunk, calculating a corresponding set of repository distinguishing characteristics (RDCs), each set of RDCs comprising a plurality of distinguishing characteristics, said means arranged to partition the respective data chunks into a plurality of seeds, each seed being a smaller part of the respective data chunk and ordered in a seed sequence and to apply a hash function to each of the seeds to generate a plurality of hash values wherein each seed yields one hash value; means for maintaining an index associating each set of RDCs and the corresponding repository chunk; means for comparing input distinguishing characteristics of an input chunk of input data to one or more sets of RDCs stored in the index to determine whether a similarity exists between the input chunk and the distinguishing repository chunk, characterized in that; said comparing means is configured to determine a similarity exists if a similarity threshold (j) of a set of input distinguishing characteristics is found in a set of RDCs stored in the index; and in that said calculating means is configured to select a subset (k) of the plurality of hash values; to determine positions of the seeds within the seed sequence corresponding to the selected subset of hash values; to apply a function to the determined positions to determine corresponding other positions within the seed sequence; and to define the set of distinguishing characteristics as the hash values of the seeds at the determined other positions. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A method of searching in repository data for data that are similar to an input data, wherein the repository data is divided into one or more repository chunks, the method comprising:
-
for each repository chunk, calculating a corresponding set of repository distinguishing characteristics (RDCs), each set of RDCs comprising a plurality (n) of distinguishing characteristics; maintaining an index associating each set of RDCs and the corresponding repository chunk; calculating input distinguishing characteristics (IDCs) for an input chunk of data; comparing the IDCs to one or more sets of RDCs stored in the index to determine if a similarity exists between the input chunk and the corresponding repository chunk, wherein the RDCs and the IDCs are obtained by;
partitioning the respective data chunk into a plurality of seed(s), each seed being a smaller part of the respective data chunk and ordered in a seed sequence;applying a hash function to each of the seeds to generate a plurality of hash values, wherein each seed yields one hash value; characterized in that a set of IDCs is calculated for each input data chunk, the set comprising a plurality (k) of distinguishing characteristics, said set being compared with the sets of RDCs; in that it is determined that similarity exists if a similarity threshold (j) of the distinguishing characteristics in the set of IDCs is found in a set of RDCs stored in the index; and in that each set of RDCs and IDCs is obtained by the further steps of selecting a subset (k) of the plurality of hash values;
determining positions of the seeds within the seed sequence corresponding to the selected subset of hash values;applying a function to the determined positions to determine corresponding other positions within the seed sequence; and defining the set of distinguishing characteristics as the hash values of the seeds at the determined other positions. - View Dependent Claims (7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A method of searching in repository data for data that is similar to an input data wherein the repository data is divided into one or more repository chunks, the method comprising:
-
for each repository chunk, calculating a corresponding set of repository distinguishing characteristics, each set of RDCs comprising a plurality of distinguishing characteristics and being obtained by partitioning the respective data chunk into a plurality of seeds(s), each seed being a smaller part of the respective data chunk and ordered in a seed sequence, and applying a hash function to each of the seeds to generate a plurality of hash values, wherein each seed yields one hash value; maintaining an index associating each set of ROCs and the corresponding repository chunk; comparing input distinguishing characteristics of an input chunk of input data to one or more sets of RDCs stored in the index to determine whether a similarity exists between the input chunk and the corresponding repository chunk, characterized in that; it is determined that a similarity exists between the input chunk and the corresponding repository chunk if a similarity threshold (j) of the distinguishing characteristics in the set of IDCs is found in a set of ROCs stored in the index; and in that the set of RDCs is obtained by;
selecting a subset (k) of the plurality of hash values;determining positions of the seeds within the seed sequence corresponding to the selected subset of hash values; applying a function to the determined positions to determine corresponding other positions within the seed sequence; and
defining the set of distinguishing characteristics as the hash values of the seeds at the determined other positions. - View Dependent Claims (20)
-
Specification