System for determining storage or output of data objects

• US 10,133,759 B1
• Filed: 09/15/2015
• Issued: 11/20/2018
• Est. Priority Date: 09/15/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Abuomar, Mohammad Haroon Mohammad
• Primary Examiner(s): Vu, Bai D
• Application Number:
  US14/855,257
• Time in Patent Office: 1,162 Days
• Field of Search: None
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

Described are techniques for storing data objects heterogeneously, among multiple data stores, based on the values associated with one or more data object characteristics. A central device, entity, or network may receive data objects and determine a data store in which to store each data object. One or more rules that correspond to a received data object may be determined, a rule including expressions that associate characteristics of data objects to threshold values. The rules may specify particular data stores in which to store data objects based on the outcome of the expressions. The central device may generate a communication configured to access the determined data store(s) based on data store characteristics specific to the data store(s) and provide the data objects to the determined data stores. Data objects may be moved to other data stores responsive to a modification to a rule or to characteristics of a data object.

Claims

1. A system comprising:

one or more memories storing computer-executable instructions; and

one or more hardware processors configured to execute the computer-executable instructions to;

receive, from a device associated with a first service;

an identifier indicative of the first service;

a data object having characteristics including one or more of size, access frequency, rank relative to other data objects, revenue associated with the data object, or a count of views of a webpage associated with the data object; and

an indication of a category of the data object, the category associated with a characteristic of the data object;

determine one or more rules corresponding to the identifier and the category, the one or more rules associating the characteristic with one or more data stores, wherein the one or more rules include;

an expression associating a threshold value with the characteristic;

an indication of a first data store associated with a first outcome of the expression; and

an indication of a second data store associated with a second outcome of the expression;

determine a first storage outcome associated with the expression based at least in part on a value of the characteristic and the threshold value, the first storage outcome including the first outcome of the expression;

determine storage data associated with the first data store, the storage data indicating one or more of a protocol or a programming language associated with the first data store;

based on the storage data generate a communication using the one or more of the protocol or the programming language, the communication including the data object;

provide the communication and the data object to the first data store;

determine a modification to one or more of the characteristic or the expression;

in response to the modification, determine a second storage outcome based on the characteristic and the expression, the second storage outcome including the second outcome of the expression; and

provide the data object to the second data store.

2. The system of claim 1, further comprising computer-executable instructions to:

receive, from the device associated with the first service, an indication of at least one data store characteristic, the at least one data store characteristic including one or more of;

an access speed of the data store, a financial cost of the data store, or a computational cost of the data store;

wherein the first outcome is further based on the at least one data store characteristic.

3. The system of claim 1, wherein the comprising computer-executable instructions to determine the one or more rules corresponding to the identifier and the category include computer-executable instructions to:

determine an absence of a rule corresponding to the identifier and to the category that was received from the device associated with the first service; and

determine a general rule that was received from a device associated with a second service, wherein the general rule is associated with on one or more of the category or the characteristic of the data object, the general rule associating the characteristic with the one or more data stores.

4. The system of claim 1, further comprising computer-executable instructions to:

determine that the data object includes a first portion associated with confidential information and a second portion associated with non-confidential information, wherein the characteristic includes a presence of the confidential information and the first storage outcome is determined for the first portion of the data object;

determine a third storage outcome based on the second portion; and

based on the third storage outcome, provide the second portion in the second data store.

5. A method comprising:

receiving a data object that includes a first portion having a first characteristic and a second portion having a second characteristic;

determining at least one rule associating characteristics of data objects with a plurality of data stores, the at least one rule including one or more expressions associating a first value of the first characteristic with a threshold value and a second value of the second characteristic with the threshold value, wherein one or more outcomes of the one or more expressions may be determined based on the first value and the second value;

determining a first outcome of the one or more expressions based on the first value of the first characteristic;

determining a first data store corresponding to the first outcome, wherein the first data store includes at least one first data store characteristic associated with the first characteristic of the first portion of the data object;

determining one or more of a first protocol or a first programming language associated with the first data store;

generating a first communication using the one or more of the first protocol or the first programming language, the first communication including the first portion of the data object; and

providing the first communication and the first portion of the data object to the first data store;

determining a second outcome of the one or more expressions based on the second value of the second characteristic;

determining a second data store corresponding to the second outcome, wherein the second data store includes at least one second data store characteristic associated with the second characteristic of the second portion of the data object;

determining one or more of a second protocol or a second programming language associated with the second data store;

generating a second communication using the one or more of the second protocol or the second programming language, the second communication including the second portion of the data object; and

providing the second communication and the second portion of the data object to the second data store.

6. The method of claim 5, wherein the first characteristic of the first portion includes a presence of confidential information, the second characteristic of the second portion includes an absence of the confidential information, and the at least one first data store characteristic includes one or more security features to control access to the confidential information.

7. The method of claim 5, further comprising receiving the at least one rule from a device associated with the data object, the at least one rule further including an indication of one or more particular data characteristics, the particular data characteristics including one or more of rank of the data object relative to other data objects, revenue associated with the data object, or a count of views of a webpage associated with the data object.

8. The method of claim 5, wherein determining the at least one rule includes:

determining correspondence between one or more of the first characteristic of the data object or a category associated with the data object and the at least one rule, wherein the at least one rule includes a preexisting rule.

9. The method of claim 5, further comprising:

receiving at least one third data store characteristic indicative of one or more of an access speed of the first data store, a financial cost of the first data store, or a computational cost of the first data store;

wherein determining the first data store corresponding to the first outcome of the at least one rule comprises;

determining a plurality of data stores corresponding to the first outcome of the at least one rule; and

determining correspondence between the at least one third data store characteristic and the first data store.

10. The method of claim 5, further comprising:

determining a modification to the at least one rule to form a modified rule;

access the first portion of the data object previously stored in the first data store based on the first outcome of the at least one rule;

determining a third outcome using the modified rule, the third outcome associated with the first portion of the data object;

determining one or more of the second data store or a third data store corresponding to the third outcome of the modified rule;

generating a third communication including the first portion of the data object; and

providing the third communication and the first portion of the data object to the one or more of the second data store or the third data store.

11. The method of claim 5, further comprising:

determining one or more of a third protocol or a third programming language associated with a third data store for archiving data objects;

generating a third communication using the one or more of the third protocol or the third programming language associated with the third data store, the third communication including the data object; and

providing the third communication and the data object to the third data store for archival.

12. The method of claim 5, wherein the one or more expressions of the at least one rule associate an access frequency of the first portion of the data object with the first threshold value, the method further comprising:

determining a change in the access frequency of the first portion of the data object to form a modified access frequency;

determining a third outcome of the at least one rule based on the modified access frequency;

determining one or more of the second data store or a third data store corresponding to the third outcome;

generating a third communication including the first portion of the data object; and

providing the third communication and the first portion of the data object to the one or more of the second data store or the third data store.

13. A system comprising:

one or more memories storing computer-executable instructions; and

one or more hardware processors configured to execute the computer-executable instructions to;

receive a data object, an identifier associated with a first entity, and an indication of a category associated with the data object;

determine an absence of a rule received from the first entity that corresponds to the identifier and the category;

in response to the absence of the rule, determine one or more rules received from a second entity, wherein the one or more rules correspond to the identifier and the category, the one or more rules associating values of characteristics of the data object with one or more outputs;

determine a value of a characteristic of the data object;

determine, based on at least a subset of the one or more rules, at least one output corresponding to the value of the characteristic;

determine a configuration corresponding to the at least one output;

generate a communication based at least in part on the configuration, the communication including at least a portion of the data object; and

provide the communication and the at least a portion of the data object to a destination corresponding to the at least one output.

14. The system of claim 13, wherein the computer-executable instructions to provide the communication and the at least a portion of the data object to the destination include computer-executable instructions to:

stream the at least a portion of the data object to an output device.

15. The system of claim 13, wherein the computer-executable instructions to provide the communication and the at least a portion of the data object to the destination comprise computer-executable instructions to provide the communication and the at least a portion of the data object to at least one data store, the system further comprising computer-executable instructions to:

receive an indication of a data store characteristic; and

determine the at least one data store corresponding to the data store characteristic and to the value of the characteristic of the data object.

16. The system of claim 13, wherein the computer-executable instructions to determine the value of the characteristic of the data object include computer-executable instructions to:

determine a first value corresponding to a first portion of the data object; and

determine a second value corresponding to a second portion of the data object;

wherein the at least one output includes a first output corresponding to the first value and a second output corresponding to the second value, the first output includes providing the first portion of the data object to a first data store having a first data store characteristic that corresponds to the first portion, and the second output includes providing the second portion of the data object to a second data store having a second data store characteristic that corresponds to the second portion.

17. The system of claim 16, further comprising computer-executable instructions to:

provide metadata associated with the first portion of the data object and the second portion of the data object, the metadata indicating an association between the first portion of the data object and the second portion of the data object;

receive a request to access the data object;

determine the metadata associated with one or more of the first portion or the second portion;

based at least in part on the metadata, access the first portion and the second portion;

generate the data object using the first portion and the second portion; and

provide the data object responsive to the request.

18. The system of claim 13, wherein

the at least a subset of the one or more rules includes:

an expression associating the value of the characteristic with a threshold value;

an indication of a first output associated with a first outcome of the expression; and

an indication of a second output associated with a second outcome of the expression.

19. The system of claim 13, further comprising computer-executable instructions to:

determine a modification to one or more of a rule or the value of the characteristic;

determine, based on the modification, a second output corresponding to the value of the characteristic;

determine a second configuration corresponding to the second output;

generate a second communication based at least in part on the second configuration, the second communication including the at least a portion of the data object; and

provide the second communication and the at least a portion of the data object to a second destination corresponding to the second output.

20. The system of claim 16, wherein the first value includes a presence of confidential information, the second value includes an absence of the confidential information, and the first data store characteristic includes at least one security feature for controlling access to the confidential information.