Method and apparatus for a searchable data service

• US 7,801,912 B2
• Filed: 03/29/2006
• Issued: 09/21/2010
• Est. Priority Date: 12/29/2005
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Saha, Partha, Collette, James R., Ransil, Patrick W., Chu, Robert Wai-Chi, Martynov, Aleksey V., Larson, James S.
• Primary Examiner(s): Ali; Mohammad
• Assistant Examiner(s): Smith; Brannon W
• Application Number:
  US11/392,482
• Publication Number:
  US 20070168336A1
• Time in Patent Office: 1,637 Days
• Field of Search: 707/3, 707/770, 707/827
• US Class Current: 707/770
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Method and apparatus for a searchable data service are described. The searchable data service may be implemented as a Web service with a Web service interface that allows the storage of locators and other attributes associated with entities stored in a data store in a searchable index. The attributes may be expressed as {name, value} pairs. The interface may allow client applications to query the searchable index to retrieve locators for entities in the backend data store according to the attributes associated with each locator. Search speed may be automatically optimized using, for example, indexes, query planning, and parallelism. The searchable data service provides a searchable index and is not a data store per se. The searchable data service separates searching and indexing of data from the actual storage of the data.

Claims

1. A system, comprising:

one or more computer devices configured to implement a web services platform configured to provide a web service interface to a searchable data service, wherein the web services platform is configured to receive service requests from a plurality of client applications in accordance with the web service interface, wherein the service requests comprise query requests and storage requests, and wherein the web service interface provides a common message endpoint to the plurality of client applications to send the query requests and storage requests; and

a plurality of computer devices configured to implement a plurality of nodes configured to participate in the searchable data service to;

store searchable data service objects specified in received storage requests in respective searchable indexes for a plurality of independent data stores used by the client applications, wherein the searchable indexes are on the plurality of nodes, wherein the data stores are on one or more storage devices each on a network and separate from the one or more computer devices that implement the plurality of nodes configured to participate in the searchable data service, wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores such that each searchable index provides a complete index for only one of the independent data stores, wherein each searchable data service object specifies two or more attributes of a particular entity in a particular data store, wherein the attributes include a unique entity identifier for locating the particular entity in the particular data store;

locate sets of one or more searchable data service objects from the searchable indexes that satisfy received query requests, wherein the received query requests specify one of the searchable indexes; and

return at least the entity identifiers from the sets of one or more searchable data service objects that satisfy the query requests to the client applications in accordance with the web service interface.

2. The system as recited in claim 1, wherein each attribute is expressed as a {name, value} pair.

3. The system as recited in claim 1, wherein the plurality of nodes comprises one or more storage nodes configured to persistently store the searchable indexes.

4. The system as recited in claim 1, wherein the plurality of nodes comprises one or more query nodes each configured to maintain a local query cache of responses to previous query requests.

5. The system as recited in claim 1, wherein the plurality of nodes comprises:

one or more storage nodes configured to persistently store the searchable indexes;

one or more query nodes each configured to maintain a local query cache of responses to previous query requests; and

one or more coordinator nodes each configured to;

receive service requests from the web services platform;

forward each storage request to one or more of the storage nodes; and

forward each query request to one or more of the query nodes.

6. The system as recited in claim 5, wherein each of the one or more storage nodes is configured to:

receive a storage request from a coordinator node; and

add a searchable data service object specified in the storage request to a respective searchable index.

7. The system as recited in claim 5, wherein each of the one or more storage nodes is configured to:

receive a storage request from a coordinator node; and

modify a searchable data service object stored in a particular searchable index as specified in the storage request.

8. The system as recited in claim 5, wherein each of the one or more storage nodes is configured to:

receive a storage request from a coordinator node; and

delete at least a portion of a searchable data service object from a particular searchable index as specified in the storage request.

9. The system as recited in claim 5, wherein each of the query nodes is configured to:

receive a query request from a coordinator node;

access the local query cache on the query node to determine if the query request can be satisfied from the local query cache;

if the query request can be satisfied from the local query cache, return at least the entity identifiers from a set of one or more searchable data service objects from the local query cache that satisfy the query request to a client application that initiated the query request in accordance with the web service interface;

if the query request cannot be satisfied from the local query cache, forward the query request to one or more of the storage nodes.

10. The system as recited in claim 9, wherein each of the one or more storage nodes is configured to:

receive a query request from a query node;

search a partition of a particular searchable index persistently stored by the storage node to locate a set of one or more searchable data service objects in the searchable index that satisfy the query request; and

return at least the entity identifiers from the set of one or more searchable data service objects that satisfy the query request to a client application that initiated the query request in accordance with the web service interface.

11. The system as recited in claim 5, wherein each of the query nodes is configured to:

receive a query request from a coordinator node;

forward the query request to two or more of the storage nodes that each persistently store a different partition of a particular searchable index, wherein the different partitions are within the scope of the query request;

receive a different set of searchable data service objects from the particular searchable index that satisfy the query request from each of the two or more storage nodes to which the query request was forwarded;

merge the different sets of searchable data service objects; and

return at least the entity identifiers from the merged set of searchable data service objects to a client application that initiated the query request in accordance with the web service interface.

12. The system as recited in claim 5, wherein each of the query nodes is configured to:

receive a query request from a coordinator node;

forward the query request to one or more of the storage nodes;

receive searchable data service objects from a particular searchable index that satisfy the query request from at least one of the one or more storage nodes to which the query request was forwarded;

sort the received searchable data service objects in accordance with a sort specification in the query request; and

return at least the entity identifiers from the sorted searchable data service objects to a client application that initiated the query request in accordance with the web service interface and in sorted order.

13. The system as recited in claim 1, wherein the plurality of nodes comprises two or more storage nodes each configured to:

persistently store a partition of a particular searchable index;

repartition the partition to generate two or more new partitions on the storage node, wherein each of the two or more new partitions includes a different subset of the searchable data service objects stored in the searchable index;

communicate with others of the storage nodes to locate another storage node that volunteers to receive one of the two or more new partitions; and

cooperate with the other storage node to move one of the two or more new partitions to the other storage node.

14. The system as recited in claim 1, wherein the plurality of nodes comprises two or more storage nodes each configured to:

persistently store a partition of a particular searchable index;

communicate with one or more others of the storage nodes to locate another storage node that volunteers to receive a replica of the partition; and

cooperate with the other storage node to replicate the partition to the other storage node.

15. The system as recited in claim 1, wherein the plurality of nodes comprises two or more storage nodes each configured to:

persistently store a partition of a particular searchable index;

replicate the partition to another storage node in accordance with an anti-entropy protocol; and

communicate subsequent storage requests for the replicated partition to the other storage node in accordance with a gossip protocol to update the replica of the partition on the other storage node.

16. The system as recited in claim 1, wherein the web service interface exposes a plurality of service requests for performing operations of the searchable data service to the client applications of the searchable data service, wherein the service requests include:

an add request to add a specified searchable data service object to a specified searchable index;

a replace request to replace a specified portion or all of a specified searchable data service object in a specified searchable index;

a delete request to delete at least a portion of a specified searchable data service object from a specified searchable index;

a list attributes request to return a list of attributes for one or more specified searchable data service objects in a specified searchable index; and

a query request to return a set of entity identifiers of searchable data service objects in a specified searchable index that match a query expression specified in the query request.

17. The system as recited in claim 1, wherein the web service interface exposes a plurality of service requests for performing operations of the searchable data service to developers of client applications of the searchable data service, wherein the exposed plurality of service requests is configured for use in implementing search frontends to the data stores for the client applications.

18. The system as recited in claim 1, wherein each query request includes:

a bucket identifier that identifies a particular searchable index in the searchable data service, wherein a bucket defines the scope of a query request to the searchable data service; and

a query expression that specifies one or more search criteria to be applied to specified attributes of the searchable index identified by the bucket identifier to locate searchable data service objects that satisfy the query expression.

19. The system as recited in claim 18, wherein each query request further includes a subscriber identifier that uniquely identifies a particular subscriber to the searchable data service, wherein each subscriber identifier is associated with one or more bucket identifiers in the searchable data service, and wherein each bucket identifier is associated with exactly one subscriber.

20. The system as recited in claim 18, wherein a query expression includes:

one or more attributes of a searchable data service object in the particular searchable index; and

one or more operators to be applied to the one or more attributes in the query expression to locate a set of searchable data service objects that satisfy the query expression in the particular searchable index.

21. The system as recited in claim 20, wherein the operators include one or more Boolean operators.

22. The system as recited in claim 20, wherein the operators include one or more arithmetic operators.

23. The system as recited in claim 20, wherein the operators include one or more string comparison operators.

24. The system as recited in claim 1, wherein the plurality of nodes is further configured to:

paginate a list of the entity identifiers from the set of one or more searchable data service objects that satisfy a particular query request to generate two or more pages of entity identifiers that satisfy the particular query request; and

return the two or more pages of entity identifiers to a client application that initiated the query request in two or more response messages, wherein each response message includes one of the pages of entity identifiers.

25. The system as recited in claim 1, wherein the plurality of nodes is distributed among two or more data centers.

26. The system as recited in claim 25, wherein the data centers are geographically dispersed.

27. The system as recited in claim 25, wherein each data center includes, from the plurality of nodes:

one or more storage nodes configured to persistently store at least a portion of the searchable indexes;

one or more query nodes each configured to maintain a local query cache of responses to previous query requests; and

at least one coordinator node configured to;

receive service requests from the web services platform;

forward each storage request to one or more of the storage nodes in the data center; and

forward each query request to one or more of the query nodes in the data center.

28. The system as recited in claim 1, wherein the plurality of nodes comprises a plurality of storage nodes, wherein the plurality of storage nodes is distributed among two or more data centers, wherein each data center comprises two or more of the plurality of storage nodes;

wherein at least two of the storage nodes in a data center persistently store replicas of a partition of a particular searchable index; and

wherein at least two other storage nodes in at least one other data center persistently store replicas of the partition of the particular searchable index.

29. The system as recited in claim 1, wherein each of the plurality of nodes is configured to propagate searchable data service information to others of the plurality of nodes in accordance with a gossip protocol.

30. The system as recited in claim 29, wherein the propagated searchable data service information includes update information for partitions of a particular searchable index replicated to two or more of the plurality of nodes.

31. The system as recited in claim 29, wherein the propagated searchable data service information includes local health and status information for each of the plurality of nodes.

32. The system as recited in claim 29, wherein the plurality of nodes is configured to organize into groups of nodes each including a subset of the plurality of nodes, and wherein each group of nodes is configured to propagate searchable data service information within the group in accordance with the gossip protocol.

33. The system as recited in claim 1, wherein the web services platform further comprises one or more services that perform metering, billing, authentication, and access control of subscribers to the searchable data service, wherein a subscriber is an entity associated with one or more client applications each configured to access the searchable data service via the web service interface as a search frontend to one or more of the plurality of data stores used by the client application.

34. The system as recited in claim 1, wherein at least two of the plurality of nodes configured to participate in the searchable data service are implemented on heterogeneous devices.

35. A computer-implemented method, comprising:

receiving service requests from a plurality of client applications on a web service interface to a searchable data service, wherein the service requests comprise query requests and storage requests, and wherein said receiving comprises receiving the query requests and storage requests at a common message endpoint provided by the web service interface to the plurality of client applications to send the query requests and storage requests;

forwarding each service request from the web service interface to one of a plurality of nodes configured to participate in the searchable data service;

processing received storage requests on the plurality of nodes to store searchable data service objects specified in the storage requests in respective searchable indexes for a plurality of independent data stores used by the client applications, wherein the searchable indexes are on the plurality of nodes, wherein the data stores are on one or more storage devices each on a network and separate from the one or more computer devices that implement the plurality of nodes configured to participate in the searchable data service, wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores such that each searchable index provides a complete index for only one of the independent data stores, wherein each searchable data service object specifies two or more attributes of a particular entity in a particular data store, and wherein the attributes include a unique entity identifier for locating the particular entity in the particular data store;

processing each received query request on the plurality of nodes to locate a set of one or more searchable data service objects from the searchable indexes that satisfy the query request, wherein the received query requests specify one of the searchable indexes; and

returning at least the entity identifiers from the set of one or more searchable data service objects that satisfy the query request to the client applications in accordance with the web service interface.

36. The computer-implemented method as recited in claim 35, wherein each attribute is expressed as a {name, value} pair.

37. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises one or more storage nodes configured to persistently store the searchable indexes.

38. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises one or more query nodes each configured to maintain a local query cache of responses to previous query requests.

39. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises:

one or more storage nodes configured to persistently store the searchable indexes;

one or more query nodes each configured to maintain a local query cache of responses to previous query requests; and

one or more coordinator nodes configured to route service requests to others of the plurality of nodes.

40. The computer-implemented method as recited in claim 39, further comprising:

receiving the service requests from the web service interface on a coordinator node;

forwarding each storage request from the coordinator node to one or more of the storage nodes; and

forwarding each query request from the coordinator node to one or more of the query nodes.

41. The computer-implemented method as recited in claim 40, further comprising:

receiving a storage request from the coordinator node on a storage node; and

adding a searchable data service object specified in the storage request to a respective searchable index on the storage node.

42. The computer-implemented method as recited in claim 40, further comprising:

receiving a storage request from the coordinator node on a storage node; and

modifying a searchable data service object stored in a particular searchable index on the storage node as specified in the storage request.

43. The computer-implemented method as recited in claim 40, further comprising:

receiving a storage request from the coordinator node on a storage node; and

deleting at least a portion of a searchable data service object from a particular searchable index as specified in the storage request.

44. The computer-implemented method as recited in claim 40, further comprising:

receiving a query request from the coordinator node on a query node;

accessing a local query cache on the query node to determine if the query request can be satisfied from the local query cache;

if the query request can be satisfied from the local query cache, returning at least the entity identifiers from a set of one or more searchable data service objects from the local query cache that satisfy the query request from the query node to a client application that initiated the query request in accordance with the web service interface;

if the query request cannot be satisfied from the local query cache, forwarding the query request from the query node to one or more of the storage nodes.

45. The computer-implemented method as recited in claim 44, further comprising:

receiving on a storage node a query request from the query node;

searching a partition of a particular searchable index persistently stored by the storage node to locate a set of one or more searchable data service objects in the searchable index that satisfy the query request; and

returning at least the entity identifiers from the set of one or more searchable data service objects that satisfy the query request to a client application that initiated the query request in accordance with the web service interface.

46. The computer-implemented method as recited in claim 40, further comprising:

receiving on a query node a query request from a coordinator node;

forwarding the query request to two or more of the storage nodes that each persistently store a different partition of a particular searchable index from the query node, wherein the different partitions are within the scope of the query request;

receiving on the query node a different set of searchable data service objects from the particular searchable index that satisfy the query request from each of the two or more storage nodes to which the query request was forwarded;

merging the different sets of searchable data service objects on the query node;

and

returning at least the entity identifiers from the merged set of searchable data service objects to a client application that initiated the query request in accordance with the web service interface.

47. The computer-implemented method as recited in claim 40, further comprising:

receiving on a query node a query request from a coordinator node;

forwarding the query request to one or more of the storage nodes;

receiving on the query node searchable data service objects from a particular searchable index that satisfy the query request from at least one of the one or more storage nodes to which the query request was forwarded;

sorting the received searchable data service objects in accordance with a sort specification in the query request; and

returning at least the entity identifiers from the sorted searchable data service objects to a client application that initiated the query request in accordance with the web service interface and in sorted order.

48. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises two or more storage nodes, and wherein the method further comprises:

repartitioning a partition of a particular searchable index persistently stored by one of the storage nodes to generate two or more new partitions on the storage node, wherein each of the two or more new partitions includes a different subset of the searchable data service objects stored in the searchable index;

communicating between the storage node and one or more others of the storage nodes to locate another storage node that volunteers to receive one of the two or more new partitions; and

performing a cooperative operation between the storage node and the other storage node to move one of the two or more new partitions from the storage node to the other storage node.

49. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises two or more storage nodes, and wherein the method further comprises:

communicating between the storage nodes to locate a volunteer storage node that volunteers to receive a replica of a partition of a particular searchable index persistently stored by one of the storage nodes; and

performing a cooperative operation between the storage node and the volunteer storage node to replicate the partition to the other storage node.

50. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises two or more storage nodes, and wherein the method further comprises:

performing a cooperative operation between one of the storage nodes and another one of the storage nodes to replicate a partition of a particular searchable index persistently stored by the storage node to the other storage node in accordance with an anti-entropy protocol; and

communicating subsequent storage requests for the replicated partition from the storage node to the other storage node in accordance with a gossip protocol to update the replica of the partition on the other storage node.

51. The computer-implemented method as recited in claim 35, further comprising exposing a plurality of service requests for performing operations of the searchable data service to the client applications of the searchable data service via the web service interface, wherein the service requests exposed via the web service interface include:

an add request to add a specified searchable data service object to a specified searchable index;

a replace request to replace a specified portion or all of a specified searchable data service object in a specified searchable index;

a delete request to delete at least a portion of a specified searchable data service object from a specified searchable index;

a list attributes request to return a list of attributes for one or more specified searchable data service objects in a specified searchable index; and

a query request to return a set of entity identifiers of searchable data service objects in a specified searchable index that match a query expression specified in the query request.

52. The computer-implemented method as recited in claim 35, further comprising exposing a plurality of service requests for performing operations of the searchable data service to developers of client applications of the searchable data service via the web service interface, wherein the exposed plurality of service requests is configured for use in implementing search frontends to the data stores for the client applications.

53. The computer-implemented method as recited in claim 35, wherein each query request includes:

a bucket identifier that identifies a particular searchable index in the searchable data service, wherein a bucket defines the scope of a query request to the searchable data service; and

a query expression that specifies one or more search criteria to be applied to specified attributes of the searchable index identified by the bucket identifier to locate searchable data service objects that satisfy the query expression.

54. The computer-implemented method as recited in claim 53, wherein each query request further includes a subscriber identifier that uniquely identifies a particular subscriber to the searchable data service, wherein each subscriber identifier is associated with one or more bucket identifiers in the searchable data service, and wherein each bucket identifier is associated with exactly one subscriber.

55. The computer-implemented method as recited in claim 53, wherein a query expression includes:

one or more attributes of a searchable data service object in the particular searchable index; and

one or more operators to be applied to the one or more attributes in the query expression to locate a set of searchable data service objects that satisfy the query expression in the particular searchable index.

56. The computer-implemented method as recited in claim 55, wherein the operators include one or more Boolean operators.

57. The computer-implemented method as recited in claim 55, wherein the operators include one or more arithmetic operators.

58. The computer-implemented method as recited in claim 55, wherein the operators include one or more string comparison operators.

59. The computer-implemented method as recited in claim 35, further comprising:

paginating a list of the entity identifiers from the set of one or more searchable data service objects that satisfy a particular query request to generate two or more pages of entity identifiers that satisfy the query request; and

returning the two or more pages of entity identifiers to a client application that initiated the query request in two or more response messages, wherein each response message includes one of the pages of entity identifiers.

60. The computer-implemented method as recited in claim 35, wherein the plurality of nodes is distributed among two or more data centers.

61. The computer-implemented method as recited in claim 60, wherein the data centers are geographically dispersed.

62. The computer-implemented method as recited in claim 60, wherein each data center includes, from the plurality of nodes:

one or more storage nodes configured to persistently store at least a portion of the searchable indexes;

one or more query nodes each configured to maintain a local query cache of responses to previous query requests; and

at least one coordinator node;

wherein the method further comprises;

receiving on a coordinator node in a data center a service requests from the web service interface;

forwarding each storage request from the coordinator node to one or more of the storage nodes in the data center; and

forwarding each query request from the coordinator node to one or more of the query nodes in the data center.

63. The computer-implemented method as recited in claim 35, wherein the plurality of nodes comprises a plurality of storage nodes, wherein the plurality of storage nodes is distributed among two or more data centers, wherein each data center comprises two or more of the plurality of storage nodes;

wherein at least two of the storage nodes in a data center persistently store replicas of a partition of a particular searchable index; and

wherein at least two other storage nodes in at least one other data center persistently store replicas of the partition of the particular searchable index.

64. The computer-implemented method as recited in claim 35, further comprising propagating at least some searchable data service information from each of the plurality of nodes to others of the plurality of nodes in accordance with a gossip protocol.

65. The computer-implemented method as recited in claim 64, wherein the propagated searchable data service information includes update information for partitions of a particular searchable index replicated to two or more of the plurality of nodes.

66. The computer-implemented method as recited in claim 64, wherein the propagated searchable data service information includes local health and status information for each of the plurality of nodes.

67. The computer-implemented method as recited in claim 64, further comprising:

organizing the plurality of nodes into groups of nodes each including a subset of the plurality of nodes; and

propagating searchable data service information within each group of nodes in accordance with the gossip protocol.

68. The computer-implemented method as recited in claim 35, wherein the web service interface is provided by a web services platform, and wherein the web services platform further comprises one or more services that perform metering, billing, authentication, and access control of subscribers to the searchable data service, wherein a subscriber is an entity associated with one or more client applications each configured to access the searchable data service via the web service interface as a search frontend to one or more of the plurality of data stores used by the client application.

69. A computer-accessible memory medium comprising program instructions, wherein the program instructions are configured to implement a searchable data service on a plurality of nodes, wherein the searchable data service is configured to:

receive service requests from a plurality of client applications via a web services platform configured to provide a web service interface to the searchable data service, wherein the service requests comprise query requests and storage requests, and wherein the web service interface provides a common message endpoint to the plurality of client applications to send the query requests and storage requests;

process received storage requests on the plurality of nodes to store searchable data service objects specified in the storage requests in respective searchable indexes for a plurality of independent data stores used by the client applications, wherein the searchable indexes are on the plurality of nodes, wherein the data stores are on one or more storage devices each on a network and separate from the one or more computer devices that implement the plurality of nodes configured to participate in the searchable data service, wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores such that each searchable index provides a complete index for only one of the independent data stores, wherein each searchable data service object specifies two or more attributes of a particular entity in a particular data store, and wherein the attributes include a unique entity identifier for locating the particular entity in the particular data store;

process received query requests on the plurality of nodes, wherein each received query request is processed to locate a set of one or more searchable data service objects from the searchable indexes that satisfy the query request wherein the query request specifies one of the searchable indexes; and

return at least the entity identifiers from the set of one or more searchable data service objects that satisfy the query request to the client applications in accordance with the web service interface.

70. The computer-accessible memory medium as recited in claim 69, wherein each attribute is expressed as a {name, value} pair.

71. The computer-accessible memory medium as recited in claim 69, wherein the plurality of nodes comprises one or more storage nodes configured to persistently store the searchable indexes.

72. The computer-accessible memory medium as recited in claim 69, wherein the plurality of nodes comprises one or more query nodes each configured to maintain a local query cache of responses to previous query requests.

73. The computer-accessible memory medium as recited in claim 69, wherein the program instructions are configured to implement the searchable data service on the plurality of nodes as:

one or more storage nodes configured to persistently store the searchable indexes;

one or more query nodes each configured to maintain a local query cache of responses to previous query requests; and

one or more coordinator nodes each configured to;

receive service requests from the web services platform;

forward each storage request to one or more of the storage nodes; and

forward each query request to one or more of the query nodes.

74. The computer-accessible memory medium as recited in claim 73, wherein each of the one or more storage nodes is configured to:

receive a storage request from a coordinator node; and

add a searchable data service object specified in the storage request to a respective searchable index.

75. The computer-accessible memory medium as recited in claim 73, wherein each of the one or more storage nodes is configured to:

receive a storage request from a coordinator node; and

modify a searchable data service object stored in a particular searchable index as specified in the storage request.

76. The computer-accessible memory medium as recited in claim 73, wherein each of the one or more storage nodes is configured to:

receive a storage request from a coordinator node; and

delete at least a portion of a searchable data service object from a particular searchable index as specified in the storage request.

77. The computer-accessible memory medium as recited in claim 73, wherein each of the query nodes is configured to:

receive a query request from a coordinator node;

access the local query cache on the query node to determine if the query request can be satisfied from the local query cache;

if the query request can be satisfied from the local query cache, return at least the entity identifiers from a set of one or more searchable data service objects from the local query cache that satisfy the query request to a client application that initiated the query request in accordance with the web service interface;

if the query request cannot be satisfied from the local query cache, forward the query request to one or more of the storage nodes.

78. The computer-accessible memory medium as recited in claim 77, wherein each of the one or more storage nodes is configured to:

receive a query request from a query node;

search a partition of a particular searchable index persistently stored by the storage node to locate a set of one or more searchable data service objects in the searchable index that satisfy the query request; and

return at least the entity identifiers from the set of one or more searchable data service objects that satisfy the query request to a client application that initiated the query request in accordance with the web service interface.

79. The computer-accessible memory medium as recited in claim 73, wherein each of the query nodes is configured to:

receive a query request from a coordinator node;

forward the query request to two or more of the storage nodes that each persistently store a different partition of a particular searchable index, wherein the different partitions are within the scope of the query request;

receive a different set of searchable data service objects from the searchable index that satisfy the query request from each of the two or more storage nodes to which the query request was forwarded;

merge the different sets of searchable data service objects; and

return at least the entity identifiers from the merged set of searchable data service objects to a client application that initiated the query request in accordance with the web service interface.

80. The computer-accessible memory medium as recited in claim 73, wherein each of the query nodes is configured to:

receive a query request from a coordinator node;

forward the query request to one or more of the storage nodes;

receive searchable data service objects from a particular searchable index that satisfy the query request from at least one of the one or more storage nodes to which the query request was forwarded;

sort the received searchable data service objects in accordance with a sort specification in the query request; and

return at least the entity identifiers from the sorted searchable data service objects to a client application that initiated the query request in accordance with the web service interface and in sorted order.

81. The computer-accessible memory medium as recited in claim 69, wherein the program instructions are configured to implement two or more of the plurality of nodes as storage nodes each configured to:

persistently store a partition of a particular searchable index;

repartition the partition to generate two or more new partitions, wherein each of the two or more new partitions includes a different subset of the searchable data service objects stored in the searchable index;

communicate with others of the storage nodes to locate another storage node that volunteers to receive one of the two or more new partitions; and

cooperate with the other storage node to move one of the two or more new partitions to the other storage node.

82. The computer-accessible memory medium as recited in claim 69, wherein the program instructions are configured to implement two or more of the plurality of nodes as storage nodes each configured to:

persistently store a partition of a particular searchable index;

communicate with one or more others of the storage nodes to locate another storage node that volunteers to receive a replica of the partition; and

cooperate with the other storage node to replicate the partition to the other storage node.

83. The computer-accessible memory medium as recited in claim 69, wherein the program instructions are configured to implement two or more of the plurality of nodes as storage nodes each configured to:

persistently store a partition of a particular searchable index;

replicate the partition to another storage node in accordance with an anti-entropy protocol; and

communicate subsequent storage requests for the replicated partition to the other storage node in accordance with a gossip protocol to update the replica of the partition on the other storage node.

84. The computer-accessible memory medium as recited in claim 69, wherein the web service interface exposes a plurality of service requests for performing operations of the searchable data service to the client applications of the searchable data service, wherein the service requests include:

an add request to add a specified searchable data service object to a specified searchable index;

a replace request to replace a specified portion or all of a specified searchable data service object in a specified searchable index;

a delete request to delete at least a portion of a specified searchable data service object from a specified searchable index;

a list attributes request to return a list of attributes for one or more specified searchable data service objects in a specified searchable index; and

a query request to return a set of entity identifiers of searchable data service objects in a specified searchable index that match a query expression specified in the query request.

85. The computer-accessible memory medium as recited in claim 69, wherein the web service interface exposes a plurality of service requests for performing operations of the searchable data service to developers of client applications of the searchable data service, wherein the exposed plurality of service requests is configured for use in implementing search frontends to the data stores for the client applications.

86. The computer-accessible memory medium as recited in claim 69, wherein each query request includes:

a bucket identifier that identifies a particular searchable index in the searchable data service, wherein a bucket defines the scope of a query request to the searchable data service; and

a query expression that specifies one or more search criteria to be applied to specified attributes of the searchable index identified by the bucket identifier to locate searchable data service objects that satisfy the query expression.

87. The computer-accessible memory medium as recited in claim 86, wherein each query request further includes a subscriber identifier that uniquely identifies a particular subscriber to the searchable data service, wherein each subscriber identifier is associated with one or more bucket identifiers in the searchable data service, and wherein each bucket identifier is associated with exactly one subscriber.

88. The computer-accessible memory medium as recited in claim 86, wherein a query expression includes:

one or more attributes of a searchable data service object in the particular searchable index; and

one or more operators to be applied to the one or more attributes in the query expression to locate a set of searchable data service objects that satisfy the query expression in the particular searchable index.

89. The computer-accessible memory medium as recited in claim 88, wherein the operators include one or more Boolean operators.

90. The computer-accessible memory medium as recited in claim 88, wherein the operators include one or more arithmetic operators.

91. The computer-accessible memory medium as recited in claim 88, wherein the operators include one or more string comparison operators.

92. The computer-accessible memory medium as recited in claim 69, wherein the searchable data service is further configured to:

paginate a list of the entity identifiers from the set of one or more searchable data service objects that satisfy a particular query request to generate two or more pages of entity identifiers that satisfy the query request; and

return the two or more pages of entity identifiers to a client application that initiated the query request in two or more response messages, wherein each response message includes one of the pages of entity identifiers.

93. The computer-accessible memory medium as recited in claim 69, wherein the plurality of nodes is distributed among two or more data centers.

94. The computer-accessible memory medium as recited in claim 93, wherein the data centers are geographically dispersed.

95. The computer-accessible memory medium as recited in claim 93, wherein the program instructions are configured to implement, in each of the data centers:

one or more storage nodes configured to persistently store at least a portion of the searchable indexes;

one or more query nodes each configured to maintain a local query cache of responses to previous query requests; and

at least one coordinator node configured to;

receive service requests from the web services platform;

forward each storage request to one or more of the storage nodes in the data center; and

forward each query request to one or more of the query nodes in the data center.

96. The computer-accessible memory medium as recited in claim 69, wherein the program instructions are configured to implement a plurality of storage nodes for the searchable data service, wherein the plurality of storage nodes is distributed among two or more data centers, wherein each data center comprises two or more of the plurality of storage nodes;

wherein at least two of the storage nodes in a data center persistently store replicas of a partition of a particular searchable index; and

wherein at least two other storage nodes in at least one other data center persistently store replicas of the partition of the particular searchable index.

97. The computer-accessible memory medium as recited in claim 69, wherein each of the plurality of nodes is configured to propagate searchable data service information to others of the plurality of nodes in accordance with a gossip protocol.

98. The computer-accessible memory medium as recited in claim 97, wherein the propagated searchable data service information includes update information for partitions of a particular searchable index replicated to two or more of the plurality of nodes.

99. The computer-accessible memory medium as recited in claim 97, wherein the propagated searchable data service information includes local health and status information for each of the plurality of nodes.

100. The computer-accessible memory medium as recited in claim 97, wherein the plurality of nodes is configured to organize into groups of nodes each including a subset of the plurality of nodes, and wherein each group of nodes is configured to propagate searchable data service information within the group in accordance with the gossip protocol.

101. The computer-accessible memory medium as recited in claim 69, wherein the web services platform further comprises one or more services configured to perform metering, billing, authentication, and access control of subscribers to the searchable data service, wherein a subscriber is an entity associated with one or more client applications each configured to access the searchable data service via the web service interface as a search frontend to one or more of the plurality of data stores used by the client application.

102. A searchable data service node, comprising:

one or more processors;

a network interface configured to communicatively couple to a network; and

a memory comprising program instructions, wherein the program instructions are executable by the one or more processors to;

receive via the network searchable data service requests from a plurality of client applications in accordance with a web service interface, wherein the searchable data service requests comprise query requests, and wherein the web service interface provides a common message endpoint to the plurality of client applications to send the query requests;

locate sets of one or more searchable data service objects in respective searchable indexes for a plurality of independent data stores used by the client applications that satisfy received query requests, wherein the query requests specify one of the searchable indexes, wherein the searchable indexes are distributed on a plurality of searchable data service nodes on the network, wherein the data stores are on one or more storage devices each on the network and separate from computer devices on the network that implement the searchable data service nodes, wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores such that each searchable index provides a complete index for only one of the independent data stores, wherein each searchable data service object specifies two or more attributes of a particular entity in a particular data store, and wherein the attributes include a unique entity identifier for locating the particular entity in the particular data store; and

return via the network at least the entity identifiers from the sets of one or more searchable data service objects that satisfy the query requests to the client applications in accordance with the web service interface.

103. The searchable data service node as recited in claim 102, wherein the searchable data service requests further comprise storage requests, and wherein the program instructions are further executable by the one or more processors to store searchable data service objects specified in received storage requests in respective searchable indexes for the plurality of independent data stores used by the client applications.

104. The searchable data service node as recited in claim 103, wherein the searchable data service node is a searchable data service coordinator node;

wherein, to store searchable data service objects specified in received storage requests in the searchable indexes, the program instructions are further executable by the one or more processors to forward each storage request to one or more searchable data service storage nodes; and

wherein, to locate sets of one or more searchable data service objects from the searchable indexes that satisfy received query requests, the program instructions are further executable by the one or more processors to forward each query request to one or more searchable data service query nodes.

105. The searchable data service node as recited in claim 102, wherein each attribute is expressed as a {name, value} pair.

106. The searchable data service node as recited in claim 102, wherein the searchable data service node is a searchable data service storage node further comprising one or more data storage devices, and wherein the program instructions are further executable by the one or more processors to persistently store at least one partition of at least one searchable index on the one or more data storage devices.

107. The searchable data service node as recited in claim 106, wherein the program instructions are further executable by the one or more processors to:

repartition a partition of a particular searchable index stored on the one or more data storage devices to generate two or more new partitions on the one or more data storage devices, wherein each of the two or more new partitions includes a different subset of the searchable data service objects stored in the particular searchable index;

communicate with one or more other searchable data service storage nodes to locate another searchable data service storage node to receive one of the two or more new partitions; and

cooperate with the other searchable data service storage node to move one of the two or more new partitions to the other searchable data service storage node.

108. The searchable data service node as recited in claim 106, wherein the program instructions are further executable by the one or more processors to:

communicate with one or more other searchable data service storage nodes to locate another searchable data service storage node to receive a replica of a partition of a particular searchable index stored on the one or more data storage devices; and

cooperate with the other searchable data service storage node to replicate the partition to the other searchable data service storage node.

109. The searchable data service node as recited in claim 106, wherein the program instructions are further executable by the one or more processors to:

replicate a partition of a particular searchable index stored on the one or more data storage devices to another searchable data service storage node in accordance with an anti-entropy protocol; and

communicate subsequent storage requests for the replicated partition to the other s searchable data service storage node in accordance with a gossip protocol to update the replica of the partition on the other searchable data service storage node.

110. The searchable data service node as recited in claim 102, wherein the searchable data service node is a searchable data service query node, and wherein the program instructions are further executable by the one or more processors to maintain a local query cache of responses to previous query requests.

111. The searchable data service node as recited in claim 110, wherein, to locate sets of one or more searchable data service objects from the searchable indexes that satisfy received query requests, the program instructions are further executable by the one or more processors to:

receive a query request;

access the local query cache on the query node to determine if the query request can be satisfied from the local query cache;

if the query request can be satisfied from the local query cache, return at least the entity identifiers from a set of one or more searchable data service objects from the local query cache that satisfy the query request to a client application that initiated the query request in accordance with the web service interface;

if the query request cannot be satisfied from the local query cache, forward the query request to one or more searchable data service storage nodes.

112. The searchable data service node as recited in claim 102, wherein each query request includes:

a bucket identifier that identifies a particular searchable index in the searchable data service, wherein a bucket defines the scope of a query request to the searchable data service; and

a query expression that specifies one or more search criteria to be applied to specified attributes of the searchable index identified by the bucket identifier to locate searchable data service objects that satisfy the query expression.

113. The searchable data service node as recited in claim 102, wherein the program instructions are further executable by the one or more processors to propagate searchable data service information to one or more other searchable data service nodes in accordance with a gossip protocol.

114. The searchable data service node as recited in claim 113, wherein the propagated searchable data service information includes update information for one or more partitions of a particular searchable index replicated to two or more of the searchable data service nodes.

115. The searchable data service node as recited in claim 102, wherein the propagated searchable data service information includes local health information for the searchable data service node.

116. A computer-implemented method, comprising:

receiving query requests from a plurality of client applications on a web service interface to a searchable data service, wherein said receiving comprises receiving the query requests at a common message endpoint provided by the web service interface to the plurality of client applications to send the query requests;

forwarding each query request from the web service interface to at least one of a plurality of nodes configured to participate in the searchable data service, wherein each query request comprises an identification of a searchable index and search criteria for the searchable data service, wherein the plurality of nodes store searchable indexes for a plurality of independent data stores used by the client applications, wherein the data stores are on one or more storage devices separate from the plurality of nodes, and wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores such that each searchable index provides a complete index for only one of the independent data stores;

receiving a query result for each query request from a respective one of the plurality of nodes, wherein each query result comprises an entity identifier from each of one or more searchable data service objects from a particular searchable index of the searchable data service that satisfy the search criteria of the query request; and

returning at least the entity identifier from each of the one or more searchable data service objects that satisfy the query request to the client application in accordance with the web service interface, wherein each entity identifier locates a particular entity in a particular data store used by the client applications.

117. The computer-implemented method as recited in claim 116, further comprising:

receiving storage requests from the client applications on the web service interface to the searchable data service;

forwarding each storage request from the web service interface to at least one of the plurality of nodes configured to participate in the searchable data service, wherein each storage request specifies an update operation for a particular searchable index;

receiving an update result for each storage request from a respective one of the plurality of nodes; and

returning an update response for each storage request to a client application that initiated the storage request in accordance with the web service interface, wherein each update response comprises the update result for the update operation specified by the corresponding update request.

118. The computer-implemented method as recited in claim 116, wherein each searchable data service object specifies two or more attributes of a particular entity in a particular data store, and wherein the attributes include the unique entity identifier for locating the particular entity in the particular data store.

119. The computer-implemented method as recited in claim 116, wherein the plurality of nodes comprises:

one or more storage nodes configured to persistently store the searchable indexes and to satisfy query requests;

one or more query nodes each configured to satisfy query requests from a local query cache of responses to previous query requests and to forward unsatisfied query requests to the storage nodes; and

one or more coordinator nodes configured to route requests including query requests received from the web service interface to others of the plurality of nodes.

120. The computer-implemented method as recited in claim 116, further comprising exposing a plurality of service requests for performing operations of the searchable data service to the client applications of the searchable data service via the web service interface, wherein the service requests exposed via the web service interface include:

an add request to add a specified searchable data service object to a specified searchable index;

a replace request to replace a specified portion or all of a specified searchable data service object in a specified searchable index;

a delete request to delete at least a portion of a specified searchable data service object from a specified searchable index;

a list attributes request to return a list of attributes for one or more specified searchable data service objects in a specified searchable index; and

a query request to return a set of entity identifiers of searchable data service objects in a specified searchable index that match a query expression specified in the query request.

121. The computer-implemented method as recited in claim 116, further comprising exposing a plurality of service requests for performing operations of the searchable data service to developers of client applications of the searchable data service via the web service interface, wherein the exposed plurality of service requests is configured for use in implementing search frontends to the data stores for the client applications.

122. The computer-implemented method as recited in claim 116, wherein each query request includes:

a bucket identifier that identifies a particular searchable index in the searchable data service, wherein a bucket defines the scope of a query request to the searchable data service; and

a query expression that specifies one or more search criteria to be applied to specified attributes of the searchable index identified by the bucket identifier to locate searchable data service objects that satisfy the query expression.

123. The computer-implemented method as recited in claim 116, wherein the web service interface is provided by a web services platform, and wherein the web services platform further comprises one or more services that perform metering, billing, authentication, and access control of subscribers to the searchable data service, wherein a subscriber is an entity associated with one or more client applications each configured to access the searchable data service via the web service interface as a search frontend to one or more of the plurality of data stores used by the client application.