Publish-subscribe messaging with distributed processing

• US 10,666,712 B1
• Filed: 06/10/2016
• Issued: 05/26/2020
• Est. Priority Date: 06/10/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Dandekar, Chetan Manohar, Jain, Saral
• Primary Examiner(s): Huq, Farzana B
• Application Number:
  US15/179,886
• Time in Patent Office: 1,446 Days
• Field of Search: 709206, 709222, 709217, 709228, 709245
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: Y04S 40/18 Network protocols supporting networked applications, e.g. including control of end-device applications over a network

Abstract

Technology for a publish-subscribe messaging method may include determining a schema for structuring raw data published in a publication by a publisher in a publish-subscribe system at a channel node in a network and distributing the schema from the channel node to other channel nodes in the network. The method may include identifying the publication from the publisher at one of the channel nodes and applying the schema to the raw data of the publication at the channel node, transforming the raw data to transformed data. The transformed data may be provided at a subscriber in the publish-subscribe system.

Claims

1. A computer-implemented method for distributed processing of raw data in a publish-subscribe system, comprising:

determining, by a broker node managing publish-subscribe messaging of nodes in the publish-subscribe system, a schema defining a structure to be applied to the raw data in a publication message published by a publisher device in the publish-subscribe system, using publisher data included in the publication message;

distributing, using a publish-subscribe channel that includes other nodes in the publish-subscribe system, the schema defining the structure to be applied to the raw data from the broker node to the other nodes to enforce the schema across the publish-subscribe channel;

receiving the publication message from the publisher device;

transforming the raw data in the publication message to comply with the schema defining the structure to be applied to the raw data, resulting in transformed data;

identifying a rule to execute on the transformed data based on the schema, the rule being distributable across the other nodes;

determining whether to execute the rule at one of the other nodes based in part on one or more device conditions or network conditions being monitored for the other nodes; and

sending the rule from the broker node to the determined one of the other nodes to process the transformed data using the rule, resulting in processed data at a subscriber in the publish-subscribe system.

2. The method of claim 1, wherein the raw data from the publisher device is streaming data.

3. The method of claim 1, wherein the schema comprises a plurality of schemas and the publisher device is a single publisher device, the method further comprising applying the plurality of schemas to the raw data.

4. The method of claim 1, further comprising monitoring the other nodes for the one or more device conditions or network conditions, wherein the one or more device conditions or network conditions include at least one of battery power, network bandwidth, network latency, privacy, processor availability, processor speed, or memory availability.

5. The method of claim 1, wherein determining the schema comprises automatically detecting a device schema used by the publisher device and providing the device schema to an administrator for approval or modification.

6. A computer-implemented method, comprising:

determining, at a channel node in a network, a schema defining a structure to be applied to raw data published in a publication message by a publisher device in a publish-subscribe system that includes the channel node, using data of the publication message;

distributing, using a publish-subscribe channel that includes other nodes in the network, the schema defining the structure to be applied to the raw data from the channel node to the other nodes in the network to enforce the schema across the network;

identifying the publication message from the publisher device at the channel node;

determining, based in part on one or more device conditions or network conditions at the other nodes in the network, whether to apply the schema to the raw data of the publication message, at the channel node, in order to transform the raw data to transformed data;

determining, based in part on the one or more device conditions or network conditions at the other nodes, whether to execute a rule at the channel node on the transformed data to process the transformed data using the rule, resulting in processed data; and

providing the processed data to a subscriber in the publish-subscribe system using the channel node.

7. The method of claim 6, further comprising determining whether the publisher device is producing invalid raw data by comparing the raw data in the publication message to the schema or to historical data in a historical publication message.

8. The method of claim 7, wherein determining whether the publisher device is producing invalid raw data by comparing the raw data to the historical data comprises comparing a number of data points in the raw data of the publication message with a number of data points in the historical data of the historical publication message, and identifying the raw data as invalid when the number of data points in the raw data of the publication message and the number of data points in the historical data are different.

9. The method of claim 6, further comprising instructing the publisher device to follow the schema and publish the transformed data.

10. The method of claim 6, further comprising:

determining schemas for raw data from a plurality of publisher devices,

determining a common schema to be applied to the raw data from the plurality of publisher devices; and

transforming the raw data from the plurality of publisher devices to the common schema.

11. The method of claim 6, wherein determining the schema comprises building a schema based at least in part on a topic structure of the raw data, a message structure of the raw data, message content, or device information for the publisher device.

12. The method of claim 6, wherein determining the schema comprises receiving definition of the schema from an administrator.

13. The method of claim 6, further comprising:

receiving a definition of the rule to execute on the transformed data from an administrator; and

distributing the rule to one of a plurality of channel nodes in the network.

14. The method of claim 13, further comprising:

determining one of the plurality of channel nodes at which to execute the rule based on at least one of battery power, network bandwidth, processor speed, or network latency; and

sending the rule from the channel node to the determined one of the plurality of channel nodes, to process the transformed data.

15. The method of claim 13, further comprising sending the rule from the channel node to one of the plurality of channel nodes to partially execute the rule and sending the rule from the channel node to another one of the plurality of channel nodes to completing execution of the rule.

16. The method of claim 13, wherein the rule comprises a plurality of rules, the method further comprising executing a first rule of the plurality of rules at a first node of the plurality of channel nodes and executing a second rule of the plurality of rules at a second node of the plurality of channel nodes.

17. A non-transitory computer-readable medium comprising computer-executable instructions which implement a publish-subscribe network, comprising:

a schema detector configured for detecting a schema defining a structure to be applied to raw data in a publication message published by a publisher device in the publish-subscribe network that includes a plurality of channel nodes, using publisher data included in the publication message at one channel node in the plurality of channel nodes;

a network distribution engine configured for distributing, using a publish-subscribe channel that includes the plurality of nodes, the schema defining the structure to be applied to the raw data from the one channel node to another channel node in the plurality of channel nodes in the publish-subscribe network, including to the publisher device, to enforce the schema across the publish-subscribe network;

a schema engine for applying the schema to the raw data to transform the raw data to transformed data; and

a rules engine for executing a rule on the transformed data for processing the data into processed data;

wherein the network distribution engine monitors the plurality of channel nodes to distribute, based in part on one or more device conditions or network conditions monitored for the plurality of nodes, the rule from the one channel node to the other channel node in the publish-subscribe network to optimize performance of the plurality of channel nodes.

18. The non-transitory computer readable medium of claim 17, wherein the rules engine is configured to determine at which of the plurality of channel nodes to execute the rule based on the one or more device conditions or network conditions monitored for the plurality of channel nodes.

19. The non-transitory computer readable medium of claim 17, further comprising a validation engine to validate the raw data from the publisher device to ensure the raw data is valid, and wherein the network distribution engine is further configured to notify an administrator if the raw data is invalid.

20. The non-transitory computer readable medium of claim 17, wherein the publisher device is an internet of things (IoT) device having a transducer for detecting a condition and expressing the condition as the raw data.