RESILIENT DATA PROCESSING PIPELINE ARCHITECTURE

• US 20140007132A1
• Filed: 03/08/2013
• Published: 01/02/2014
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: GAXIOLA, DAVID GABRIEL, ALLEN, SEAN IAN
• Granted Patent: US 9,396,038 B2
• Time in Patent Office: Days
• US Class Current: 718/105
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 67/1004 Server selection for load balancing
• CPC Class Codes: H04L 69/08 Protocols for interworking; Protocol conversion

Abstract

A private cloud-based processing pipeline apparatus and method for its use is disclosed. A first load balancer directs data packets to one of a plurality of collectors. The collectors authenticate the data packet. Then a second load balancer receives the data packet from the collector and to direct the data packet to one of a plurality of extract transform load (ETL) frontends, where the data packet is converted from a platform dependent form into a platform independent form and loaded into a queue. Handlers then process the converted data packets. It is emphasized that this abstract is provided to comply with the rules requiring an abstract that will allow a searcher or other reader to quickly ascertain the subject matter of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

Claims

1. A processing pipeline apparatus, comprising:

a first load balancer configured to direct a declarative and defined data packet, to one of a plurality of collectors, wherein each collector in the plurality is configured to verify that the data packet is authorized to access the private cloud-based processing pipeline;

a second load balancer configured to receive the data packet from a collector in the plurality and to direct the data packet to one of a plurality of extract transform load (ETL) frontends, wherein each of the ETL frontends is configured to convert the data packet from a platform dependent form into a platform independent form and load the resulting converted data packet into a queue; and

a plurality of handlers, wherein one or more handlers in the plurality are configured to process the data packet within the queue.

2. The apparatus of claim 1, wherein the platform dependent form is a binary format.

3. The apparatus of claim 1, wherein the platform independent form is Extensible Markup Language (XML) format.

4. The apparatus of claim 1, wherein the platform independent form is JavaScript Object Notation (JSON) format.

5. The apparatus of claim 1, wherein one or more of the collectors is a stateless machine.

6. The apparatus of claim 5, wherein at least one collector in the plurality of collectors is configured to validate data packets containing telemetry data.

7. The apparatus of claim 5, wherein at least one collector in the plurality of collectors is configured to validate data packets containing advertising data.

8. The apparatus of claim 5, wherein at least one collector in the plurality of collectors is configured to validate data packets containing virtual world data.

9. The apparatus of claim 5, wherein the number of collectors in the plurality of collectors is dynamically scalable.

10. The apparatus of claim 1, wherein the data packet is verified by the collector when the data packet originates from a validated client device platform.

11. The apparatus of claim 1, wherein a data packet is verified by the collector when the data packet contains an encrypted message indicating that access to the private cloud-based processing pipeline has been paid for.

12. The apparatus of claim 1, wherein each of the ETL frontends in the plurality of ETL frontends is stateless a machine.

13. The apparatus of claim 12, wherein the number of ETL frontends in the plurality of ETL frontends is dynamically scalable.

14. The apparatus of claim 1, wherein each of the handlers in the plurality of handlers is a stateless machine.

15. The apparatus of claim 14, wherein the number of handlers in the plurality of handlers is dynamically scalable.

17. The apparatus of claim 14, wherein at least one handler in the plurality of handlers is configured to process a data packet containing user chat information.

18. The apparatus of claim 14, wherein at least one handler in the plurality of handlers is configured to store the data packet to a back-up drive.

19. The apparatus of claim 14, wherein at least one handler in the plurality of handlers is configured to return the data packet to the queue if it is not able to process the data packet.

20. The apparatus of claim 19, wherein the at least one handler in the plurality of handlers is further configured to generate and deliver a message to the queue indicating that the unprocessed data packet needs to be revisited at a later time.

21. The apparatus of claim 14, wherein the data packet is processed in a different order than an order in which the data packet was delivered to the queue.

22. The apparatus of claim 14, wherein at least one handler in the plurality of handlers is configured to deliver the data packet to a data processing software system.

23. The apparatus of claim 22, wherein the data processing software system is developed on a Hadoop framework.

24. The apparatus of claim 1, wherein the first load balancer has an election amongst a plurality of first load balancers from other pipelines to determine which pipeline of a plurality of pipelines will process the data packet.

16. The apparatus of 14, wherein at least one handler in the plurality of handlers is configured to process a data packet containing email information.

25. A method for processing plurality of data packets in a processing pipeline, comprising:

a) receiving a plurality of declarative and defined data packets;

b) directing each data packet to one of a plurality of collectors;

c) verifying that each data packet is authorized to access the private cloud-based processing pipeline with one or more collectors in the plurality;

d) delivering each of the data packets to a particular extract transform load (ETL) frontend in a plurality of ETL frontends;

e) converting the data packets from a platform dependent form to a platform independent form with the particular ETL frontend to produce converted data packets;

f) delivering the converted data packets to a queue; and

g) processing each of the converted data packets with at least one handler of a plurality of handlers.

26. The method of claim 25, wherein step b) is performed by a first load balancer.

27. The method of claim 25, wherein step d) is performed by a second load balancer.

28. The method of claim 25, wherein the platform dependent form is a binary format.

29. The method of claim 25, wherein the platform independent form is Extensible Markup Language (XML) format.

30. The method of claim 25, wherein the platform independent form is JavaScript Object Notation (JSON) format.

31. The method of claim 25, wherein each of the collectors in the plurality of collectors is a stateless machine.

32. The method of claim 31, wherein at least one collector in the plurality of collectors is configured to validate one or more data packets in the plurality of declarative and defined data packets that contain telemetry data.

33. The method of claim 31, wherein at least one collector in the plurality of collectors is configured to validate one or more data packets in the plurality of declarative and defined data packets that contain advertising data.

34. The method of claim 31, wherein at least one collector in the plurality of collectors is configured to validate one or more data packets in the plurality of declarative and defined data packets that contain virtual world data.

35. The method of claim 31, wherein the number of collectors in the plurality of collectors is dynamically scalable.

36. The method of claim 25, wherein validating the data packets includes determining whether one or more data packets in the plurality of declarative and defined data packets originated from a validated client device platform.

37. The method of claim 25, wherein validating the data packets includes determining whether one or more data packets in the plurality of declarative and defined data packets contains an encrypted message indicating that access to the private cloud-based processing pipeline has been paid for.

38. The method of claim 25, wherein each of the ETL frontends in the plurality of ETL frontends is a stateless machine.

39. The method of claim 38, wherein the number of ETL frontends in the plurality of ETL frontends is dynamically scalable.

40. The method of claim 25, wherein the handlers are stateless machines.

41. The method of claim 40, wherein the number of handlers in the plurality of handlers is dynamically scalable.

42. The method of claim 40, wherein at least one handler in the plurality of handlers is configured to process one or more data packets in the plurality of declarative and defined data packets that contain email information.

43. The method of claim 40, wherein at least one handler in the plurality of handlers is configured to process one or more data packets in the plurality of declarative and defined data packets that contain user chat information.

44. The method of claim 40, wherein at least one handler is configured to store one or more data packets in the plurality of declarative and defined data packets to a back-up drive.

45. The method of claim 40, wherein at least one handler is configured to deliver the data packet to a data processing software system.

46. The method of claim 45, wherein the data processing software system is developed on a Hadoop framework.

47. The method of claim 25, wherein a processing order of the data packets in the plurality of data packets is different than the order in which the data packets were delivered to the queue.

48. The method of claim 25, further comprising:

f′

) returning a given data packet in the plurality of data packets to the queue if the at least one handler is not able to process the given data packet.

49. The method of claim 25, further comprising:

g′

) generating and delivering a message to the queue indicating that an unprocessed data packet in the plurality of data packets needs to be revisited at a later time.

50. The method of claim 25, wherein a) further includes holding an election amongst a plurality of other pipelines to determine which pipeline will receive a given data packet in the plurality of data packets.

51. A nontransitory computer readable medium containing program instructions for optical switching, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out a method for processing plurality of data packets in a processing pipeline, the method comprising:

a) receiving a plurality of declarative and defined data packets;

b) directing each data packet to one of a plurality of collectors;

c) verifying that each data packet is authorized to access the private cloud-based processing pipeline with one or more collectors in the plurality;

d) delivering each of the data packets to a particular extract transform load (ETL) frontend in a plurality of ETL frontends;

e) converting the data packets from a platform dependent form to a platform independent form with the particular ETL frontend to produce converted data packets;

f) delivering the converted data packets to a queue; and

g) processing each of the converted data packets with at least one handler of a plurality of handlers.