Flow control in very large query result sets using a release message to confirm that a client computer is ready to receive the data associated with a data collection operation

• US 9,213,735 B1
• Filed: 01/25/2012
• Issued: 12/15/2015
• Est. Priority Date: 01/25/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Legakis, Justin, Gudmundson, Stephan, Halem, Daniel, Manoharan, Gokul Nath Babu
• Primary Examiner(s): Murray, Daniel C
• Application Number:
  US13/357,950
• Time in Patent Office: 1,420 Days
• Field of Search: 709/224, 709/223, 709/232, 370/235, 707/736, 707/769
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24549 Run-time optimisation
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2457 with adaptation to user needs
• CPC Class Codes: G06F 9/4484 Executing subprograms
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

Apparatuses and methods for providing flow control over a network are disclosed. One such method comprises receiving a first remote procedure call at a first computer from a second computer remote from the first computer, the first computer including a first memory and a first processor configured to execute instructions stored in the first memory, generating a second remote procedure call from the first processor to the second computer, the second remote procedure call responsive to the first remote procedure call, monitoring the first processor for receipt of a release signal from the second computer, the release signal indicating when the second computer is able to receive data responsive to the first remote procedure call, and transmitting the data from the first memory of the first computer to the second computer responsive to receipt of the release signal at the first processor.

Claims

1. A computer-implemented method comprising:

receiving, by a coordinator computer and from a client computer that is remote from the coordinator computer, a request to initiate execution of a data collection operation;

in response to receiving the request to initiate execution of the data collection operation, establishing a feedback communication channel by transmitting a flow control message from the coordinator computer to the client computer;

after transmitting the flow control message from the coordinator computer to the client computer, receiving, from the client computer, a release message that indicates that the client computer is ready to receive data associated with the data collection operation;

in response to receiving the release message from the client computer, relaying, by the coordinator computer to at least one worker computer, the release message that indicates that the client computer is ready to receive data associated with the data collection operation, as received by the coordinator computer and from the client computer;

after relaying the release message that indicates that the client computer is ready to receive data associated with the data collection operation to the at least one worker computer, receiving data associated with the data collection operation from the at least one worker computer; and

transmitting at least a portion of the data associated with the data collection operation, as received by the coordinator computer and from the at least one worker computer, from the coordinator computer to the client computer.

2. The computer-implemented method of claim 1, wherein receiving,

by the coordinator computer and from the client computer that is remote from the coordinator computer, the request to initiate execution of the data collection operation, comprises receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, a user-initiated request to initiate execution of the data collection operation.

3. The computer-implemented method of claim 1, comprising:

after establishing the feedback communication channel by transmitting the flow control message from the coordinator computer to the client computer, and before transmitting data associated with the data collection operation from the coordinator computer to the client computer;

generating, based at least on the request to initiate execution of the data collection operation, a plurality of different queries at the coordinator computer,

transmitting each of the plurality of different queries from the coordinator computer to each of a plurality of worker computers, and

receiving, by the coordinator computer and from each of the plurality of worker computers, data associated with the data collection operation,

wherein transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer comprises transmitting a portion of the data associated with the data collection operation, as received by the coordinator and from the plurality of worker computers, from the coordinator computer to the client computer.

4. The computer-implemented method of claim 3, comprising:

after transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer, receiving, from the client computer, another release message that indicates that the client computer is ready to receive data associated with the data collection operation;

determining that all of the portions of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of worker computers, have already been transmitted from the coordinator computer to the client computer; and

in response to determining that all of the portions of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of worker computers, have already been transmitted from the coordinator computer to the client computer, transmitting, from the coordinator computer to the client computer, a message that indicates that the client computer has already transmitted all of the data associated with the data collection operation that the coordinator computer has received from the plurality of worker computers.

5. The computer-implemented method of claim 3, wherein:

receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, the request to initiate execution of the data collection operation, comprises receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, a request to initiate execution of the data collection operation that includes instructions associated with a plurality of keywords; and

generating, based at least on the request to initiate execution of the data collection operation, the plurality of different queries at the coordinator computer, comprises generating, at the coordinator computer and based at least on the instructions associated with the plurality of keywords, a plurality of different queries that each include instructions associated with a different set of the plurality of keywords, respectively.

6. The computer-implemented method of claim 3, comprising:

after transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer, receiving, from the client computer, another release message that indicates that the client computer is ready to receive data associated with the data collection operation; and

in response to receiving the other release message from the client computer, transmitting, from the coordinator computer to the client computer, another, different portion of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of work computers.

7. A system comprising:

one or more data processing apparatus; and

a computer-readable storage device having stored thereon instructions that, when executed by the one or more data processing apparatus, cause the one or more data processing apparatus to perform operations comprising;

receiving, by a coordinator computer and from a client computer that is remote from the coordinator computer, a request to initiate execution of a data collection operation;

in response to receiving the request to initiate execution of the data collection operation, establishing a feedback communication channel by transmitting a flow control message from the coordinator computer to the client computer;

after transmitting the flow control message from the coordinator computer to the client computer, receiving, from the client computer, a release message that indicates that the client computer is ready to receive data associated with the data collection operation;

in response to receiving the release message from the client computer, relaying, by the coordinator computer to at least one worker computer, the release message that indicates that the client computer is ready to receive data associated with the data collection operation, as received by the coordinator computer and from the client computer;

after relaying the release message that indicates that the client computer is ready to receive data associated with the data collection operation to the at least one worker computer, receiving data associated with the data collection operation from the at least one worker computer; and

transmitting at least a portion of the data associated with the data collection operation, as received by the coordinator computer and from the at least one worker computer, from the coordinator computer to the client computer.

8. The system of claim 7, wherein receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, the request to initiate execution of the data collection operation, comprises receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, a user-initiated request to initiate execution of the data collection operation.

9. The system of claim 7, wherein the operations further comprise:

after establishing the feedback communication channel by transmitting the flow control message from the coordinator computer to the client computer, and before transmitting data associated with the data collection operation from the coordinator computer to the client computer;

generating, based at least on the request to initiate execution of the data collection operation, a plurality of different queries at the coordinator computer,

transmitting each of the plurality of different queries from the coordinator computer to each of a plurality of worker computers, and

receiving, by the coordinator computer and from each of the plurality of worker computers, data associated with the data collection operation,

wherein transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer comprises transmitting a portion of the data associated with the data collection operation, as received by the coordinator and from the plurality of worker computers, from the coordinator computer to the client computer.

10. The system of claim 9, wherein the operations further comprise:

after transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer, receiving, from the client computer, another release message that indicates that the client computer is ready to receive data associated with the data collection operation; and

in response to determining that all of the portions of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of worker computers, have already been transmitted from the coordinator computer to the client computer, transmitting, from the coordinator computer to the client computer, a message that indicates that the client computer has already transmitted all of the data associated with the data collection operation that the coordinator computer has received from the plurality of worker computers.

11. The system of claim 9, wherein:

receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, the request to initiate execution of the data collection operation, comprises receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, a request to initiate execution of the data collection operation that includes instructions associated with a plurality of keywords; and

generating, based at least on the request to initiate execution of the data collection operation, the plurality of different queries at the coordinator computer, comprises generating, at the coordinator computer and based at least on the instructions associated with the plurality of keywords, a plurality of different queries that each include instructions associated with a different set of the plurality of keywords, respectively.

12. The system of claim 9, wherein the operations further comprise:

after transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer, receiving, from the client computer, another release message that indicates that the client computer is ready to receive data associated with the data collection operation; and

in response to receiving the other release message from the client computer, transmitting, from the coordinator computer to the client computer, another, different portion of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of work computers.

13. A non-transitory computer-readable storage device having instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations comprising:

receiving, by a coordinator computer and from a client computer that is remote from the coordinator computer, a request to initiate execution of a data collection operation;

in response to receiving the request to initiate execution of the data collection operation, establishing a feedback communication channel by transmitting a flow control message from the coordinator computer to the client computer;

after transmitting the flow control message from the coordinator computer to the client computer, receiving, from the client computer, a release message that indicates that the client computer is ready to receive data associated with the data collection operation;

in response to receiving the release message from the client computer, relaying, by the coordinator computer to at least one worker computer, the release message that indicates that the client computer is ready to receive data associated with the data collection operation, as received by the coordinator computer and from the client computer;

after relaying the release message that indicates that the client computer is ready to receive data associated with the data collection operation to the at least one worker computer, receiving data associated with the data collection operation from the at least one worker computer; and

transmitting at least a portion of the data associated with the data collection operation, as received by the coordinator computer and from the at least one worker computer, from the coordinator computer to the client computer.

14. The storage device of claim 13, wherein the operations further comprise:

after establishing the feedback communication channel by transmitting the flow control message from the coordinator computer to the client computer, and before transmitting data associated with the data collection operation from the coordinator computer to the client computer;

generating, based at least on the request to initiate execution of the data collection operation, a plurality of different queries at the coordinator computer,

transmitting each of the plurality of different queries from the coordinator computer to each of a plurality of worker computers, and

receiving, by the coordinator computer and from each of the plurality of worker computers, data associated with the data collection operation,

wherein transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer comprises transmitting a portion of the data associated with the data collection operation, as received by the coordinator and from the plurality of worker computers, from the coordinator computer to the client computer.

15. The storage device of claim 14, wherein the operations further comprise:

after transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer, receiving, from the client computer, another release message that indicates that the client computer is ready to receive data associated with the data collection operation; and

in response to determining that all of the portions of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of worker computers, have already been transmitted from the coordinator computer to the client computer, transmitting, from the coordinator computer to the client computer, a message that indicates that the client computer has already transmitted all of the data associated with the data collection operation that the coordinator computer has received from the plurality of worker computers.

16. The storage device of claim 14, wherein:

receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, the request to initiate execution of the data collection operation, comprises receiving, by the coordinator computer and from the client computer that is remote from the coordinator computer, a request to initiate execution of the data collection operation that includes instructions associated with a plurality of keywords; and

generating, based at least on the request to initiate execution of the data collection operation, the plurality of different queries at the coordinator computer, comprises generating, at the coordinator computer and based at least on the instructions associated with the plurality of keywords, a plurality of different queries that each include instructions associated with a different set of the plurality of keywords, respectively.

17. The storage device of claim 14, wherein the operations further comprise:

after transmitting at least a portion of the data associated with the data collection operation from the coordinator computer to the client computer, receiving, from the client computer, another release message that indicates that the client computer is ready to receive data associated with the data collection operation; and

in response to receiving the other release message from the client computer, transmitting, from the coordinator computer to the client computer, another, different portion of the data associated with the data collection operation, as received by the coordinator computer and from the plurality of work computers.