Computer architecture and process for implementing a virtual vertical perimeter framework for an overloaded CPU having multiple network interfaces

• US 20060047856A1
• Filed: 07/06/2004
• Published: 03/02/2006
• Est. Priority Date: 07/06/2004
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Tripathi, Sunay
• Granted Patent: US 8,032,658 B2
• Time in Patent Office: Days
• US Class Current: 709/250
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers

Abstract

Techniques and modes of operation for using network interfaces in a single or multi-CPU environment are disclosed. The techniques provide a virtual “vertical perimeter” framework suitable for processing data through multiple network interfaces assigned to a single CPU. In this framework, techniques for installing and servicing a plurality of network interfaces with a single CPU utilizing virtual vertical perimeters and sub-queues is disclosed. A computer system including at least one CPU and a plurality of network interfaces configured to have more network interfaces than CPU'"'"'s is disclosed. At least two of the interfaces are assigned to an associated CPU. The computer system includes an operating system with plurality of sub-queues and an associated sub-queue controller. The sub-queues are associated with at least two network interfaces assigned to the CPU. The sub-queue controller is configured for coordinating the generation and operation of the sub-queues thereby to facilitating message traffic through the at least two network interfaces.

Claims

1. A method of servicing message packets in a computer network architecture, the method comprising:

providing a computer system architecture having;

at least one central processing unit (CPU) and a plurality of network interfaces,

wherein the number of network interfaces is greater than the number of CPU'"'"'s and wherein the system includes at least one overloaded CPU having more than one network interface assigned to it, and

wherein each overloaded CPU has associated therewith a sub-queue controller and an associated sub-queue for each network interface assigned to said overloaded CPU, and

wherein each sub-queue is capable of storing message packets received from the network interface assigned to said sub-queue;

determining, for each overloaded CPU, which of the sub-queues assigned to said CPU shall be selected for message processing;

retrieving a message packet from the selected sub-queue; and

processing the message packet.

2. The method of claim 1 wherein said processing of the message packet retrieved from the sub-queue comprises conducting all of the processing with the overloaded CPU.

3. The method of claim 1 wherein said determining which of the sub-queues shall be selected for message processing includes selecting from among the sub-queues using a time based priority system.

4. The method of claim 1 wherein said determining which of the sub-queues shall be selected for message processing includes selecting from among the sub-queues using a packet based priority system.

5. The method of claim 1 wherein said determining which of the sub-queues shall be selected for message processing includes selecting from among the sub-queues using a priority system based on the relative data transfer capacities of the network interfaces assigned to said overloaded CPU.

6. The method of claim 1 wherein said determining which of the sub-queues shall be selected for message processing includes selecting from among the sub-queues using a priority system that can be adjusted at will by a user.

7. The method of claim 1 further including:

receiving message packets in the sub-queues, wherein each sub-queue receives message packets only from the network interface assigned to said sub-queue; and

controlling the rate at which message packets are received by each sub-queue from the its assigned network interface.

8. The method of claim 7 wherein controlling the rate at which message packets are received by each sub-queue is achieved using the occupancy rate of each sub-queue.

9. The method of claim 8 wherein controlling the rate at which message packets are received by each sub-queue is achieved for a sub-queue that exceeds a specified occupancy threshold by using a polling technique to solicit message packets from the assigned network interface.

10. The method of claim 8 wherein controlling the rate at which message packets are received by each sub-queue is achieved for a sub-queue that is below specified occupancy threshold by using an interrupt technique to receive message packets from the assigned network interface.

11. A computer program product embodied in a computer readable media including computer program code for servicing message packets in a computer network architecture, the computer readable media comprising:

computer program code for providing a computer system architecture having at least one central processing unit (CPU) and a plurality of network interfaces wherein the number of network interfaces is greater than the number of CPU'"'"'s and wherein the system includes at least one overloaded CPU having more than one network interface assigned to it and wherein each overloaded CPU has associated therewith a sub-queue controller and an associated sub-queue for each network interface assigned to said overloaded CPU and wherein each sub-queue is capable of storing message packets received from the network interface assigned to said sub-queue;

computer program code for determining, for each overloaded CPU, which of the sub-queues assigned to said CPU shall be selected for message processing;

computer program code for retrieving a message packet from the selected sub-queue; and

computer program code for processing the message packet.

12. The computer program product of claim 11 wherein said computer program code for processing of the message packet retrieved from the sub-queue comprises computer program code for conducting all of the processing with the overloaded CPU.

13. The computer program product of claim 11 wherein said computer program code for determining which of the sub-queues shall be selected for message processing includes computer program code for selecting from among the sub-queues using one a time based priority system and a packet based priority system.

14. The computer program product of claim 11 wherein said computer program code for determining which of the sub-queues shall be selected for message processing includes computer program code for selecting from among the sub-queues using a priority system based on the relative data transfer capacities of the network interfaces assigned to said overloaded CPU.

15. The computer program product of claim 11 wherein said computer program code for determining which of the sub-queues shall be selected for message processing includes computer program code for selecting from among the sub-queues using a priority system that can be adjusted at will by a user.

16. The computer program product of claim 11 further including:

computer program code for receiving message packets in a selected one of said sub-queues from the network interface assigned to said selected sub-queue; and

computer program code for controlling the rate at which message packets are received by each sub-queue from the its assigned network interface.

17. The computer program product of claim 16 wherein the computer program code for controlling the rate at which message packets are received by each sub-queue controlling the rate using the occupancy rate of each sub-queue.

18. The computer program product of claim 17 wherein the computer program code for controlling the rate at which message packets are received by each sub-queue is achieved for a sub-queue that exceeds a specified occupancy threshold by using a polling technique to solicit message packets from the assigned network interface.

19. The computer program product of claim 17 wherein the computer program code for controlling the rate at which message packets are received by each sub-queue is achieved for a sub-queue that is below specified occupancy threshold by using an interrupt technique to receive message packets from the assigned network interface.

20. A method of installing a network interface in a computer network architecture, the method comprising:

providing a computer system architecture having at least one central processing unit (CPU) and a plurality of existing network interfaces wherein the number of network interfaces is at least as great as the number of CPU'"'"'s and wherein each network interface is assigned to a designated CPU using a queue;

introducing a new network interface to the computer system architecture;

determining which CPU that the new network interface shall be assigned thereby generating an overloaded CPU;

generating a plurality of sub-queues sufficient to accommodate network interfaces of the overloaded CPU;

assigning the plurality of sub-queues to the overloaded CPU; and

generating a sub-queue controller for coordinating the operation of the sub-queues of the overloaded CPU.

21. The method of claim 20 wherein said introducing a new network interface to the computer system architecture comprises plugging a network interface card into the computer system architecture.

22. The method of claim 20 wherein said providing a computer system architecture having each network interface assigned to a designated CPU using a queue includes providing network interfaces that are assigned to the designated CPU using a queue that comprises one of a serialization queue and a sub-queue.

23. The method of claim 22 wherein said sub-queue comprises a serialization sub-queue.

24. The method of claim 20 wherein determining which CPU that the new network interface shall be assigned comprises evaluating the queues for each CPU and assigning the new network interface to the CPU having the least busy queue.

25. The method of claim 20 wherein determining which CPU that the new network interface shall be assigned comprises

evaluating the queues for each CPU;

determining that at least two queues are empty;

analyzing a message packet processing history for each queue;

determining which queue has been the least busy;

determining which CPU is associated with said least busy queue and designating that CPU as available; and

assigning the new network interface to the available CPU.

26. The method of claim 20 wherein generating a controller for coordinating the operation of the sub-queues of the overloaded CPU comprises:

generating a controller structure for coordinating the operation of sub-queues associated with the overloaded CPU;

generating sub-queues for the overloaded CPU; and

assigning the sub-queues to the overloaded CPU.

27. The method of claim 26 wherein generating sub-queues for the overloaded CPU includes generating a new sub-queue for servicing the new network interface and converting an existing queue that previously serviced an existing network interface to a sub-queue to continue servicing the existing network interface.

28. The method of claim 26 wherein generating a controller structure for coordinating the operation of sub-queues associated with the overloaded CPU includes:

providing an indicator for each sub-queue, wherein each indicator identifies a memory location for each sub-queue assigned to each of the network interfaces associated the overloaded CPU; and

providing an indicator for an allocation engine, wherein the indicator identifies a memory location for the allocation engine which regulates the flow of message packets from the sub-queues to the overloaded CPU to facilitate message traffic between the network interfaces and the overloaded CPU.

29. A computer program product embodied in a computer readable media including computer program code for installing a network interface in a computer network architecture, computer readable media comprising:

computer program code for providing a computer system architecture having at least one central processing unit (CPU) and a plurality of existing network interfaces wherein the number of network interfaces is at least as great as the number of CPU'"'"'s and wherein each network interface is assigned to a designated CPU using a queue;

computer program code for introducing a new network interface to the computer system architecture;

computer program code for determining which CPU that the new network interface shall be assigned thereby generating an overloaded CPU;

computer program code for generating a plurality of sub-queues sufficient to accommodate network interfaces of the overloaded CPU;

computer program code for assigning the plurality of sub-queues to the overloaded CPU; and

computer program code for generating a sub-queue controller for coordinating the operation of the sub-queues of the overloaded CPU.

30. The computer program product of claim 29 wherein the computer program code for determining which CPU that the new network interface shall be assigned comprises computer program code for reviewing the queues for each CPU and computer program code for assigning the new network interface to the CPU having the least busy queue.

31. The computer program product of claim 29 wherein the program code for determining which CPU that the new network interface shall be assigned comprises

program code for reviewing the queues for each CPU;

program code for determining that at least two queues are empty;

program code for analyzing a message packet processing history for each queue;

program code for determining which queue has been the least busy;

program code for determining which CPU is assigned to said least busy queue and designating that CPU as available; and

program code for assigning the new network interface to the available CPU.

32. The computer program product of claim 29 wherein the program code for generating a controller that coordinates the operation of the sub-queues of the overloaded CPU comprises:

program code for generating a controller structure for coordinating the operation of sub-queues associated with the overloaded CPU;

program code for generating sub-queues for the overloaded CPU; and

program code for assigning the sub-queues to the overloaded CPU.

33. The computer program product of claim 32 wherein the program code for generating sub-queues for the overloaded CPU includes program code for generating a new sub-queue for servicing the new network interface and converting an existing queue that previously serviced an existing network interface to a sub-queue to continue servicing the existing network interface.

34. The computer program product of claim 32 wherein the program code for generating a controller structure for coordinating the operation of sub-queues associated with the overloaded CPU includes:

program code for providing an indicator for each sub-queue, wherein each indicator identifies a memory location for each sub-queue assigned to each of the network interfaces associated the overloaded CPU; and

program code for providing an indicator for an allocation engine, wherein the indicator identifies a memory location for the allocation engine which regulates the flow of message packets from the sub-queues to the overloaded CPU to facilitate message traffic between the network interfaces and the overloaded CPU.

35. A computer system comprising:

at least one CPU, including a first CPU;

a plurality of network interfaces wherein the number of network interfaces is greater than the number of CPU'"'"'s, said interfaces being configured such that at least two of the network interfaces are assigned to the first CPU;

an operating system that includes;

a plurality of sub-queues wherein each sub-queue is associated with one of the network interfaces that are assigned to the first CPU;

a sub-queue controller for coordinating the generating and operating of the sub-queues to facilitate message traffic through the at least two network interfaces.

36. The computer system of claim 35 wherein the network interfaces are configured to include network interface cards (NIC).

37. The computer system of claim 35 wherein the controller allocates when message packets in each sub-queue are processed by the first CPU.

38. The computer system of claim 37 wherein the controller determines a priority schedule for allocating when message packets in each sub-queue are processed by the first CPU.

39. The computer system of claim 38 wherein the priority schedule for allocating when the message packets in each sub-queue are to be processed by the CPU comprises a time-based allocation.

40. The computer system of claim 38 wherein the priority schedule for allocating when the message packets in each sub-queue are to be processed by the CPU comprises a packet-based allocation dependent on the number of message packets processed for each sub-queue.

41. The computer system of claim 38 wherein the priority schedule for allocating when the message packets in each sub-queue are to be processed by the CPU is based upon the relative message processing capacities of the network interfaces associated with each sub-queue.

42. The computer system of claim 38 wherein the priority schedule for allocating when the message packets in each sub-queue are to be processed by the CPU can be adjusted by a user.

43. The computer system of claim 37 wherein the determining of the priority schedule for allocating when message packets in each sub-queue are processed by the first CPU is determined by a scheduler.

44. The computer system of claim 35 wherein the controller operates to set-up new sub-queues when an additional network interface is added to the computer system.

45. The computer system of claim 44 wherein the controller enables the dynamic installation of the additional network interface and the dynamic set-up of new sub-queues to be assigned to the additional network interface while the computer system remains in operation.

46. The computer system of claim 44 wherein the set-up of the new sub-queues when the additional network interface is added to the computer system is performed by a sub-queue set-up engine.

47. The computer system of claim 46 wherein the set-up engine is included as a portion of the controller.

48. The computer system of claim 35 wherein the controller is configured to control the rate at which data is received by a sub-queue from its associated network interface.

49. The computer system of claim 48 wherein the controller is configured to control the rate at which data is received by a sub-queue by using the occupancy rate of each sub-queue.

50. The computer system of claim 49 wherein the controller controls the rate at which message packets are received by each sub-queue by alternatively using polling and interrupt techniques to facilitate the transmission of message packets from the assigned network interface.

51. The computer system of claim 37 wherein the controller is configured to control the rate at which data is received by a sub-queue by using the occupancy rate of each sub-queue.

52. The computer system of claim 51 wherein the controller controls the rate at which message packets are received by each sub-queue by alternatively using polling and interrupt techniques to facilitate the transmission of message packets from the assigned network interface.

53. A sub-queue controller module for implementation in an operating system of a computer system having at least one CPU and a plurality of network interfaces wherein the number of network interfaces is greater than the number of CPU(s), the controller being configured for coordinating the generation and operation of sub-queues used to facilitate message traffic between a first CPU of the at least one CPU and an associated set of network interfaces associated with the first CPU, wherein the set of network interfaces is selected from the plurality of network interfaces, the controller comprising:

data for identifying a plurality of sub-queues operating in the operating system wherein each identified sub-queue is associated with one of the set of network interfaces associated with the first CPU;

an indicator for each sub-queue, wherein each indicator identifies a memory location for each sub-queue assigned to each of the network interfaces associated the first CPU; and

an allocation engine for regulating the operation of said sub-queues to facilitate message traffic between the set of network interfaces and the first CPU.

54. The controller of claim 53 wherein the controller can put a sub-queue into a polling mode of operation when the sub-queue reaches a data occupancy threshold.

55. The controller of claim 53 wherein the controller enables a sub-queue to operate in one of a polling mode of operation and an interrupt mode of operation.

56. The controller of claim 53 further including a means for recognizing when new network interfaces are added to the system and setting-up new sub-queues to service the new network interfaces

57. The controller of claim 56 wherein said means includes a sub-queue set-up engine configured to recognize when new network interfaces are added to the system and set-up new sub-queues to service the new network interfaces.

58. The controller of claim 56 wherein said means includes a sub-queue set-up engine configured to enable dynamic installation of the new network interface and dynamic set-up of new sub-queues associated with the new interface while the computer system remains in operation.

59. The controller of claim 56 wherein said means includes a sub-queue set-up engine that is stored in the operating system outside the controller and wherein the means includes an indicator that identifies the location of the set-up engine such that the controller can access the functionality of the set-up engine.

60. The controller of claim 53 wherein the allocation engine is configured to allocate when message packets in each sub-queue are processed by the first CPU.

61. The controller of claim 53 wherein the allocation engine is configured to process the message packets in each sub-queue in accordance with one of time-based allocation or packet-based allocation.

62. The controller of claim 53 wherein the allocation engine is configured to process the message packets in each sub-queue in accordance with the relative message processing capacities of the network interfaces associated with each sub-queue.

63. The controller of claim 53 wherein the allocation engine includes an indicator that points to a scheduler in the operating system, where the scheduler is configured to allocate when message packets in each sub-queue are processed by the first CPU.