Method and apparatus for distributing network traffic processing on a multiprocessor computer

• US 8,447,803 B2
• Filed: 05/14/2003
• Issued: 05/21/2013
• Est. Priority Date: 10/14/1997
• Status: Expired due to Fees

• Current Assignee: Alacritech, Inc.
• Original Assignee: Alacritech, Inc.
• Inventors: Boucher, Laurence B., Blightman, Stephen E. J., Craft, Peter K., Higgen, David A., Philbrick, Clive M., Starr, Daryl D.
• Primary Examiner(s): Nguyen, Tammy
• Application Number:
  US10/438,719
• Publication Number:
  US 20040030745A1
• Time in Patent Office: 3,660 Days
• Field of Search: 709/203, 709/235, 709/221, 455/564, 370/528, 712/28, 712/11
• US Class Current: 709/203
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4221 being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus
• CPC Class Codes: H04L 45/66 Layer 2 routing, e.g. in Ethernet based MAN's
• CPC Class Codes: H04L 45/745 Address table lookup; Address filtering
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/193 at the transport layer, e.g. TCP related
• CPC Class Codes: H04L 47/36 by determining packet size, e.g. maximum transfer unit [MTU]
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 47/6225 Fixed service order, e.g. Round Robin
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 49/901 using storage descriptor, e.g. read or write pointers
• CPC Class Codes: H04L 49/9021 Plurality of buffers per packet
• CPC Class Codes: H04L 49/9042 Separate storage for different parts of the packet, e.g. header and payload
• CPC Class Codes: H04L 49/9052 with buffers of different sizes
• CPC Class Codes: H04L 49/9063 Intermediate storage in different physical parts of a node or terminal
• CPC Class Codes: H04L 49/9068 in the network interface card
• CPC Class Codes: H04L 49/9073 Early interruption upon arrival of a fraction of a packet
• CPC Class Codes: H04L 49/9094 Arrangements for simultaneous transmit and receive, e.g. simultaneous reading/writing from/to the storage element
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 61/10 : of different types
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 67/62 : Establishing a time schedule for servicing the requests
• CPC Class Codes: H04L 67/63 : Routing a service request depending on the request content or context
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: H04L 69/12 : Protocol engines
• CPC Class Codes: H04L 69/16 : Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/161 : Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
• CPC Class Codes: H04L 69/162 : involving adaptations of sockets based mechanisms secure socket layer H04L63/168
• CPC Class Codes: H04L 69/163 : In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: H04L 69/166 : IP fragmentation; TCP segmentation
• CPC Class Codes: H04L 69/22 : Parsing or analysis of headers
• CPC Class Codes: H04L 69/32 : Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 9/40 : Network security protocols

Abstract

An intelligent network interface card (INIC) or communication processing device (CPD) works with a host computer for data communication. The device provides a fast-path that avoids protocol processing for most messages, greatly accelerating data transfer and offloading time-intensive processing tasks from the host CPU. The host retains a fallback processing capability for messages that do not fit fast-path criteria, with the device providing assistance such as validation even for slow-path messages, and messages being selected for either fast-path or slow-path processing. A context for a connection is defined that allows the device to move data, free of headers, directly to or from a destination or source in the host. The context can be passed back to the host for message processing by the host. The device contains specialized hardware circuits that are much faster at their specific tasks than a general purpose CPU. A preferred embodiment includes a trio of pipelined processors devoted to transmit, receive and utility processing, providing full duplex communication for four Fast Ethernet nodes.

Claims

1. A method of distributing network traffic processing among a plurality of processors in a multiprocessor computer, comprising:

receiving a first packet at a network interface that is a part of a multiprocessor computer;

retrieving from said first packet an identifier of a source of said first packet;

retrieving from said first packet an identifier of a destination of said first packet, wherein said destination is disposed in said multiprocessor computer;

combining said source identifier and said destination identifier to form a first flow index for identifying a first communication flow comprising said first packet;

storing a control block holding the state of said first communication flow;

generating from said first flow index an identifier of a first processor of a plurality of processors of said multiprocessor computer;

associating said first processor identifier with said first packet;

submitting said first packet for processing by said first processor; and

transferring control of said control block from said first processor to a second processor of said plurality of processors.

2. The method of claim 1, further comprising:

receiving a second packet of said first communication flow at said network interface;

associating said first processor identifier with said second packet; and

submitting said second packet from said network interface for processing by said first processor.

3. The method of claim 1, further comprising:

receiving a second packet at said network interface;

producing a second flow index to identify a second communication flow comprising said second packet;

generating from said second flow index an identifier of the second processor;

associating said second processor identifier with said second packet; and

submitting said second packet for processing by said second processor.

4. The method of claim 3, wherein said producing a second flow index comprises retrieving a virtual connection identifier from said second packet.

5. The method of claim 1, wherein said source identifier and said destination identifier belong to a first protocol for which processing is not offloaded to the network interface, and further comprising receiving a second packet having a second source identifier and a second destination identifier that belong to a second protocol for which processing is offloaded to said network interface.

6. The method of claim 1, wherein transferring control of said control block from said first processor to said second processor includes transferring control of a Transport Control Protocol (TCP) connection from said first processor to said second processor.

7. The method of claim 1, wherein said source identifier and said destination identifier correspond to a Transaction TCP (T/TCP) protocol.

8. The method of claim 1, wherein said submitting comprises:

storing said first packet in a first memory of said multiprocessor computer; and

storing said first processor identifier in a second memory.

9. The method of claim 8, wherein said submitting further comprises issuing an alert to said multiprocessor computer to notify said multiprocessor computer of said storage of said first packet.

10. The method of claim 1, further comprising searching a flow database for said first flow index, wherein said flow database comprises a collection of flow indexes.

11. The method of claim 10, wherein said searching comprises if said first flow index is not found in said flow database, storing said first flow index in said flow database.

12. The method of claim 10, wherein said searching comprises if said first flow index is found in said flow database, updating an entry in said flow database associated with said first flow index.

13. The method of claim 1, further comprising storing said first flow index in a flow database comprising flow indexes of communication flows involving entities operating on the multiprocessor computer.

14. A method of distributing the load of processing packets received from a network among multiple processors in a multiprocessor computer, comprising:

receiving a first packet at a network interface that is a part of a multiprocessor host computer;

receiving a second packet at said network interface;

retrieving a source identifier and a destination identifier from said first packet, wherein said destination identifier identifies a destination in said multiprocessor host computer;

combining said source identifier and said destination identifier to form a first flow key;

storing a control block holding the state of a first communication flow comprising said first packet;

creating a second flow key to identify a second communication flow comprising said second packet;

generating an identifier of a first processor from said first flow key;

generating an identifier of a second processor from said second flow key;

submitting said first packet to said first processor;

transferring control of said control block from said first processor to a second processor of said plurality of processors; and

submitting said second packet to said second processor.

15. The method of claim 14, wherein each of said receiving, creating and generating are performed on said network interface.

16. The method of claim 14, wherein said retrieving comprises extracting one or more fields from a protocol header of said first packet, said protocol being one of a set of pre-selected protocols.

17. The method of claim 14, wherein said creating a second flow key comprises extracting a virtual connection identifier from said second packet.

18. The method of claim 14, wherein transferring control of said control block from said first processor to said second processor includes transferring control of a Transport Control Protocol (TCP) connection from said first processor to said second processor.

19. The method of claim 14, wherein said first flow corresponds to a first protocol and said second flow corresponds to a second protocol.

20. A multiprocessor computer for receiving and processing packets from a network, comprising:

a plurality of processors; and

a network interface configured to receive packets from a network and submit said packets to said plurality of processors, said network interface comprising;

a flow identifier configured to;

retrieve a source identifier and a destination identifier, wherein said destination identifier identifies a destination in said multiprocessor computer from a packet; and

combine said source identifier and said destination identifier to form a flow key for identifying a communication flow comprising said packet; and

a load distributor configured to generate, from said flow key, an identifier of a processor of said plurality of said processors for processing said packet; and

a memory adapted to store a control block holding the state of said communication flow;

wherein said load distributor generates a first identifier of a first processor for a first set of packets in said communication flow and generates a second identifier of a second processor for a second set of packets in a second communication flow.

21. The multiprocessor computer of claim 20, said network interface further comprising:

a flow database for storing one or more flow keys, and

a flow database manager for managing said flow database.

22. The multiprocessor computer of claim 20, said network interface further comprising:

a first data structure for storing said first set of packets; and

a second data structure for storing said first processor identifier.

23. A non-transitory computer readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method of distributing the processing of network traffic among multiple processors, the method comprising:

receiving a first packet at a network interface that is a part of a multiprocessor computer;

retrieving from said first packet an identifier of a source of said first packet;

retrieving from said first packet an identifier of a destination of said first packet, wherein said destination is disposed in said multiprocessor computer;

combining said source identifier and said destination identifier to form a first flow index for identifying a first communication flow comprising said first packet;

storing a control block holding the state of said first communication flow;

generating from said first flow index an identifier of a first processor of a plurality of processors of said multiprocessor computer;

associating said first processor identifier with said first packet; and

submitting said first packet for processing by said first processor; and

transferring control of said control block from said first processor to a second processor of said plurality of processors.

24. The method of claim 23, wherein said generating from said first flow index an identifier of said first processor is performed by said network interface.

25. The method of claim 23, further comprising establishing a transport-level network connection corresponding to said first communication flow.

26. The method of claim 25, wherein transferring control of said control block from said first processor to said second processor includes transferring control of said transport-level network connection from said first processor to said second processor.

27. The method of claim 26, further comprising transferring control of said transport-level network connection from said second processor to said first processor.

28. The method of claim 23, wherein transferring control of said control block from said first processor to said second processor includes transferring control of said first communication flow from said first processor to said second processor.

29. The method of claim 28, further comprising transferring control of said first communication flow from said second processor to said first processor.

30. The method of claim 23, further comprising processing a header of said first packet to obtain a buffer address for storing application data from a second packet.

31. The method of claim 23, further comprising processing a header of said first packet to obtain a buffer address for storing application data from a second packet.

32. The method of claim 31, further comprising processing said second packet by said second processor.

33. The method of claim 23, wherein said retrieving from said first packet said identifier of said source of said first packet and said retrieving from said first packet said identifier of said destination of said first packet includes parsing network layer and transport layer headers of said first packet by said network interface.

34. A method of distributing network traffic processing among a plurality of processors in a multiprocessor computer, comprising:

receiving a first packet at a network interface that is a part of a multiprocessor computer;

producing a first flow index to identify a first communication flow comprising said first packet;

storing a control block holding the state of said first communication flow;

generating from said first flow index an identifier of a first processor of a plurality of processors of said multiprocessor computer;

associating said first processor identifier with said first packet;

storing said first packet in a first memory of said multiprocessor computer;

storing said first processor identifier in a second memory;

issuing an alert to said multiprocessor computer to notify said multiprocessor computer of said storage of said first packet; and

transferring control of said control block from said first processor to a second processor of said plurality of processors.

35. The method of claim 34, wherein said producing a first flow index comprises:

retrieving from said first packet an identifier of a source of said first packet; and

retrieving from said first packet an identifier of a destination of said first packet.

36. The method of claim 35, wherein said producing a first flow index further comprises combining said source identifier and said destination identifier.

37. The method of claim 34, wherein said producing a first flow index comprises retrieving a virtual connection identifier from said first packet.

38. The method of claim 34, further comprising searching a flow database for said first flow index, wherein said flow database comprises a collection of flow indexes.

39. The method of claim 38, wherein said searching comprises:

if said first flow index is not found in said flow database, storing said first flow index in said flow database; and

if said first flow index is found in said flow database, updating an entry in said flow database associated with said first flow index.

40. A non-transitory computer readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method of distributing network traffic processing among a plurality of processors in a multiprocessor computer, the method comprising:

receiving a first packet at a network interface that is a part of a multiprocessor computer;

producing a first flow index to identify a first communication flow comprising said first packet;

storing a control block holding the state of said first communication flow;

generating from said first flow index an identifier of a first processor of a plurality of processors of said multiprocessor computer;

associating said first processor identifier with said first packet;

storing said first packet in a first memory of said multiprocessor computer;

storing said first processor identifier in a second memory;

issuing an alert to said multiprocessor computer to notify said multiprocessor computer of said storage of said first packet; and

transferring control of said control block from said first processor to a second processor of said plurality of processors.

41. A method of distributing network traffic processing among a plurality of processors in a multiprocessor computer, comprising:

receiving a first packet at a network interface that is a part of a multiprocessor computer;

producing a first flow index to identify a first communication flow comprising said first packet;

storing a control block holding the state of said first communication flow;

generating from said first flow index an identifier of a first processor of a plurality of processors of said multiprocessor computer;

associating said first processor identifier with said first packet;

submitting said first packet for processing by said first processor;

searching a flow database for said first flow index, said flow database comprising a collection of flow indexes;

if said first flow index is not found in said flow database, storing said first flow index in said flow database; and

transferring control of said control block from said first processor to a second processor of said plurality of processors.

42. The method of claim 41, further comprising:

if said first flow index is found in said flow database, updating an entry in said flow database associated with said first flow index.

43. The method of claim 41, wherein said producing a first flow index comprises:

retrieving from said first packet an identifier of a source of said first packet; and

retrieving from said first packet an identifier of a destination of said first packet.

44. The method of claim 43, wherein said producing a first flow index further comprises combining said source identifier and said destination identifier.

45. The method of claim 41, wherein said producing a first flow index comprises retrieving a virtual connection identifier from said first packet.

46. The method of claim 41, further comprising processing a header of said first packet to obtain a buffer address for storing application data from a second packet.

47. The method of claim 46, further comprising processing said second packet by said second processor.

48. A non-transitory computer readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method of distributing network traffic processing among a plurality of processors in a multiprocessor computer, the method comprising:

receiving a first packet at a network interface that is a part of a multiprocessor computer;

producing a first flow index to identify a first communication flow comprising said first packet;

storing a control block holding the state of said first communication flow;

generating from said first flow index an identifier of a first processor of a plurality of processors of said multiprocessor computer;

associating said first processor identifier with said first packet;

submitting said first packet for processing by said first processor;

searching a flow database for said first flow index, said flow database comprising a collection of flow indexes;

if said first flow index is not found in said flow database, storing said first flow index in said flow database; and

transferring control of said control block from said first processor to a second processor of said plurality of processors.