TCP/IP processor and engine using RDMA

• US 20040037319A1
• Filed: 06/10/2003
• Published: 02/26/2004
• Est. Priority Date: 06/11/2002
• Status: Active Grant

• Current Assignee: Memory Access Technologies LLC
• Original Assignee: Ashish A. Pandya
• Inventors: Pandya, Ashish A.
• Granted Patent: US 7,376,755 B2
• Time in Patent Office: Days
• US Class Current: 370/469
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 69/10 Streamlined, light-weight or high-speed protocols, e.g. express transfer protocol [XTP] or byte stream
• CPC Class Codes: H04L 69/12 Protocol engines
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/161 Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: H04L 69/165 Combined use of TCP and UDP protocols; selection criteria therefor
• CPC Class Codes: H04L 69/166 IP fragmentation; TCP segmentation
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 69/323 in the physical layer [OSI layer 1]
• CPC Class Codes: H04L 69/326 in the transport layer [OSI layer 4] H04L69/16 takes precedence
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A TCP/IP processor and data processing engines for use in the TCP/IP processor is disclosed. The TCP/IP processor can transport data payloads of Internet Protocol (IP) data packets using an architecture that provides capabilities to transport and process Internet Protocol (IP) packets from Layer 2 through transport protocol layer and may also provide packet inspection through Layer 7. The engines may perform pass-through packet classification, policy processing and/or security processing enabling packet streaming through the architecture at nearly the full line rate. A scheduler schedules packets to packet processors for processing. An internal memory or local session database cache stores a TCP/IP session information database and may also store a storage information session database for a certain number of active sessions. The session information that is not in the internal memory is stored and retrieved to/from an additional memory. An application running on an initiator or target can in certain instantiations register a region of memory, which is made available to its peer(s) for access directly without substantial host intervention through RDMA data transfer.

Claims

1. A TCP/IP stack providing transport layer remote direct memory access capability for use in transporting information in active sessions or connections over IP networks, said stack providing an interface to the upper layer protocol functions in a host processor to carry data traffic, and performing at least one of the hardware implemented functions of:

a. sending and receiving data, including upper layer data;

b. establishing transport sessions and session teardown functions;

c. executing error handling functions;

d. executing time-outs;

e. executing retransmissions;

f. executing segmenting and sequencing operations;

g. maintaining protocol information regarding said active transport sessions;

h. maintaining TCP/IP state information for each of said one or more session connections.

i. fragmenting and defragmenting data packets;

j. routing and forwarding data and control information;

k. sending to and receiving from a peer, memory regions reserved for RDMA;

l. recording said memory regions reserved for RDMA in an RDMA database and maintaining said database;

m. executing operations provided by RDMA capability;

n. executing security management functions;

o. executing policy management and enforcement functions;

p. executing virtualization functions;

q. communicating errors;

r. processing Layer 2 media access functions to receive and transmit data packets, validate the packets, handle errors, communicate errors and other Layer 2 functions;

s. processing physical layer interface functions;

t. executing TCP/IP checksum generation and verification functions;

u. processing Out of Order packet handling;

v. CRC calculation functions;

w. processing Direct Data Placement/Transfer;

x. Upper Layer Framing functions;

y. processing functions and interface to socket API'"'"'s;

z. forming packet headers for TCP/IP for transmitted data and extraction of payload from received packets; and

aa. processing header formation and payload extraction for Layer 2 protocols of data to be transmitted and received data packets;

respectively.

2. The TCP/IP stack of claim 1 further comprising memory for storing a database to maintain various information regarding said active sessions or connections and TCP/IP state information for each of the sessions or connections.

3. The TCP/IP stack of claim 1 further comprising an interface that includes circuitry for interfacing to at least one layer that is a wired or wireless Ethernet, MII, GMII, XGMII, XPF, XAUI, TBI, SON ET, DSL, POS, POS-PHY,SPI Interface, SPI-4 or other SPI derivative Interface, Infiniband, or FC layer.

4. A TCP/IP processor engine having RDMA capability, said processor for processing Internet Protocol packets and comprising at least one of each of:

a. an RDMA mechanism for performing RDMA data transfer;

b. a checksum hardware for performing checksum operations;

c. a data memory for storing data used in the TCP/IP processor engine;

d. an instruction memory for storing instructions used in the TCP/IP processor;

e. an instruction fetch mechanism for fetching said instructions;

f. an instruction decoder for decoding said instructions;

g. an instruction sequencer for sequencing said instructions;

h. a session database memory for storing TCP/IP session data;

or

i. a session database memory controller for controlling said session database memory;

or a combination of any of the foregoing items a through i; and

a host interface, or a fabric interface, or bus controller, or memory controller or combination of any of the foregoing for coupling to host or a fabric.

5. The TCP/IP processor engine of claim 4 further comprising at least one of:

a. a hash engine for performing hash functions;

b. a sequencer manager for sequencing operations;

c. a window operations manager for performing windowing operations to position packets within, and/or verify packets to be within, agreed windows;

d. a classification tag interpreter for interpreting classification tags;

e. a frame controller for controlling data framing;

f. an out of order manager for handling out of order data;

g. a register file for storing data;

h. a TCP state manager for managing TCP session states;

i. a CRC component for performing CRC operations;

j. an execution resource unit or ALU for data processing;

k. a TCP session database lookup engine for accessing session entries;

l. an SNACK engine for selective negative acknowledgment;

m. an SACK engine for selective positive acknowledgment;

n. a segmentation controller for controlling the segmentation of data;

o. a timer for event timing;

p. a packet memory for storing packets; and

q. a combination of any of the foregoing.

6. The combination of claim 5 wherein said register file stores packet headers, pointers, contexts or session states or any combination thereof.

7. A TCP/IP processor implemented in hardware and capable of implementing transport level RDMA functions, said processor including a session memory for storing session information for a plurality of sessions.

8. A programmable TCP/IP processor engine, said processor having RDMA capability and used for processing Internet Protocol packets, said TCP/IP processor engine comprising:

a checksum component for TCP/IP checksum verification and for new checksum generation;

a

data memory for storing said packets;

an execution resource;

a packet look-up interface for assisting an execution resource and an instruction sequencer for providing access to said data packets or predetermined data packet fields thereof;

an instruction decoder to direct said TCP/IP processor engine operation based on the results of a classification processor;

a sequence and window operation manager providing specific segmenting, sequencing and windowing operations for use in TCP/IP data sequencing calculations;

and further comprising;

a hash engine used to perform hash operations against predetermined fields of the packet to perform a hash table walk to determine the correct session entry for said packet;

a register file for extracting predetermined header fields from said packets for TCP processing;

pointer registers for indicating data source and destination;

context register sets for holding multiple contexts for packet execution;

said multiple contexts allowing, in response to a given packet execution stalling, another context to be invoked to enable said TCP/IP processor engine to continue the execution of another packet stream;

said TCP/IP processor engine having a cache for holding recently or frequently used session entries, including connection IDs, for local use;

and further having an interface for informing a packet scheduler of the connection IDs that are cached for each TCP/IP processor engine resource.

9. The TCP/IP processor engine of claim 8 further comprising a session database lookup engine and a session manager which, in response to an indication that the packet processor does not hold the session entry for the specific connection required for said session, work with said hash engine to retrieve said session entry from a global session memory through a memory controller interface and to replace said session entry in said packet processor.

10. The combination of claim 9 wherein the session manager includes means, operative upon fetching of a session entry or its fields corresponding to a packet being processed by said TCP/IP processor engine, said means working with said hash engine to generate a session identifier for retrieving the corresponding session entry or its fields from a session database cache.

11. The combination of claim 9 wherein the session manager includes means, operative upon storing of a session entry or its fields corresponding to the packet being processed by said TCP/IP processor engine, said means acting with the said hash engine to generate a session identifier, for storing the corresponding session entry or its fields to the session database cache to replace a session entry as a result of packet processing.

12. The session database look-up engine of claim 9 further comprising means operative upon the fetching of a new session entry from said global session memory, said means storing the replaced session entry in said global session memory.

13. The TCP/IP processor engine of claim 9 wherein said session entry in said cache is exclusively cached to one of a plurality of processors so that request for access to said cache by more than one of said plurality does not cause any race conditions by non-exclusive access.

14. The TCP/IP processor engine of claim 13 wherein when a session entry is exclusively cached to one processor, and another processor requests said session entry, said entry is transferred to the requesting processor with exclusive access to said session entry.

15. The TCP/IP processor engine of claim 8 further comprising a TCP state machine capable of state transitions and capable of having a current state and generating a next state, said state machine receiving

a state information stored in the session entry;

and

appropriate fields affecting said state information from a fetched or newly received packet being processed for allowing the state machine to generate said next state if there is a state transition, and updating information in said session entry in cache to indicate said next state.

16. The TCP/IP processor engine of claim 9 further comprising a programmable frame controller and out of order manager used for extracting frame information from said packets and performing operations for execution of packets received out of order from the expected sequence for their session or flow.

17. The TCP/IP processor engine of claim 16 operating on an upper layer and having an upper layer framing mechanism used by said programmable frame controller and out of order manager to extract the embedded PDUs from packets arriving out of order and allowing said PDUs to be directed to an end buffer destination.

18. The TCP/IP processor engine of claim 17 wherein said programmable frame controller operates on retransmitted packets.

19. The combination of claim 17 wherein the frame controller and out of order manager includes a cyclic redundancy check generator for identifying, verifying and delineating markers in the upper layer frames from the received network packets or generating the upper layer frame markers using CRC codes for packets directed to the network.

20. The TCP/IP processor engine of claim 8 including a storage engine queue which, if a packet belongs to a storage data transfer, receives one of said packets having a storage payload, or the storage payload of said packet, for processing by a storage engine.

21. The TCP/IP processor engine of claim 8 including a segmentation controller for segmenting data to be sent on the IP Network to create valid packets to transport said segmented data on the IP network.

22. The TCP/IP processor engine of claim 8 including a DMA engine for retrieving packets or commands or combinations thereof from a scheduler or a host, and storing said packets or commands or data to internal memory of a packet processor for further processing by the packet processor.

23. The TCP/IP processor engine of claim 22 including a processor for processing the said packets having packet data and extracting the packet data for transfer to an end buffer destination in a host processor.

24. The TCP/IP processor engine of claim 22 including at least one processor for processing the said host commands or retrieving outgoing host data or a combination of any of the foregoing using the said DMA engine, for additional processing by the TCP/IP processor engine to form an outgoing packet for transfer onto the IP network.

25. The TCP/IP processor engine of claim 8 further comprising means in a packet processor engine for additional processing of said packet by execution of additional processing application code.

26. The TCP/IP processor engine of claim 25 wherein said additional processing application code can be executed on said packet before or after processing by the TCP/IP, processor engine or an IP Storage processor engine or a combination thereof.

29. The TCP/IP processor engine of claim 8.wherein said session entry in cache is cached using the Modified Exclusive Shared Invalid algorithm.

27. The process, using a hardware TCP/IP processor capable of executing a transport layer RDMA protocol, of transferring data from a host processor to an output media interface of said host system using said RDMA protocol, said process including said TCP/IP processor forming headers and forming segmentation for said data to be transferred.

28. The process of claim 27 wherein at least one of said headers is a header used for transporting data through a storage area network, a local area network, a metro area network, a system area network, a wireless local area network, a personal area network, a home network, a wide area network or a combination of any of the foregoing.

30. A TCP/IP processor having transport level RDMA capability for enabling TCP or other session oriented protocols, over IP networks, said processor comprising:

a. an RDMA mechanism for performing RDMA data transfer

b. at least one TCP/IP processor engine for processing IP packets;

c. a session memory for storing session information;

d. at least one memory controller for controlling memory accesses;

e. at least one media interface for coupling to at least one network; and

f. a host interface for coupling to at least one host or a fabric interface for coupling to a fabric.

31. The TCP/IP processor of claim 30 further comprising at least one of:

a. a packet processor engine for processing packets;

b. a classification processor for classifying IP packets;

c. a flow controller for controlling data flow;

d. a policy processor for applying policies;

e. a security processor for performing security operations;

f. a controller for control plane processing;

g. a packet scheduler;

h. a packet memory for storing packets;

i. a connection manager or session controller for managing sessions;

or

j. a combination of any of the foregoing.

32. A TCP/IP processor having transport level RDMA capability for enabling TCP over IP networks, said processor including a TCP/IP stack providing TCP/IP protocol termination and origination, said processor comprising:

a. An RDMA mechanism for performing RDMA data transfer;

b. At least one TCP/IP processor engine for processing IP packets;

c. A session memory for storing session information;

d. At least one memory controller for controlling memory accesses;

e. At least one media interface for coupling to at least one network; and

f. A host interface for coupling to at least one host or fabric interface for coupling to a fabric.

33. The TCP/IP processor of claim 32 further comprising at least one of:

a. A packet processor engine for processing packets;

b. A classification processor for classifying IP packets;

c. A flow controller for controlling data flow;

d. A policy processor for applying policies;

e. A security processor for performing security operations;

f. A controller for control plane processing;

g. A packet scheduler for scheduling packets;

h. A packet memory for storing packets;

i. A connection manager or session controller for managing TCP/IP sessions;

or.

j. A combination of any of the foregoing;

37. The TCP/IP processor of claim 32 wherein said processor operates in multiple stages, including one or more stages of

a. Receiving incoming IP packets;

b. Providing security processing for said incoming IP packets if necessary;

c. Classifying said incoming IP packets;

d. Scheduling IP packets for processing;

e. Executing data and/or protocol processing operations on IP packets;

f. Providing direct memory access for transferring data/packets to or from the memory of a system external to said processor;

g. Executing protocol processing operations on data or commands forming IP packets;

h. Providing security processing for outgoing IP packets if necessary;

i. Transmitting outgoing IP packets onto a network;

or

j. A combination of any of the foregoing; and

i. Each of said stages is capable of operating on different IP packets concurrently.

34. A TCP/IP processor having transport level RDMA capability for enabling TCP over IP networks, said processor including a TCP/IP stack providing TCP/IP protocol termination and origination, said stack providing an interface to sockets layer functions in a host processor to transport data traffic, said processor comprising:

a. An RDMA mechanism for performing RDMA data transfer;

b. At least one TCP/IP processor engine for processing IP packets;

c. A session memory for storing session information;

d. At least one memory controller for controlling memory accesses;

e. A media interface for coupling to at least one network; and

f. A host interface for coupling to at least one host or fabric interface for coupling to a fabric.

35. The TCP/IP processor of claim 34 further comprising at least one of:

a. A packet processor engine for processing packets;

b. A classification processor for classifying IP packets;

c. A flow controller for controlling data flow;

d. A policy processor for applying policies;

e. A security processor for performing security operations;

f. A controller for control plane processing;

g. A packet scheduler for scheduling packets;

h. A packet memory for storing packets;

i. A connection manager or session controller for managing TCP/IP sessions;

or

j. A combination of any of the foregoing;

36. The TCP/IP processor of claim 34 wherein said processor operates in multiple stages, including one or more stages of

a. Receiving incoming IP packets;

b. Providing security processing for said incoming IP packets if necessary;

c. Classifying said incoming IP packets;

d. Scheduling IP packets for processing;

e. Executing data and/or protocol processing operations on IP packets;

f. Providing direct memory access for transferring data/packets to or from the memory of a system external to said processor;

g. Executing protocol processing operations on data or commands forming IP packets;

h. Providing security processing for outgoing IP packets if necessary;

i. Transmitting outgoing IP packets onto a network;

or

j. A combination of any of the foregoing; and

i. Each of said stages is capable of operating on different IP packets concurrently.