Method and apparatus for providing security in a star network connection using public key cryptography

• US 6,311,218 B1
• Filed: 10/28/1997
• Issued: 10/30/2001
• Est. Priority Date: 10/17/1996
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: 3Com Corporation (HP Inc.)
• Inventors: Sherer, William Paul, Nessett, Danny M., Jain, Vipin Kumar
• Primary Examiner(s): Maung, Zarni
• Assistant Examiner(s): EL HADY, NABIL M
• Application Number:
  US08/955,869
• Time in Patent Office: 1,463 Days
• Field of Search: 713/200, 713/201, 713/202, 709/225-229, 370/427, 370/446, 380/23, 380/25
• US Class Current: 709/229
• CPC Class Codes: H04L 12/413 with random access, e.g. carrier-sense multiple-access with collision detection [CSMA-CD]
• CPC Class Codes: H04L 12/44 Star or tree networks
• CPC Class Codes: H04L 12/4625 Single bridge functionality, e.g. connection of two networks over a single bridge
• CPC Class Codes: H04L 63/0442 wherein the sending and receiving network entities apply asymmetric encryption, i.e. different keys for encryption and decryption cryptographic mechanisms or cryptographic arrangements for public-key encryption H04L9/30
• CPC Class Codes: H04L 63/0823 using certificates cryptographic mechanisms or cryptographic arrangements for entity authentication involving certificates H04L9/3263
• CPC Class Codes: H04L 69/324 in the data link layer [OSI layer 2], e.g. HDLC
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

An intermediate system authenticates using cryptography. The authentication routine requires a user to supply a secret known only to the user before allowing data to be transmitted. The secret is never transmitted. The invention may be incorporated into an intermediate system, into intermediate system software, or into application specific integrated circuits designed for use in an intermediate system. The invention may include components that interact specifically with installed components in an end system or elsewhere in a network.

Claims

1. In a packet switched network wherein a user can physically connect and disconnect at an arbitrary port of said network, a method for limiting transmission or reception of data from an unauthenticated user, said method not including user identification or port identification information in data packets which are ultimately transmitted in said packet switched network, said method comprising the steps of:

storing in a network intermediate system an indication for each port indicating whether said port is authenticated or unauthenticated;

prior to freely transmitting or receiving network data on any said port, authenticating each said user at the port to which said user is connected, said authenticating of said user by said network intermediate system comprising;

requesting via said network intermediate system an identification from said user connected at said specific port;

receiving at said network intermediate system said identification from said user;

using from said network intermediate system said identification to determine a public encryption key for said user;

generating a challenge by said network for said user;

presenting said challenge from said network intermediate system to said user;

receiving from said user at said network intermediate system a response to said challenge, said response generated at said user by operating on said challenge using a private key supplied by said user;

verifying via said network intermediate system said response by performing a processing operation on said response and said challenge using said public key associated with said user;

authenticating said user via said network intermediate system only if said verifying indicates that said private key corresponds to said public key;

computing a response at a physically identified end system from said challenge, said response requiring a user to supply a private key at said end system said step of computing requiring a human user to supply at said end system a private key that is stored on a physical device that must be present at said end system, and

once said user is authenticated, allowing data packets to be transmitted or received by said user without including user identification or port identification information in said data packets and without requiring said network intermediate system to create and maintain an association between said user and a corresponding port of said packet switched network.

2. The method according to claim 1 wherein said step of requesting comprises requesting an identification to be supplied by a user said identification possibly shared with or passed from other login routines.

3. The method according to claim 1 wherein said step of requesting comprises prompting a user for a user name.

4. The method according to claim 1 wherein said step of using said identification comprises passing said identification to a security server to determine a public key for said user.

5. The method according to claim 1 wherein said step of using said identification comprises using said identification to look up a certificate in a table stored in a first network resource and decoding that certificate via a second network resource to determine a public key for said user.

6. The method according to claim 1 wherein said step of using comprises using said identification to look up a certification path in a table stored in a first network resource and decoding that certification path via a second network resource to retrieve said public key.

7. The method according to claim 1 wherein said step of using said identification comprises transmitting an identification from said networ intermediate system to a directory server, said directory server retrieving a certificate that corresponds to said identification and returning said certificate to said network intermediate server.

8. The method according to claim 1 wherein said step of using said identification comprises transmitting an identification from said network intermediate system to a directory server, said directory server retrieving a certification path that corresponds to said identification and returning said certification path to said network intermediate server.

9. The method according to claim 1 wherein said step of computing comprises using a private key for said user and public-key/private-key encryption algorith to generate said response.

10. The method according to claim 1 further comprising:

filtering network data to and from said arbitrary port based on the authentication status of the user at that port.

11. The method according to claim 1 wherein said step of generating a challenge comprises generating a random number.

12. The method according to claim 1 wherein said step of generating a challenge comprises using said public key and public-key/private-key encryption algorithm to generate said challenge by encrypting a message, said challenge comprising said encrypted message.

13. The method according to claim 1 wherein said step of verifying comprises comparing said response to an expected response at said intermediate system using a fast compare engine.

14. The method according to claim 1 wherein said step of generating a challenge is performed by a security server.

15. The method according to claim 12 wherein said step of verifying comprises comparing said response to said message.

16. The method according to claim 1 wherein said step of verifying comprises decrypting said response using said public key and comparing the result to said challenge.

17. The method according to claim 16 wherein said decrypting of said response is performed by a security server.

18. A secure hub in a packet switched communication network comprising:

a port state variable containing state for a port indicating whether an end system connected to said port has been authenticated;

an authentication process capable of receiving a user identification from a port and of communicating with network devices in order to receive a public key for said user and challenge data to present to a port, said authentication process allowing data packets to be transmitted or received by said user without including said user identification or port identification information in said data packets and without requiring a network intermediate system to create and maintain an association between said user and a corresponding port of said packet switched communication network,

a verification engine for determining if a response is valid by operating on said response and said challenge using said public key and a public key/private key encryption algorithm, said verification engine further adapted to compute a response at a physically identified end system from said challenge, said response requiring a user to supply a private key at said end system said computing of said response requiring a human user to supply at said end system a private key that is stored on a physical device that must be present at said end system; and

a controller capable of setting an authentication bit for a port in response to a result from said comparator.

19. The device according to claim 18 further comprising a second state indicating whether at least one port is an unprotected port out of which data from unauthenticated ports may be transmitted.

20. In a packet switched communication network, a method for improving network security comprising:

storing at a first intermediate system to which a network end system connects state indicating whether a network port connecting said intermediate system to an end system is authenticated;

setting said state to an unauthenticated state at intermediate system power up or as programmed by a network manager;

performing an authentication process on an unauthenticated port prior to an unrestricted transmission or reception of network data through said intermediate system to said unauthenticated port said authentication process comprising using a public key/private key encryption algorithm to determine that an end system connected to an unauthenticated port is in possession of a valid private key, said authentication process further adapted to compute a response at said end system from a challenge, said response requiring a user to supply a private key at said end system said computing of said response requiring a human user to supply at said end system a private key that is stored on a physical device that must be present at said end system, said authentication process allowing data packets to be transmitted or received by said user without including a user identification or port identification information in said data packets and without requiring said network intermediate system to create and maintain an association between said user and a corresponding port of said packet switched communication network;

resetting said authentication state to an unauthenticated state whenever an interruption in the physical connection between a port and an end system is detected; and

setting said state to an authenticated state only when said authentication process returns a message that authentication has been validated.

21. A packet switched communication network having enhanced security comprising:

an end system interacting with a user;

a state variable containing state indicating whether a connected end system has been authenticated;

a secure hub providing a private connection between said end system and network data and capable of filtering data to said end system based on an authentication state for said end system;

an authentication process capable of receiving a user identification from an end system and capable of retrieving a mechanism for generating a challenge to present to an end system, said authentication process further adapted to compute a response at said end system from said challenge, said response requiring a user to supply a private key at said end system said computing of said response requiring a human user to supply at said end system a private key that is stored on a physical device that must be present at said end system, said authentication process allowing data packets to be transmitted or received by said user without including a user identification or port identification information in said data packets and without requiring a network intermediate system to create and maintain an association between said user and a corresponding port of said packet switched communication network;

a comparator for comparing a response from an end system to a challenge, the validity of said response depending on a secret key being known at said end system; and

a controller capable of setting an authentication bit for an end system in response to a result from said comparator.

22. The device according to claim 21 wherein said network is a local area network.