Oracle DBMS - Protect Critical Servers From Unauthorized Access With IPsec isolation

If you manage an Oracle database on a Windows server, you may wonder how to secure the host machine from intruders who might try to steal data or even bring down the network. You can use IPsec to isolate servers from communications with computers that aren't Windows domain members for an extra layer of security.

To help you protect our critical servers, we'll:

  • Explain the concepts of server and domain isolation.
  • Discuss the benefits and drawbacks of isolating critical servers with IPsec.
  • Show you how to configure Group Policy to require IPsec authentication for isolated servers.

Unauthorized access to critical servers can result in network downtime, lost productivity, loss of data, loss of business revenue, legal ramifications, and more. You can protect these critical servers by isolating them from computers that might pose a security risk. You can restrict communications so isolated computers can only communicate with other domain members, or only with specific computers (identified by IP address) within or outside the domain.

Server and domain isolation concepts

Critical servers fall into two categories:

    File servers and database servers that store confidential or sensitive data, such as trade secrets, financial information, employees' personal data, etc.

    Servers on which network operations are dependent, such as domain controllers, name resolution servers, DHCP servers, etc. These are sometimes called infrastructure servers.

Isolation refers to the practice of logically, rather than physically, separating servers or entire subnetworks from other computers on the network, and restricting which computers can connect to and communicate with them.

Note: It's important to note that IPsec isolation doesn't offer complete security for critical servers. It should be used as part of a multi-layered defense in-depth security plan that includes access controls, file encryption, and other security mechanisms.

You can isolate individual servers or groups of servers (logical networks), regardless of where they're physically located or how they're physically connected. There are a number of ways to create isolated networks:

  • You can use firewalls to isolate local networks from the internet, or departmental subnetworks from the larger corporate network.
  • You can create isolated networks within the organization's larger network by using Virtual LAN (VLAN) technology.
  • You can use Active Directory, Group Policy, and IPsec to isolate managed computers (that are members of a Windows domain) from non-managed computers (that aren't domain members).

Accomplishing server isolation with IPsec doesn't require any extra expenditure for hardware or software if you already have your network set up as a Windows domain. Windows 2000, XP, and Server 2003 have IPsec support built into the operating system.

Benefits of server and domain isolation

There are many benefits to isolating your critical servers:

  • Isolation protects the servers from some types of viruses and other malicious software, as well as address spoofing, session hijacking, data injection, and other common attack types.
  • Computers outside the isolated network can't initiate communications with the servers inside, preventing common attacks against those servers even if the intruder maanges to get through your firewall.
  • By requiring that computers must join the domain in order to communicate with your critical severs, you gain more control over those computers because you can then manage them as domain members.
  • Because IPsec supports not only authentication, but data integrity and confidentiality as well, an isolated server can confirm that the communications packets haven't been changed in transit, and the data inside the packet can be encrypted to prevent it from being read if it's captured in transit.
  • IPsec works at the network layer of the OSI model, so applications don't have to be modified nor have to be IPsec-aware for you to use this method.

Note: Server and domain isolation shouldn't be used as a substitute for anti-virus software, but rather, should be used in conjunction with it.

Create an isolated network

You create an isolated network within your organization and ensure that the computers you want to isolate are protected by dong the following:

  • Make the computers that you want to isolate members of an Active Directory domain.
  • Configure a policy that domain members will accept communications only from other domain members and only when those communications are authenticated and secured.
  • If desired, you can configure Group Policy to encrypt the data so that if it's intercepted by a network monitor or "sniffer," it can't be deciphered.

Tip: You can apply Group Policy settings to an Active Directory site, a domain, or an organizational unit. To apply policies more granularly within a domain, put the computers to which you want a policy to apply into an OU and then apply the Group Policy Object to that OU container.

Computers on the isolated network

The computers on the isolated network don't have to be physically located on the LAN; they can be connected through a VPN or remote access dial-in connection.

They do have to be Windows systems that are capable of joining a domain. That means the following computers can't be on the isolated network:

  • Windows 9x/Me and Windows NT computers that don't support Group Policy.
  • Handheld computers/PDAs that don't support IPsec
  • Computers running non-Windows operating systems (UNIX, Linux, Macintosh)
  • Windows computers that are members of workgroups or other (untrusted) domains

Tip: If there are specific computers on the network that aren't members of the domain, but that you trust and want to be able to communicate with the isolated computers, you can configure exceptions to allow them to initiate communications with the domain member computers.

Isolate specific servers

You can isolate individual servers that are Active Directory domain members by creating network policies that apply to those specific. Similarly to the domain policies described above, the policies would prevent the server from accepting communications from any computer that isn't a domain member, and the traffic must be authenticated and secured.

In fact, you can go a step further and allow the server to accept communications only from other domain members that belong to specific security groups. This gives you the ability to prevent communications even from other domain members if they don't belong to the right group.

Giving a new computer access to the server is simple:

  • Make the computer a member of the domain.
  • Make the computer a member of the specified security group.

How server isolation works

When you configured server isolation for a managed Active Directory domain member using IPsec, in order for communications to be successful, the following steps must be completed:

  1. The sending computer (client) must be able to send secure IPsec communications and must have domain credentials.
  2. The sending computer and the isolated server complete the mutual authentication process; that is, each verifies the identity of the other.
  3. The sending computer and the server negotiate the use of IPsec. The sending computer's IPsec policy settings must match those of the server.
  4. Subsequent packets sent between the client and the server are protected by IPsec. If you've configured the policy to provide confidentiality, the data inside the packets is encrypted.

Communications attempts by unmanaged computers

If an unmanaged computer (non-domain member) tries to open a communications session with the isolated server, and you haven't created an exception to identify it as a trusted computer, the server will recognize that the computer's packets aren't secured by IPsec and that it doesn't have domain credentials.

The server won't accept the sending computer's packets and will discard them. The server doesn't send any response to the sending computer, so no connection is ever established.

Communications attempts by excepted computers

If an unmanaged computer tries to open a communications session with the isolated server, and you've created an exception to identify it as a trusted computer, the server will recognize that the sending computer's IP address matches the lists of excepted computers in the IPsec policy.

The server will respond, and subsequent communications between the client and server will be successfully exchanged, but won't be protected by IPsec.

How server isolation works. Note that two-way communications can proceed between the server and managed computers, and between the server and excepted computers, but the communications attempt from an unmanaged computer doesn't get through to the server.

Deploy an IPsec isolation solution

To ensure that the IPsec component of the operation system is the latest version, if you're using Windows 2000 or XP computers for IPsec communications, you should ensure that the following service packs are installed:

  • Windows 2000: Service Pack 4 or above
  • Windows XP: Service Pack 2 or above

Note: Windows XP and Windows Server 2003 support Network Address Translation (NAT) Traversal, or NAT-T. This allows IPsec to work with NAT. Service Pack 4 adds this capability to Windows 2000.

Create isolation groups

In planning your IPsec isolation solution, you must separate the computers that re on or connect to your network into logical categories, or isolation groups. Examples of isolation groups include:

  • Isolated group.Computers that are trusted domain members configured to use IPsec policy to restrict computers with which they can communicate.
  • Untrusted group. Computers that can't or don't belong to the domain or can't or don't support IPsec and shouldn't be allowed to communicate with trusted computers.
  • Boundary group. Trusted computers that need to be able to communicate with specific untrusted computers, as well as with other trusted computers.

IPsec policy for computers in the isolated group is configured to only allow communications from other computers within the isolated group. If an untrusted computer attempts to initiate communications not secured by IPsec, it will be rejected.

IPsec policy for computers in the boundary group is configured to attempt to communicate using IPsec secured communications. However, if a computer in the boundary group initiates communication and is unable to use IPsec, the boundary group computer will establish communications without IPsec protection.

Warning: Boundary group computers can pose a security risk because they are to communicate both with untrusted computers and with trusted computers in the isolated group. You should only place a computer in the boundary group if absolutely necessary, and you should ensure that it has other security mechanisms in place (hardened operating system, anti-virus software, etc.).

Create exemptions

Some servers can't practically require IPsec for all communications because they're infrastructure servers that must be accessible by other computers that don't support IPsec or don't belong to the domain. Examples are DHCP servers, DNS servers, and domain controllers.

Putting these servers in the boundary group is one way to solve the problem, but this could result in performance problems because of the time involved in first attempting IPsec-secured communications and then falling back to unsecured communications.

The best way to handle this is to create an exemption list that contains the IP addresses of computers that can't use IPsec, but need to access the server.

Note: IPsec between domain controllers and domain members isn't supported by Windows, so you should use exemption lists rather than placing domain controllers in the boundary group.

To avoid having a large number of exemptions, which increases the policy size and slows performance, you should consider consolidating multiple exempt functions on the same physical server/IP address.

Go back