So…first post. For this first post, I thought I’d finally publish my write-up on graceful Domain Controller promotions and demotions. For those of you I’ve worked with directly, you’ll recognize this as a more comprehensive version of the write-ups I’ve provided over the last few years and helped many of you implement.
A little bit of background
Back in late 2011, I started doing a lot of work as part of a team focusing on strategies around Active Directory infrastructure upgrades. While every scenario had it’s own unique quirks, there were a number of common pitfalls that AD admins face when upgrading (whether they develop a strategy or just forge ahead and hope for the best).
Of the issues encountered, I was particularly interested in the challenges posed with promoting new Domain Controllers and decommissioning old ones. A big part of which really broke down to a larger issue of Domain Controller lifecycle management. This led me to rethink how I helped customers and instead develop a strategy.
One important note that I’ll make here: This post is not going to cover steps required to develop a back out plan prior to the AD Schema extensions. Check here for a good write up on the only real back out plan.
As most of us already know, major changes to your Active Directory environment should be comprehensively tested before production. In an ideal world, you’d have a full test environment to use to develop tested roll out and back out plans *before* production deployment…hopefully avoiding little things like mission critical application outages.
But, let’s be practical. Does your organization have the time, staff, and resources to maintain a comprehensive dev/test environment that lets you test major AD changes against critical applications and services? Does your organization have comprehensive documentation on application dependencies? Unfortunately, these aren’t uncommon gaps. Which leads us to an important question: Can we implement an upgrade strategy that reduces the risk to production applications and services even without these two? The answer, of course, is YES we can.
Promoting new Domain Controllers the old way
In the past, upgrading the Active Directory environment primarily revolved around one question: Do we take the name/IP address from a legacy Domain Controller and put it on the new DC or do we use new names/IP addresses. Unfortunately, this doesn’t deal with the bigger issue of testing mission-critical applications against new Domain Controllers. And, the easier solution of keeping the same name/IP address can trap you into retaining old DNS records, DC names, and bad applications configurations.
In short, if you want the ability to test applications before full production deployment and also provide a means to identify issues with DC name/IP dependencies during the later DC demotion phase, continue reading.
Promoting new Domain Controllers the new way – Prerequisites
The first part of gracefully promoting our new Domain Controllers is laying the foundation. We’ll do this by completing several prerequisite steps before promoting any new Domain Controllers. The goal at the end of this process is to provide a mechanism to introduce new Domain Controllers into production while simultaneously limiting the clients/servers/services that can find and interact with them. By doing this, we’ll then have the ability to test critical applications against new Domain Controllers in a tightly controlled manner.
The key to this is first identifying how clients tend to find Domain Controllers then manipulating the Domain Controller to prevent this type of discover. In my experience, the most common methods clients, servers, and applications will use to find a new Domain Controller include:
a. Targeting the Domain name as an ldap server (all DCs register IP address as alias of the domain name by default)
b. DC Locator discovery with client in DC Site (subnet present in AD Sites & Services, site-specific SRV lookup)
c. DC Locator discovery of random, generic DC (subnet NOT present in AD Sites & Services, generic SRV lookup)d. WINS lookup
These discovery methods will be managed by implementing this process.
Note that a number of other DC discovery methods are possible, but not as likely with a net-new DC, including:
a. Hard-coding a specific DC
b. Round-robin DNS A records
c. Hardware Load Balance VIP for LDAP
Step 1: Extend the schema
After developing a legitimate Forest Recovery plan (or accepting the risk…), extend the AD schema and prepare the domain to the appropriate level to promote the new Domain Controller.
Step 2: Prepare the new Domain Controller
Prepare the soon-to-be Domain Controller as you would any other server, including:
2.1 Install Operating System
2.2 Apply security updates
2.3 Install Antivirus
2.4 Reserve and document the IP address for the new server
2.5 Ensure that no WINS configuration is present in the NIC
2.6 Join the computer to the domain
2.7 Do NOT promote this server as a Domain Controller
Step 3: Prepare Active Directory Sites & Services
In this step, you will create a new AD Site, Site Link, and subnet that will be used later to control client/service discovery of the new Domain Controller.
Step 4: Create GPO to manage DNS Record registration
In this step, a GPO will be created to limit the types of DNS records registered by any DC in the MAINTENANCE site.
4.7 Edit the setting Specify DC Locator DNS records not registered by the DCs and enter:
LdapIpAddress Ldap Gc GcIPAddress Kdc Dc DcByGuid Rfc1510Kdc Rfc1510Kpwd Rfc1510UdpKdc Rfc1510UdpKpwd GenericGc
Step 5: Create GPO to enable SiteCostedReferrals (2003/2003R2 DCs only)
In this step, a GPO will be created to enabled SiteCostedReferrals on any remaining 2003 or 2003 R2 Domain Controllers. Two important notes for this steps:
i. The Group Policy Preferences Client Side Extension update must be installed (KB943729). If not, either install them or import the registry keys another way.
ii. This step is only necessary if SiteCostedReferrals is not currently enabled on DCs. Windows 2008 and above automatically enable this setting. Therefore, you may skip this step if no Windows 2003/2003R2 Domain Controllers are left.
Step 6: Promote the new Domain Controller
Now that the framework has been laid, the new Domain Controller can be promoted into the environment. Note the new Domain Controller should automatically detect the MAINTENANCE site during promotion.
Since some of you may be promoting your first 2012 (R2), I’ll walk through the steps to promote and also verify the new DC properly discovers the isolated AD Site.
6.1 Logon to the new Windows 2012 R2 server. Ensure the following prerequisites are completed:
a) Apply all security and functional updates
b) Install antivirus and updated definitions
c) Name the server
d) Join the AD Domain as a member server
d) Configure NIC with IP address and DNS servers. This should be the IP address created in AD Sites & Services in Part 1.
6.9 At the Domain Controller Options screen, confirm the MAINTENANCE site is automatically selected.
Note: If the MAINTENANCE site was not automatically selected, the IP address for this host does not match the /32 subnet created in Part 1. You may work around this by manually selecting the MAINTENANCE site, but should probably fix the NIC configuration on this host if the IP Address is incorrect.
6.12 Click Next at Review Options then Install at Prerequisites Check.
Done! Once the DC restarts and finishes final replication, the promotion is complete.
Step 7: Verify the DC Locator GPO restrictions worked as expected
Now that the Domain Controller has been promoted, we’ll follow a few steps to verify the isolation tactics are working as expected. This will include verification that all generic DNS records were properly prevented from creation.
One important note: depending on state of the server prior to promotion, the Generic SRV and other DNS records may not be removed properly for a while. In testing, if the DC is first domain-joined, then promoted, the generic DC Locator records are not present (as expected) after promotion. If, however, the DC is promoting from a WORKGROUP state, the records will be initially created and could persist for several hours before removal.
Now, on to verifying!
7.1 Open DNS Manager and navigate to the _msdcs zone for the domain.
Note that _msdcs DNS zone is not necessarily a single DNS zone in your forest and may also not be configured in exactly the same way. Let’s take a look at some examples from a fictional AD Forest corp.contoso.com.
The Forest Root Domain will include an _msdcs DNS zone. It is generally configured in one of two ways.
a. The _msdcs DNS zone may be a separate top-level DNS zone (_msdcs.corp.contoso.com and corp.contoso.com will be separate). A delegation record should be present in the corp.contoso.com DNS zone (appears as a grey folder in Windows DNS). This is the preferred configuration.
b. The corp.contoso.com DNS zone may instead have an _msdcs zone as a subdomain. Where the first option would have a grey colored delegation record, this would have a normal DNS subdomain folder.
Each child domain in a multi-domain AD Forest will also have it’s own _msdcs DNS zone as a subdomain of the child domains primary zone (ex: _msdcs.child.corp.contoso.com).
Both Root and Child domains will have SRV records for the respective domains (though the Root has additional records).
7.2 Navigate to the _tcp.dc._msdcs subdomain of the DNS zone for the domain and verify that no SRV records exist for the new Domain Controller
7.3 Navigate to the primary DNS zone (corp.contoso.com in the previous example). and verify no (same as parent folder) Host (A) record exists for the IP address of the new Domain Controller.
Note: In this example, the Host (A) record for 192.168.15.201 is not present. This is the IP Address of the new 2012 R2 DC.
7.4 Navigate to the _tcp.MAINTENANCE._sites.dc._msdcs zone and verify that SRV records DO exit for the new Domain Controller.
This ensures our site-specific records are registering properly.
There are additional SRV and A records that were specifically excluded from creation by the previously created GPO and can also be checked. However, if the above are missing, then the other should be gone as well.
Finally, keep in mind the comment near the beginning of this step. Depending on GPO refresh intervals, the state of the DC before promotion, and other variables, the generic records may or may not be present when checked.
If the generic SRV records ARE present, you can do one of two things.
a. Wait! Eventually they will be removed. They should have been prevented from creation if the steps were followed, but might be present if not.
b. Remove the GPO that disables SRV registration (unlink, disable link, move the DC to another site, change security filter, etc). Wait for changes to replicate to the new DC, then run gpupdate /force. Revert the changes to the GPO, wait for replication, and re-run gpupdate /force.
These first steps laid the foundation to the demotion/promotion strategy and introduced the first new Domain Controller.
Stay tuned for Part 2 in this series where we’ll focus on tactics to test applications, clients, and more.