Graceful Domain Controller Promotion and Demotion – Part 1

So…first post. For this first post, I thought I’d finally publish my write-up on graceful Domain Controller promotions and demotions. For those of you I’ve worked with directly, you’ll recognize this as a more comprehensive version of the write-ups I’ve provided over the last few years and helped many of you implement.

A little bit of background

Back in late 2011, I started doing a lot of work as part of a team focusing on strategies around Active Directory infrastructure upgrades. While every scenario had it’s own unique quirks, there were a number of common pitfalls that AD admins face when upgrading (whether they develop a strategy or just forge ahead and hope for the best).

Of the issues encountered, I was particularly interested in the challenges posed with promoting new Domain Controllers and decommissioning old ones. A big part of which really broke down to a larger issue of Domain Controller lifecycle management. This led me to rethink how I helped customers and instead develop a strategy.

One important note that I’ll make here: This post is not going to cover steps required to develop a back out plan prior to the AD Schema extensions. Check here for a good write up on the only real back out plan.

The problem

As most of us already know, major changes to your Active Directory environment should be comprehensively tested before production. In an ideal world, you’d have a full test environment to use to develop tested roll out and back out plans *before* production deployment…hopefully avoiding little things like mission critical application outages.

But, let’s be practical. Does your organization have the time, staff, and resources to maintain a comprehensive dev/test environment that lets you test major AD changes against critical applications and services? Does your organization have comprehensive documentation on application dependencies? Unfortunately, these aren’t uncommon gaps. Which leads us to an important question: Can we implement an upgrade strategy that reduces the risk to production applications and services even without these two? The answer, of course, is YES we can.

Promoting new Domain Controllers the old way

In the past, upgrading the Active Directory environment primarily revolved around one question: Do we take the name/IP address from a legacy Domain Controller and put it on the new DC or do we use new names/IP addresses. Unfortunately, this doesn’t deal with the bigger issue of testing mission-critical applications against new Domain Controllers. And, the easier solution of keeping the same name/IP address can trap you into retaining old DNS records, DC names, and bad applications configurations.

In short, if you want the ability to test applications before full production deployment and also provide a means to identify issues with DC name/IP dependencies during the later DC demotion phase, continue reading.

Promoting new Domain Controllers the new way – Prerequisites

The first part of gracefully promoting our new Domain Controllers is laying the foundation. We’ll do this by completing several prerequisite steps before promoting any new Domain Controllers. The goal at the end of this process is to provide a mechanism to introduce new Domain Controllers into production while simultaneously limiting the clients/servers/services that can find and interact with them. By doing this, we’ll then have the ability to test critical applications against new Domain Controllers in a tightly controlled manner.

The key to this is first identifying how clients tend to find Domain Controllers then manipulating the Domain Controller to prevent this type of discover. In my experience, the most common methods clients, servers, and applications will use to find a new Domain Controller include:
a. Targeting the Domain name as an ldap server (all DCs register IP address as alias of the domain name by default)
b. DC Locator discovery with client in DC Site (subnet present in AD Sites & Services, site-specific SRV lookup)
c. DC Locator discovery of random, generic DC (subnet NOT present in AD Sites & Services, generic SRV lookup)d. WINS lookup
These discovery methods will be managed by implementing this process.

Note that a number of other DC discovery methods are possible, but not as likely with a net-new DC, including:
a. Hard-coding a specific DC
b. Round-robin DNS A records
c. Hardware Load Balance VIP for LDAP

Step 1: Extend the schema

After developing a legitimate Forest Recovery plan (or accepting the risk…), extend the AD schema and prepare the domain to the appropriate level to promote the new Domain Controller.

Step 2: Prepare the new Domain Controller

Prepare the soon-to-be Domain Controller as you would any other server, including:
2.1 Install Operating System
2.2 Apply security updates
2.3 Install Antivirus
2.4 Reserve and document the IP address for the new server
2.5 Ensure that no WINS configuration is present in the NIC
2.6 Join the computer to the domain
2.7 Do NOT promote this server as a Domain Controller

Step 3: Prepare Active Directory Sites & Services

In this step, you will create a new AD Site, Site Link, and subnet that will be used later to control client/service discovery of the new Domain Controller.

3.1 Open Active Directory Sites & Services
Step3_1_OpenADSites

3.2 Right-Click Sites and select New Site…
Step3_2_CreateSite

3.3 Name the Site and select an existing Site Link to connect this Site (I always use Maintenance)
Step3_3_NameSite

3.4 Navigate to Sites > Inter-Site Transports > IP, right-click IP and select New Site Link…
Step3_4_CreateSiteLink

3.5 Enter a name for the Site Link and select MAINTENANCE and your main Site
Step3_5_NameSiteLink

3.6 Right-click the new Site Link and select Properties
Step3_6_EditNewSiteLink

3.7 Set “Replicate every” to 15 minutes and click OK.
Step3_7_SetSiteLinkInterval

3.8 Edit the original Site Link and remove the MAINTENANCE Site and click OK
Step3_8_EditOldSiteLink

3.9 Back at the main AD Sites & Services console, right-click and select properties on Subnets and select New Subnet…
Step3_9_CreateSubnet

3.10 Enter the IP static address reserved previously with /32, select the MAINTENANCE site, and click OK.
Step3_10_EditSubnet

Step 4: Create GPO to manage DNS Record registration

In this step, a GPO will be created to limit the types of DNS records registered by any DC in the MAINTENANCE site.

4.1 Open the Group Policy Management Console
Step4_1_OpenGPMC

4.2 Right-Click Group Policy Object and select New
Step4_2_CreateGPO

4.3 Name the new GPO (ex: [DC] DCLocator – GenericSRV_Disable_Maintenance) and click OK
Step4_3_NameGPO

4.4 Select the new GPO, remove Authenticated Users and add Domain Controllers
Step4_4_FixGPOSecurity

4.5 Edit the new GPO
Step4_5_EditGPO

4.6 In the Group Policy Management Editor, navigate to:
Computer Configuration \ Policies \ Administrative Templates \ System \ Net Logon \ DC Locator DNS Records
Step4_6_NavGPO

4.7 Edit the setting Specify DC Locator DNS records not registered by the DCs and enter:
LdapIpAddress Ldap Gc GcIPAddress Kdc Dc DcByGuid Rfc1510Kdc Rfc1510Kpwd Rfc1510UdpKdc Rfc1510UdpKpwd GenericGc
Step4_7_EnableGPOSetting

4.8 Back in the Group Policy Management Console, navigate to the Sites node, right-click and select Show Sites
Step4_8_ShowSites

4.9 Check the box next to the MAINTENANCE site
Step4_9_SelectMaintenance

4.10 Expand Sites, right-click the MAINTENANCE site, and select Link an Existing GPO…
Step4_10_LinkGPO

4.11 Select the previously created GPO and click OK
Step4_11_SelectGPO

Step 5: Create GPO to enable SiteCostedReferrals (2003/2003R2 DCs only)

In this step, a GPO will be created to enabled SiteCostedReferrals on any remaining 2003 or 2003 R2 Domain Controllers. Two important notes for this steps:

i. The Group Policy Preferences Client Side Extension update must be installed (KB943729). If not, either install them or import the registry keys another way.

ii. This step is only necessary if SiteCostedReferrals is not currently enabled on DCs. Windows 2008 and above automatically enable this setting. Therefore, you may skip this step if no Windows 2003/2003R2 Domain Controllers are left.

5.1 Open the Group Policy Management Console
Step5_1_OpenGPMC

5.2 Right-Click Group Policy Object and select New
Step5_2_CreateGPO

5.3 Name the new GPO (ex: [DC] Sysvol – SiteCostedReferrals_Enable) and click OK
Step5_3_NameGPO

5.4 Select the new GPO, remove Authenticated Users and add Domain Controllers
Step5_4_FixGPOSecurity

5.5 Edit the new GPO
Step5_5_EditGPO

5.6 In the Group Policy Management Editor, navigate to:
Computer Configuration \ Preferences \ Windows Settings \ Registry
Step5_6_NavGPP

5.7 Right-click Registry and select New Registry Item
Step5_7_NewRegItem

5.8 Enter the following in the new registry item window
Key Path: HKLM\System\CurrentControlSet\Services\Dfs\Parameters
Value Name: SiteCostedReferrals
Value type: REG_DWORD
Value data: 1
Step5_8_SetRegValue

5.9 Select the Common tab and check the Remove this item when it is no longer applied
Step5_9_ConfigRemove

5.10 Check the Item-level targeting check box
Step5_10_ItemLevel

5.11 Click Targeting… and add two items then change condition to OR:
Operating System \ Product \ Windows Server 2003
Operating System \ Product \ Windows Server 2003 R2
Step5_11_Filter2003OS

5.12 Back in the Group Policy Management Console, navigate to the Domain Controllers OU
Step5_12_NavDCOU

5.13 Right-click the Domain Controllers OU and click Link an Existing GPO…
Step5_13_LinkGPO

5.14 Select the SiteCostedReferrals GPO and click OK
Step5_14_SelectGPO

 

Step 6: Promote the new Domain Controller

Now that the framework has been laid, the new Domain Controller can be promoted into the environment. Note the new Domain Controller should automatically detect the MAINTENANCE site during promotion.

Since some of you may be promoting your first 2012 (R2), I’ll walk through the steps to promote and also verify the new DC properly discovers the isolated AD Site.

6.1 Logon to the new Windows 2012 R2 server. Ensure the following prerequisites are completed:
a) Apply all security and functional updates
b) Install antivirus and updated definitions
c) Name the server
d) Join the AD Domain as a member server
d) Configure NIC with IP address and DNS servers. This should be the IP address created in AD Sites & Services in Part 1.

6.2  Open Server Manager and select Manage > Add Roles and Features
Step6_2_AddRoles

6.3 Click Next on the Before you begin, Select installation type, and Select destination server screens
Step6_3_StartWizard

6.4 Check the Active Directory Domain Services box, confirm the additional features in the pop-up dialog.
Step6_4_SelectADDS

6.5 Click Next at Select server roles, Select features, and Active Directory Domain Services screens.
Step6_5_ADDS

6.6 At the Confirm installation selections screen, click Install.
Step6_6_Install

6.7 After install completes, click the Notifications button in Server Manager and select Promote this server to a domain controller.
Step6_7_PromoteServer

6.8 At the Deployment Configuration screen, select enter the domain name and credentials for a Domain Admin.
Step6_8_Configuration

6.9 At the Domain Controller Options screen, confirm the MAINTENANCE site is automatically selected.

Note: If the MAINTENANCE site was not automatically selected, the IP address for this host does not match the /32 subnet created in Part 1. You may work around this by manually selecting the MAINTENANCE site, but should probably fix the NIC configuration on this host if the IP Address is incorrect.
Step6_9_DCOptions

6.10 At the DNS Options screen, click Next. A warning may be present about DNS delegation,  but this is expected in most environments if not promoting a DC for a child domain.
Step6_10_DNSDelegation

6.11 At the Additional Options and Paths screens, you may choose a specific DC to replicate from and alter the AD database, logs, and SYSVOL paths. Customize if needed, and click Next at each.
Step6_11_ReplicateFrom

Step6_12_Directories

6.12 Click Next at Review Options then Install at Prerequisites Check.

Step6_13_PrerequisitesGo

Done! Once the DC restarts and finishes final replication, the promotion is complete.

 

Step 7: Verify the DC Locator GPO restrictions worked as expected

Now that the Domain Controller has been promoted, we’ll follow a few steps to verify the isolation tactics are working as expected. This will include verification that all generic DNS records were properly prevented from creation.

One important note: depending on state of the server prior to promotion, the Generic SRV and other DNS records may not be removed properly for a while. In testing, if the DC is first domain-joined, then promoted, the generic DC Locator records are not present (as expected) after promotion. If, however, the DC is promoting from a WORKGROUP state, the records will be initially created and could persist for several hours before removal.

Now, on to verifying!

7.1 Open DNS Manager and navigate to the _msdcs zone for the domain.

Step7_1_MSDCSZone

Note that _msdcs DNS zone is not necessarily a single DNS zone in your forest and may also not be configured in exactly the same way. Let’s take a look at some examples from a fictional AD Forest corp.contoso.com.

The Forest Root Domain will include an _msdcs DNS zone. It is generally configured in one of two ways.
a. The _msdcs DNS zone may be a separate top-level DNS zone (_msdcs.corp.contoso.com and corp.contoso.com will be separate). A delegation record should be present in the corp.contoso.com DNS zone (appears as a grey folder in Windows DNS). This is the preferred configuration.
Step7_1a_MSDCSDelegation
b. The corp.contoso.com DNS zone may instead have an _msdcs zone as a subdomain. Where the first option would have a grey colored delegation record, this would have a normal DNS subdomain folder.

Each child domain in a multi-domain AD Forest will also have it’s own _msdcs DNS zone as a subdomain of the child domains primary zone (ex: _msdcs.child.corp.contoso.com).

Both Root and Child domains will have SRV records for the respective domains (though the Root has additional records).

7.2 Navigate to the _tcp.dc._msdcs subdomain of the DNS zone for the domain and verify that no SRV records exist for the new Domain Controller

Step7_2_NoGenericSRV

7.3 Navigate to the primary DNS zone (corp.contoso.com in the previous example). and verify no (same as parent folder) Host (A) record exists for the IP address of the new Domain Controller.

Step7_3_NoLDAPIPAddress

Note: In this example, the Host (A) record for 192.168.15.201 is not present. This is the IP Address of the new 2012 R2 DC.

7.4 Navigate to the _tcp.MAINTENANCE._sites.dc._msdcs zone and verify that SRV records DO exit for the new Domain Controller.

This ensures our site-specific records are registering properly.

Step7_4_YesMaintenanceSRV

There are additional SRV and A records that were specifically excluded from creation by the previously created GPO and can also be checked. However, if the above are missing, then the other should be gone as well.

Finally, keep in mind the comment near the beginning of this step. Depending on GPO refresh intervals, the state of the DC before promotion, and other variables, the generic records may or may not be present when checked.

If the generic SRV records ARE present, you can do one of two things.
a. Wait! Eventually they will be removed. They should have been prevented from creation if the steps were followed, but might be present if not.
b. Remove the GPO that disables SRV registration (unlink, disable link, move the DC to another site, change security filter, etc). Wait for changes to replicate to the new DC, then run gpupdate /force. Revert the changes to the GPO, wait for replication, and re-run gpupdate /force.

 

 

Framework complete!

These first steps laid the foundation to the demotion/promotion strategy and introduced the first new Domain Controller.

Stay tuned for Part 2 in this series where we’ll focus on tactics to test applications, clients, and more.

Posted in Active Directory

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: