About a zone…Bootstrap Recovery of DNS

This is Brad again, and today we’re going to change pace a bit and take a look at a topic that’s near and dear to me…Disaster Recovery. In particular, this post is going to focus on a DR technique intended for use with Active Directory integrated DNS zones. I call this method the Bootstrap Recovery of DNS. In practice, it turns out that accidentally (or intentionally) deleting a key DNS zone is not that uncommon of a disaster scenario. If we assume that this will happen at some point to you (as you should for most disaster and security issues), then we need to develop a plan that allows to recover without this turning into a multi-day or multi-week endeavor.

 

If it ain’t broke…wait a minute

First, let’s take a look at the classic solutions to this problem. In most examples of DNS zone recovery, you’ll find one of three solutions (assuming usage of native tools):

1. Recovery using a DNSCMD backup file

2. Recovery using the AD Recycle Bin

3. Recovery using an Authoritative Restore

 

Unfortunately, each of these methods leaves something to be desired.

 

DNSCMD backups – “The Zombie”

This solution has a couple of big caveats:

a. The command to perform a dnscmd backup (dnscmd /zoneexport <zonename.tld> <zonename.tld>.bak) will not overwrite an existing file. This means that you need to write (or find) a script to perform backups with unique names, clean out old files, and notify of failures. Not a huge blocker, but important to keep in mind as I have personally worked with environments that didn’t realize the overwrite problem….and had just a single DNS backup that was *years* old.

b. If you use one of these zoneexport files to perform a restore, you lose all of your security information on the zone. In other words, if you had Secure Dynamic DNS updates required on the zone, backed it up with zoneexport, deleted the zone, then recreated using the backup file, you would lose all of the critical security information on each record. This could lead to DNS poisoning, failure of clients to update their own records, and other issues.

In short, you can think of this method as the Pet Sematary version of the restore. They come back…they just ain’t right.

 

AD Recycle Bin – “The Someday”

Possibly the most unfortunate, but most common shortcoming with the AD Recycle Bin is simply that you need to turn it on. Now, I know that this sounds like a ridiculous issue to point out…but here’s the deal. It’s extremely common to find Active Directory environments that have never enabled the Recycle Bin feature. This can be for a number of reasons, including presence of legacy 2003 / 2008 Domain Controllers, fear of moving to Windows 2008 R2 Forest Functional level (even after all DCs have been replaced), or even that it just never done. So, please, if you are already at Windows 2008 R2 Forest Functional Level and haven’t turned on the Recycle Bin…do it now.

All of your Domain Controller running Windows 2008 R2 and not sure if the Recycle Bin has been enabled? Here’s a couple options to check:

Check Recycle Bin with a Windows 2012 R2 member server (or DC)
1. Install a Windows 2012 R2 member server (no 2012 DC required)
2. Enable the AD DS RSAT tools
3. Open the Active Directory Administrative Center
4. Select the Domain on the left
5. Look on the far right for ‘Enable Recycle Bin…’.
If it’s greyed out, you are either not yet at 2008 R2 Forest Functional level, or it’s already turned on.
6. To enable, click ‘Enable Recycle Bin…’ if the link is active.

Check Forest/Domain Functional Level with AD Domain & Trusts
1. Open AD Domains and Trusts
2. Right-click the ‘Active Directory Domains and Trusts’ node and click Raise Forest Functional Level
3. Check Current forest functional level, note the value, and click Cancel
4. Right-click the domain name node and click Raise Domain Functional Level
5. Check Current domain functional level, note the value, and click Cancel

Using PowerShell
1. Logon to a Domain Controller or any Windows 2008R2/7 host with the RSAT Active Directory tools installed.

2. Check for AD Recycle Bin:
a. Get-ADOptionalFeature “Recycle Bin Feature”
Check the EnabledScopes section. If EnabledScopes is blank, the Recycle Bin Feature has not been enabled.

3. To verify Domain/Forest Functional Level:
a. get-addomain | select domainmode
b. get-adforest | select forestmode

4. To Enable: enable-adoptionalfeature “Recycle Bin Feature” -Scope ForestOrConfigurationSet -Target “Active Directory Forest FQDN”

 

Authoritative Restore – “The Cross-You-Fingers”

For some of you, Authoritative Restore just might be the least likely of your DR plans. This can be for a variety of reasons, but it’s extremely common to find that system state restores (required to do an authoritative restore) have never been tried/tested. I can’t stress this next point enough: If you fall into this category, you *must* develop a solid DR plan that includes tested and documented system state restore steps.

With this approach, here’s some issues to consider:

  1. You have to verify you in fact have legitimate System State (or BMR if they include System State) backups long before a DR event
  2. If you have legitimate System State backups configured, but you’ve never actually attempted a restore…you don’t have backups. They must be tested to verify they’ll actually work when you need them or you really can’t rely on them.
  3. In larger environments, there may be a backup team that’s wholly responsible for backups. Unfortunately, they probably have little or no visibility into Active Directory. Similarly, you may have little or no visibility in the Backup software.
  4. In a DNS zone deletion event, we have to assume worst-case scenario…its replicated to all Domain Controllers, and caches on clients/servers are beginning to or have already expired.
  5. With the assumption that DNS is now gone, connectivity to your backup server to perform the system state restore may not be possible or may require offline documentation of IP addresses.
  6. If you are able to perform the actual System State Restore, it can take anywhere from 15 minutes to 60 minutes or more.
  7. Most importantly, all or most services (AD, DNS, mission critical business apps, etc) are down throughout this whole process

Even if you have all your ducks in a row, the infrastructure is at least partially down during this restore process…whether that’s 30 minutes, hours, or even days if the restore goes badly.

 

 

Bootstrap Recovery of DNS

Now that I’ve covered some of the bad, we need to move on to the good. We need a solution that resolves the shortcomings in the traditional approaches, takes the best aspects from each, and positions us to deal with most common scenarios. In short, we’re going to go green and drive a hybrid.

 

Prerequisites

1: Identify critical DNS servers
In many environments, there are several key DNS servers that usually are located in primary Data Centers. These do not need to be used by all clients / servers (in fact that can have huge performance and network implications). Rather, you can target a small set of DNS servers in your main data centers as the primary DNS servers in the NIC configuration of your branch office Domain Controllers. The goal is to limit the number of DCs that need to be physically touched during a DNS deletion event.

After these DNS servers have been identified, document their name / IP address and add to your offline DR documentation. If DNS is down, you may not be able to reach the servers by name.

Let’s take a quick look at why this is important.

Scenario 1 – The Catch-22:
4 Domain Controllers in 3 AD Sites
DC1 & DC2 are in the main data center
DC3 & DC4 are in two different satellite offices.
All Domain Controllers target themselves as the primary DNS

In this scenario, you might restore the DNS zone on DC1. For DC2/3/4 to receive the DNS zone by replication, they would need to be able to connect to DC1. To connect to DC1, they would need name resolution working…which needs replication working to re-populate the DNS zone locally. Here, we have a Catch-22. To fix, it would require either: a) reconfigure all DCs to point to DC1, b) restore the zone on all DCs. In either scenario, every DC needs to be touched (manually or by script).

Scenario 2 – The working solution:
4 Domain Controllers in 3 AD Sites
DC1 & DC2 are in the main data center
DC3 & DC4 are in two different satellite offices.
All Domain Controllers target either DC1 or DC2 as the primary DNS server in their NIC.

In this scenario, you would restore the DNS zone on DC1 or DC2. The other core Domain Controller (1 or 2) is already using the restored partner as its primary DNS server, thus does not need to be changed to replicate the recovered zone. DC3/4 are both using either 1 or 2 as primary and will replicate the zone when recovered. As soon as DC2 receives the restored zone and loads into DNS, no changes should be needed on DC3 or DC4 regardless of which of the two primary DNS servers is used.

In short, if we architect DNS to keep DR in mind, we can substantially minimize the work required to restore in the event of a disaster.

Now that you’ve identified your core DNS servers (and either changed DNS NIC configuration on Domain Controllers or accepted the risk), let’s move on to step 2.

 

2: Configure DNS zone backups with DNSCMD

As I already mentioned, you can use DNSCMD to export copies of your DNS zones to backup files. This should be done on at least all of you critical DNS servers

The key to this is automation. My preferred method is a GPP deployed scheduled task that executes a PowerShell script. The important piece is ensuring that the scheduled task is not running with a Domain Admin credential and the script is stored in a location that only Domain Admins, Enterprise Admins, and Domain Controllers.

Take a look at my other post on Deploying Scheduled Task drive PowerShell scripts via Group Policy Preferences: Distributed Automation using Native Tools and Scripts

I’ve published an example DNS backup script here: DNS Backup by Distributed Automation Task.

 

3: Ensure legitimate System State backups on at least 2 Domain Controllers per domain.

Either with your 3rd party backup solution or Windows Server Backup, configure at least system state backups. My preference is to install Windows Server Backup on at least 1 DC per Domain (2 if you don’t have a 3rd party solution. There’s a number of reasons, but the short version is this: Disaster Recovery is substantially simpler if we can take steps to ensure that at least one DC per domain is a self-contained DR solution. Using a built-in Backup solution facilitates this.

Importantly, be certain that you know the DSRM password for each Domain Controller. If not known, you may manually reset the DSRM password. You can find additional information here. I’ll have another post in the near future that covers a better way of setting the DSRM password in an automated fashion on all Domain Controllers.

If you haven’t installed Windows Server Backup before, here’s the steps I use to configure it:

  1. Pick 1 or more Virtual Domain Controllers
  2. Add an additional drive to the Virtual Machine.
    • Note1: Drive sizing depends on several factors. Add the combined disk space usage of OS, AD database, and SYSVOL and add 20-40GB or more depending on the desired retention. This may need to be increased later if you identify that retention does not meet needs at a given size. It’s important to note that you do not define the number of backups to retain, just the amount of space that can be used.
    • Note2: You can modify the amount of space that can be used by VSS for backups. This can be very useful if you want to use the drive to backup other items (like GPO or DNS for example)
  3. Install Windows Server Backup feature
  4. Open Windows Server Backup
    1. Backup Schedule
    2. Select Custom
    3. Click Add Item
    4. Select Bare Metal Recovery (this will automatically check all critical volumes + system state)
    5. Select backup interval / time (generally 1/day after hours)
    6. Choose Back up to a volume (middle option)
    7. Add the backup drive as the Destination volume
  5. The scheduled backup job should now complete overnight.

4. Document DNS zone location in Active Directory

AD integrated DNS zones can be stored in one of several locations. To ensure the easier recovery, you must document where these zones are stored.

Possible locations (assuming no custom application partitions have been created) include:

  • CN=MicrosoftDns,DC=ForestDnsZones,DC=<DOMAIN>,DC=<TLD>
  • CN=MicrosoftDns,DC=DomainDnsZones,DC=<DOMAIN>,DC=<TLD>
  • CN=MicrosoftDns,CN=System,DC=<DOMAIN>,DC=<TLD>

4: If possible, enable the AD Recycle Bin

If you are already at the Windows 2008 R2 Forest Functional level, enable the AD Recycle Bin. If not, take steps to remove any legacy Domain Controllers and replace with at least Windows 2008 R2. Once complete, increase the Domain/Forest functional levels and enable the AD Recycle Bin.

 

Perform the DNS Zone restore

In this section, we’re going to assume that either a critical AD-integrated DNS zone (like the DNS zone for your AD domain) has been deleted or mangled so badly that it may as well be deleted.

Step 1: Begin the Bootstrap recovery

In this first phase of the recovery process, we’re going to utilize DNSCMD backups to bootstrap the recovery process. In short, this will allow the environment’s core DNS server to begin resolving DNS requests for the domain.

  1. With your Domain Admin credentials, login to one of the critical DNS servers identified in the prerequisites steps.
  2. Navigate to the DNS backup location (C:\Windows\System32\DNS\Backup or the alternate location if using my script).
  3. Copy the backup file of the DNS zone to the C:\Windows\System32\dns directory
  4. Renamed the file to <zone fqdn>.dns (ex: corp.contoso.com.bak > corp.contoso.com.dns)
  5. Open the DNS console
  6. Right-click Forward Lookup Zones and select New Zone…
  7. Click Next at the New Zone Wizard
  8. At the Zone Type screen, be certain to uncheck ‘Store the zone in Active Directory…’, and click Next
  9. Enter the name of the deleted DNS zone in the zone name box
  10. Select the ‘Use this existing file:’ radio button.
  11. At the Dynamic Update screen, you may either select to enable or disable dynamic updates.
  12. Click Finish

 

Step 2 – Option 1: Restore the original DNS Zone using an authoritative restore

Assuming steps have been taken to properly configure System State backups (or BMR backups with Windows Server Backup), you may continue with the restore using an Authoritative Restore of the DNS zone. Note that while this is somewhat more complex than the AD Recycle Bin approach, it may be the required method depending on what was done to the original DNS zone. Note, however, this method is certainly the more complicated.

  1. Login to any Domain Controller that has System State Backups configured.
    • In most cases, you may not want to do this on one of the primary DNS server. Instead, any less-critical DNS server will work.
  2. Restart the Domain Controller in Directory Service Restore Mode (DSRM).
    • On restart, hit F8 and select DSRM.
    • As an alternate approach, consider adding DSRM as a normal boot option to Domain Controllers. Stay tuned for another post that covers how to do this easily.
  3. Login to the Domain Controller with the DSRM account:
    • Username: .\Administrator
    • Password: <The DSRM password for this Domain Controller>
  4. Restore a system state backup of this Domain Controller. If using Windows Server Backup:
    • Open Windows Server Backup
    • Click Recover…
    • Select ‘This Server (<SERVER NAME>)’ and click Next
    • Select a date prior to the DNS zone deletion
    • Select System State and click Next
    • Leave Original Location and click Next
    • Click Recover
  5. Once the recovery completes, do NOT click Restart. Instead, move the Dialog box to the side.
  6. Open a CMD prompt and enter the following:
    • ntdsutil
    • activate instance ntds
    • authoritative restore
    • restore subtree <DistinguishedName of the deleted DNS Zone>
      • Example: restore subtree “DC=corp.contoso.com,CN=MicrosoftDNS,DC=ForestDnsZones,DC=contoso,DC=com”
  7. Restart the Domain Controller and login with DA credentials.
  8. At the CMD prompt (which will display after you enter credentials), click ‘any’ key.
  9. Open the DNS console and verify the DNS zone is now present and AD integrated
  10. Open a CMD prompt and enter:
    • repadmin /syncall /APed
      • This will replicate all partitions to all partners, crossing side-boundaries, and showing the name of each DC.
  11. Login to one of the primary DNS servers
  12. Open the DNS console and navigate to Forward Lookup Zones
  13. Note the Standard Primary copy of the deleted zone is still present.
  14. Restart the DNS service and verify the zone is AD Integrated.

 

Step 2 – Option 2: Restore the original DNS Zone using the Recycle Bin

If you have the AD Recycle Bin enabled, you can use this to recover the deleted DNS zone. Unfortunately, unless the DNS zone is in the legacy Domain partition, you cannot use the GUI present in the AD Administrative Center. Instead, we’ll use PowerShell.

  1. Login to any of the functioning Domain Controllers (including a primary DNS server).
  2. Open a PowerShell prompt and restore the dns zone and records.

A very simple example:

  • $d = [DateTime]::Today.AddDays(-3)
    • Enter a number that is the days in the past just before the deletion.
  • get-adobject -filter {isDeleted -eq $true -and msds-lastKnownRdn -eq “..Deleted-contoso.com” -and modified -gt $d} -includedeletedobjects -searchbase “DC=DomainDnsZones,DC=contoso,DC=com” -property *
    • This command verifies the deleted zone is where we expect it to be…the correct AD partition
  • get-adobject -filter {isdeleted -eq $ true -and msds-lastKnownRdn -eq “..Deleted-contoso.com” -and modified -gt $d} -includedeletedobjects -searchbase “DC=DomainDnsZones,DC=contoso,DC=com” | restore-adobject
    • Now that we’ve verified it’s in the right place, we’ll restore the zone. Note, however, that it retains the ‘..Deleted-‘ prefix. We’ll fix tha tlater.
  • (get-adobject -filter {isdeleted -eq $true -and lastKnownParent -eq “DC=..Deleted-contoso.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=contoso,DC=com” -and modified -gt $d} -includedeletedobjects -searchbase “DC=DomainDnsZones,DC=contoso,DC=com”).count
    • This command verifies the expected count (roughly) of deleted records are present where we expect.
  • get-adobject -filter {isdeleted -eq $true -and lastKnownParent -eq “DC=..Deleted-contoso.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=contoso,DC=com” -and modified -gt $d} -includedeletedobjects -searchbase “DC=DomainDnsZones,DC=contoso,DC=com” | restore-adobject
    • This restore the DNS records
  • rename-adobject “DC=..Deleted-contoso.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=contoso,DC=com” -newname “contoso.com”

 

For a more sophisticated example, see my script here: Restore DNS Zones/Records with the AD Recycle Bin

 

Done

Then end. Now, you’ve completed the DNS zone recovery while using a combination of both the fastest and the most accurate methods to reduce downtime while still properly recovering the zone and records.

Stay tuned for additional posts to cover backing up your DNS zones with a script on all DCs, setting the DSRM password automatically, modifying DC boot records to add DSRM, and more.

 

DNS Backup by Distributed Automation Task
https://gallery.technet.microsoft.com/DNS-Backup-by-Distributed-06ec7230

Restore DNS Zones/Records with the AD Recycle Bin
https://gallery.technet.microsoft.com/Restore-DNS-ZonesRecords-ff0782af

Posted in Active Directory, Disaster Recovery, DNS
One comment on “About a zone…Bootstrap Recovery of DNS
  1. […] a a specific problem to solve, take a look at my previous article on Bootstrap Recovery of DNS. In that post, I specifically mention the need to implement DNS zone backups in a scalable way that […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: