Windows Admins: Losing Domain Controller Replication

[choprzrul]

If the Windows Domain Controller is up and running, but can't replicate, what will happen and when?

In a large enterprise network environment with multiple domain controllers (DC) in multiple geographically separated locations; if one of those DCs were to lose its WAN connection to the rest of the domain, what happens?

The type of scenario where an island gets its fiber optic cable severed and loses all data and voice communications. No cell phones, no satellite internet, nothing.

The remote location has:

1. it's own currently replicated DC (Server 2016)
2. DNS server (Server 2012)
3. all network equipment is Cisco
4. about 1500 workstations, laptops, and printers
5. about 220 servers (Server 2008, 2012, 2016, 2019, and a handful of Linux)

The remote location will not have any time synchronization ability to the outside world and it does not have its own time server.

Physical access to the remote site may not be possible for several weeks or months. However, the site has nearly unlimited power, security, water, and food resources.

How long will that domain segment run by itself? What problems should be expected and how long before they present? What can be done ahead of time to mitigate potential issues?



.

[SkyHawk]

All the domain authenticated clients (wkstns/servers) at the site should be using the local DC for their time source, so it will not matter if the DC time drifts. The clients will just drift with it.

Clients/users will still be able to authenticate but no new users/group changes etc unless someone there can add them (assuming the DC is not just a read-only slave)

In your hypothetical scenario, it should run for a long time - nothing should stop it barring hardware problems. It may get sporty when the WAN link comes back and the DCs try and start talking to each other if the remote DC time has drifted a lot and the remote DC was not a slave. In that case I would sync the DC with an external time source (NIST etc) before bringing the WAN back up.

If you expected to need to make changes to DNS, groups users etc during this fallout scenario, then the DC should not be read only. Also if someone has any access to an authoritative time source (phone a friend or whatever), they should periodically set the DC time manually if it is observed to drift more than a few minutes.

Someone may also want to keep a log of any changes made (if feasible) at the remote site (DNS, AD groups) just so they can be checked after a DC sync. Or for future auditing purposes just dump the AD to CSV regularly and export the dns zone files (time/date stamped file names). And to the extent possible, no changes should be made from HQ to users or groups in the affected site until the issue is resolved.

Most of the 'excitement' will start when the sites start talking again, especially if there have been competing changes made to objects. AD will be fine but the admins/humans on each side of the link may have to reconcile what has happened and determine if the objects are still in the desired state.

There could be more to it - good subject for a couple of beers around the fire.

[bigmike82]

Fun fact. Maersk was able to recover relatively quickly after NotPetya due to a domain controller that was isolated by a power outage just prior to the attack.

https://redmondmag.com/blogs/scott-bekker/2018/08/domain-controller-nightmare.aspx