Recovering a Windows Domain Controller on AWS After a Network Misconfiguration

Few outages feel quite as bad as the one you caused yourself. A tidy-up of a security group, a “harmless” route table change, a new VPC DHCP option set, or a static IP typed into the wrong adapter inside Windows — and the EC2 instance running your domain controller stops answering. RDP times out. Replication stalls. Within minutes, group policy stops applying, file shares start refusing credentials, and anything that depended on that DC for DNS begins to fall over.

The good news is that a network misconfiguration almost never damages Active Directory itself. The database is fine. The machine is fine. You have simply removed your own way in. This guide walks through getting back in, fixing the fault from outside the guest OS if you have to, and — the part most write-ups skip — proving Active Directory is actually healthy afterwards.

Read this before you touch a snapshot

The instinct when a server misbehaves in the cloud is to roll it back to this morning’s EBS snapshot. On almost any other workload that is a reasonable move. On a domain controller it is the single most damaging thing you can do, and it will not fix a network problem anyway.

Active Directory tracks every change with an Update Sequence Number (USN). Each DC remembers the highest USN it has seen from every other DC. Restore a disk image and the DC goes back to an older USN — but its partners still remember the higher number. They conclude they are already up to date and quietly stop asking for anything new. That is a USN rollback, and Microsoft does not support recovering from it. The rolled-back DC is placed in a quarantined state, inbound and outbound replication is disabled, and Directory Service event 2095 appears in the log. The only supported way out is to demote it forcibly, run a metadata cleanup, and rebuild.

Step 1 — Work out which layer actually broke

Before you start attaching volumes to rescue instances, spend two minutes narrowing down where the fault sits. The symptoms are more informative than they look.

The four things to check in the console

• Security group — inbound TCP 3389 from your address, and inbound 53, 88, 135, 389, 445, 464, 636, 3268–3269 plus the RPC ephemeral range from your member subnets. Security group changes take effect immediately; no reboot is needed.
• Network ACL — NACLs are stateless. An inbound allow is useless without an outbound rule covering the ephemeral return ports. Windows Server uses 49152–65535; AWS’s own guidance is to allow 1024–65535 outbound to be safe.
• Route table — a public subnet needs 0.0.0.0/0 pointing at the internet gateway; a private subnet needs it pointing at a NAT gateway. Check the subnet the DC’s ENI actually sits in, not the one you think it sits in.
• The public IP — if the instance was stopped and started without an Elastic IP, it now has a different public address. Nothing is broken; you are knocking on the wrong door.

Step 2 — Get a session without using the network

If the console-level fixes did not bring RDP back, you need eyes on the operating system. There are three realistic routes, and they are not equally likely to work.

EC2 Serial Console

This is the only option that bypasses networking entirely — it talks to the instance below the VPC. It comes with conditions, and the important one is that most of them must be met before the incident:

• The instance must be built on the AWS Nitro System (virtualized Nitro instances, plus most bare metal types).
• Serial console access must be granted at the account level for the region.
• On Windows, you get the Special Administration Console (SAC), not a desktop — and SAC has to have been enabled in the boot configuration already.
• The instance must be in the running state.

Enabling SAC and the boot menu is a two-minute job you should do on every Windows server you care about, today, while everything still works:

bcdedit /ems {current} on
bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200

bcdedit /set {bootmgr} displaybootmenu yes
bcdedit /set {bootmgr} timeout 15
bcdedit /set {bootmgr} bootems yes

shutdown -r -t 0

From a SAC prompt you can run cmd to spawn a channel, then use netsh to repair the interface — enough to put a working IP back and let RDP in.

Systems Manager Session Manager

Excellent when it works, but SSM Agent needs to reach the Systems Manager endpoints — which is exactly the path a route table or NACL change tends to break. It only survives the outage if you have interface VPC endpoints for ssm, ssmmessages and ec2messages in the same VPC, and the security group on those endpoints still allows 443 from the DC. Check the instance’s “Ping status” in Fleet Manager: if it says Connection lost, this route is closed.

EC2 Instance Connect Endpoint

An EC2 Instance Connect Endpoint can tunnel RDP to an instance with no public IP:

aws ec2-instance-connect open-tunnel \
    --instance-id i-1234567890abcdef0 \
    --remote-port 3389 \
    --local-port 13389

Then point your RDP client at localhost:13389. Note the catch: the instance’s security group must still allow inbound from the endpoint’s security group. If a security group rule is what broke you, fix that rule first — the tunnel does not bypass it.

Step 3 — Fix it from outside the guest

Everything at the VPC layer can be corrected without touching the instance at all, and all of it applies immediately:

• Attach a corrected security group to the ENI — or create a temporary “break-glass” group allowing RDP from your office range and attach it alongside the existing ones. Security groups are additive, so this cannot make things worse.
• Repair the route table entry or re-associate the subnet with the right table.
• Restore the NACL rules, including the outbound ephemeral range.
• Re-associate the Elastic IP if it was released.

One limitation worth knowing before you plan around it: an ENI cannot be moved between subnets. If the DC genuinely ended up in the wrong subnet you are looking at attaching a second ENI in the correct subnet, or rebuilding the instance from its root volume in the right place.

Step 4 — The offline fix: rescue-instance surgery

If the misconfiguration is inside Windows — a static IP on the wrong adapter, a wiped default gateway, a firewall profile that now blocks everything — and you have no serial console, you repair it offline. This is the reliable fallback, and it does not risk USN rollback because nothing is being reverted; you are editing the live disk.

4.1 Stop the instance and detach the root volume

Stopping a domain controller cleanly is completely safe — it is a normal shutdown, not a rollback. Once it is in the stopped state, detach the root volume. On a Windows AMI the root device is exposed as /dev/sda1. Write the volume ID down before you detach it.

4.2 Attach it to a rescue instance in the same AZ

EBS volumes are locked to an Availability Zone, so your helper instance must be in the same AZ as the broken DC. Use a plain Windows Server instance — never another domain controller, and never a member server you cannot afford to reboot. Attach the volume as xvdf.

The disk will usually arrive offline because of the SAN policy. Bring it online:

diskpart
  list disk
  select disk 1
  attributes disk clear readonly
  online disk
  list volume
  select volume 2
  assign letter=D
  exit

4.3 Load the SYSTEM hive — and find the right control set

reg load HKLM\BROKEN D:\Windows\System32\config\SYSTEM
reg query HKLM\BROKEN\Select /v Current

This second command is the step almost every guide gets wrong. Offline, there is no CurrentControlSet — that key only exists at runtime as a pointer. The Current value under Select tells you which one Windows will actually use on the next boot: 0x1 means ControlSet001, 0x2 means ControlSet002. Edit the wrong one and your change will appear to have done nothing at all.

4.4 Repair the interface

Interfaces live under, using ControlSet001 as the example:

HKLM\BROKEN\ControlSet001\Services\Tcpip\Parameters\Interfaces\{adapter-GUID}

There will be several GUIDs, most of them tunnel adapters. The real NIC is the one carrying DhcpIPAddress or the static values you just set. To confirm which GUID belongs to which adapter, cross-reference the friendly name:

reg query "HKLM\BROKEN\ControlSet001\Control\Network\{4d36e972-e325-11ce-bfc1-08002be10318}\{adapter-GUID}\Connection" /v Name

The cleanest repair is to hand the adapter back to DHCP and let EC2 supply the address it expects — set EnableDHCP to 1 and delete the manual values:

set IF=HKLM\BROKEN\ControlSet001\Services\Tcpip\Parameters\Interfaces\{adapter-GUID}

reg add  "%IF%" /v EnableDHCP     /t REG_DWORD /d 1 /f
reg delete "%IF%" /v IPAddress      /f
reg delete "%IF%" /v SubnetMask     /f
reg delete "%IF%" /v DefaultGateway /f

If Windows Firewall is what locked you out, you can disable a profile offline as a temporary measure:

reg add "HKLM\BROKEN\ControlSet001\Services\SharedAccess\Parameters\FirewallPolicy\DomainProfile" /v EnableFirewall /t REG_DWORD /d 0 /f

4.5 Unload, detach, reattach, start

reg unload HKLM\BROKEN

Take the disk offline in diskpart, detach it from the rescue instance, then reattach it to the original instance as /dev/sda1 — this exact device name matters, or the instance will not boot from it. Start the instance and give it longer than usual; a DC has NTDS and DFSR to bring up before it answers.

Step 5 — Prove Active Directory is actually healthy

Getting RDP back is not the end of the job. A DC that has been off the network for hours needs to be checked, not assumed. Run these from an elevated prompt on the recovered DC:

repadmin /replsummary
repadmin /showrepl * /csv > C:\temp\repl.csv
dcdiag /v /c /e /test:dns
w32tm /query /status
nltest /dsgetdc:yourdomain.local

Or in PowerShell:

Get-ADReplicationFailure -Target DC01
Get-ADReplicationPartnerMetadata -Target DC01 | Select Partner, LastReplicationSuccess

Time is worth a special mention. Kerberos allows five minutes of clock skew by default. If the DC drifted while it was isolated, authentication will fail in ways that look nothing like a time problem. On EC2, point the guest at the Amazon Time Sync Service at 169.254.169.123 and confirm w32tm /query /status shows a sane source and a small offset.

Event IDs that matter

Step 6 — DNS, the thing that actually bites everyone

A very large share of “the domain controller is broken” incidents on AWS are really DNS incidents. Three things to check:

• The VPC DHCP option set. If domain members are being handed AmazonProvidedDNS instead of your DC’s address, nothing will resolve _ldap._tcp and every client breaks at once. DHCP option sets cannot be edited — AWS’s documentation is explicit that you must create a new set and associate it with the VPC. If someone “fixed” an option set recently, that is your prime suspect.
• The DC’s own DNS client settings. In a multi-DC domain, point the NIC at another DC first and include the loopback address further down the list — never at itself alone, or it can island itself at boot.
• Forwarders. If the DNS role forwards to an address the new route table can no longer reach, internal lookups work and everything external dies.

Quick test from a member server:

nslookup -type=SRV _ldap._tcp.dc._msdcs.yourdomain.local
ipconfig /all | findstr /i "DNS Servers"

After changing a DHCP option set, members will not pick it up until their lease renews — ipconfig /renew on each, or reboot.

Six things not to do

Making sure it cannot happen again

• Enable SAC and grant serial console access now. It is the only route in that survives a total networking mistake, and it must be configured beforehand.
• Run two domain controllers in two Availability Zones. A second DC turns this whole article into a routine repair rather than an outage.
• Deploy the SSM interface endpoints (ssm, ssmmessages, ec2messages) so Session Manager keeps working when internet routing does not.
• Keep a break-glass security group — pre-created, allowing RDP from your admin range, attached to nothing. Attaching it during an incident takes seconds.
• Take AD-aware system state backups. EBS snapshots are for the instance; Windows Server Backup system state is for Active Directory. You need both, and only one of them is safe to restore into a live domain.
• Leave the guest on DHCP. The ENI already pins the private IP. Setting a static address inside Windows on EC2 adds a way to break the instance without adding anything you did not already have.
• Alarm on it. A CloudWatch alarm on the DC’s status checks will tell you before your users do — see our guide to setting up AWS alarm email alerts.

Glossary

Frequently Asked Questions

Related reading

• How to Properly Decommission a Domain Controller (Server 2022/2025)
• Active Directory Cleanup — Find and Remove Stale Computer Accounts
• PowerShell Active Directory Management: The Definitive Guide
• AWS CloudWatch Alarms: Setting Up Email Notifications

Locked yourself out in a way this guide does not cover? Leave a comment with the symptoms and what changed just before it broke — that second detail is nearly always the answer.