Server Room

How close you get to the rack depends on the employer: some desks hand out the key in your first week, others keep the room behind a badge you will never hold. Either way you have to picture what is behind the door, because every escalation you write is read by someone who works there. A rack is a vertical frame of standard-width slots, measured in rack units, filled roughly in the same order everywhere.

One cable from a desk to the internet passes through the same five things every time.

Forty-eight desk ports at a gigabit each feeding into a single ten-gigabit uplink is normal and fine, because desks are idle most of the time. It stops being fine on the morning everyone on that floor syncs a large file at once, and it looks exactly like "the internet is slow".

Traffic is then separated into VLANs, virtual networks that share the same physical wiring: one for desks, one for servers, one for wireless, one for phones, one for cameras. Two machines plugged into the same switch can be on different networks, and a routing or firewall rule between those VLANs is what lets them talk. When a whole category of device breaks at once, "which VLAN are they on" is a better question than "which floor are they on".

Wireless has a second layer. Access points look like just another device on a switch port, but they are usually managed by a controller, physical or cloud. An access point can be up, powered and healthy while the controller it depends on is not, and then a whole building loses WiFi with every AP light still green.

A single point of failure is any component whose failure takes the service with it. Finding them is mechanical. Put a finger on a user's desk on the diagram, trace the line to whatever they are trying to reach, and count everything you pass that there is only one of.

From a desk to a website that is usually: the patch lead, the floor switch, its uplink, the core switch, the firewall, the edge router, the internet circuit. Seven things, and in a lot of small offices six of them are single.

Six single points is often the right answer. Redundancy costs money and complexity, and a fifteen-person office runs perfectly well on one firewall. The thing to check is whether somebody wrote the list down, so that on the day one of them fails the recovery time is already known.

The trap is redundancy that only exists on paper. Two of a thing is redundancy for as long as the two are separate all the way down.

Before you open the monitoring board there is a machine with a user in front of it, and a short sequence of commands names the fault.

ipconfig /all

Three lines carry most of the answer. An address starting 169.254 is APIPA, automatic private addressing, which the machine gave itself because no DHCP lease arrived: look at the port, the switch and the DHCP server. A blank default gateway leaves the machine stuck inside its own subnet. The DNS servers listed are what it will actually ask.

nslookup pointed at a server you name asks that resolver, not the one the machine was handed. ipconfig /flushdns drops a stale answer. tracert shows where the path stops.

You will usually talk somebody else through it. Much of the time you are spelling these commands down a phone and having the user read each line back: the blind talk-through from The phone.

DNS turns a name into an address, and an interview will go a question or two past that.

The resolver is the server your machine asks. It answers from cache when it can, and otherwise walks the hierarchy for you: the machine makes one request and waits.

Answers carry a TTL, a time to live, the seconds they may be cached, which is why a correct change "has not worked yet": resolvers keep serving the old address until their copy expires.

Active Directory runs on DNS. The domain controllers are usually the DNS servers as well, and a client finds one to log in against through SRV records, the service records the domain publishes. Point a machine at the wrong resolver and it cannot find a controller to authenticate against, which is why "I cannot log in" so often turns out to be DNS. It is a favorite interview question for exactly that reason.

Somewhere there is a system watching all of this, and on a good day it pages somebody before a user does. The open-source monitoring products are Nagios and Zabbix, the common commercial ones PRTG and SolarWinds, and the cloud-native one Datadog. They differ in price and polish, and the job underneath is identical.

They watch in three ways. A ping, an ICMP echo, asks whether a device answers at all. SNMP, the simple network management protocol, polls a device for counters: interface up or down, throughput, CPU, temperature, fan speed, how long it has been running. An agent installed on a server reports from inside the machine: a disk at ninety-eight percent, or a service that has stopped.

"A node is down" means the monitor stopped getting an answer from it, which is a claim about the monitoring system. Read the board against who is actually on the phone.

Cisco and Meraki are trademarks of Cisco Systems, Inc. ServiceDesk Simulator is not affiliated with, endorsed by, or sponsored by them.

Read the uptime beside the alert. A switch showing six minutes restarted itself a few minutes ago, and nobody told you.

Every ticket arrives describing one person's experience, and that person can see one screen, their own. So you ask whether anybody near them has the same problem, before you diagnose anything.

Then read the queue as a whole instead of one ticket at a time. Three tickets that say "WiFi keeps dropping", "my video call died" and "the VPN disconnected" look like three unrelated faults and are frequently one access switch. The descriptions differ because the users describe whatever they happened to be doing when the network went away.

Watch the clock, not just the words. Tickets that arrive inside the same ten minutes are related until proven otherwise, however different they sound. Tickets spread across a week that sound identical are usually a coincidence.

Once you believe it is one fault affecting many people, there is a second job sitting on top of the repair: running the incident.

Declare early. Every organization has a threshold, usually a critical service unavailable or a whole site affected. Raising one that turns out to be a floor switch costs a short call. Not raising one for ninety minutes while you work quietly costs the company the ninety minutes, and costs you the trust of everyone who was not told.

A declared incident has roles, and on a large one they are different people. The incident commander decides what happens next and keeps everyone pointed the same way, while other people do the fixing. The comms lead writes the updates, so the person elbow-deep in the fault has one job. On a small desk one person wears both hats, and the updates still go out on schedule.

Say what is affected, what is not, what you are doing, and when you will next update. Give a rough ETA and label it as an estimate. People ask so they can decide whether to wait, go for lunch, or drive to another office.

Say that you fixed it. Spotting the dead switch, rebooting it and saying nothing feels efficient. Meanwhile forty people lost an hour, a dozen rang the desk, three escalated to their managers, and no record exists of what happened or why.

Afterwards there is a review, usually called a blameless postmortem: what broke, what the impact was, how it was found, what the underlying cause was, and what stops it next time. Blameless means the write-up names causes and not people. Write your part honestly, including the twenty minutes you spent on the first user's laptop, because the review is only worth holding if it is accurate.

The tool is in the Tools menu as Server Room, and it opens on four tabs: Overview, Topology, Devices and Servers.

Overview is the dashboard. The ISP card, for the internet service provider that sells the building its connection, carries a live latency figure and a small sliding chart, Network Load reads All clear or Needs attention, and two rings carry the device and the server totals with how many of each are up written beside them. Beneath those, Device Status lists every piece of network equipment with its state and Server Status lists each server with its CPU and Memory meters. The header keeps a running count of nodes up. Confirming an upstream outage is left to you: the status line for an ISP outage appears once you have run the ISP check on the Devices tab.

Topology draws the path top to bottom, and it is worth tracing with a finger the way part one asked you to: INTERNET, the ISP, EDGE ROUTER, FIREWALL, and then a horizontal bus fanning out to three zones, SERVER ROOM A, SERVER ROOM B and WIRELESS NETWORK, each tagged with its subnet such as the SERVER VLAN at 10.0.2.0/24.

Devices is the equipment table, with columns for the Device, its IP Address, its Location, its Uptime, its Status and an Action. Each row that can be power-cycled has a three-dot menu, and what it offers depends on the state of that row: a device that is up offers reboot and shut down, one already off offers power on, and one part way through a restart keeps its menu shut until it settles. The ISP row is the exception: its one action reads check isp status, which is how you ask whether the fault is upstream before touching anything of your own.

Servers is the same idea for the five servers, with a Role column and separate CPU and Memory meters. Two of them are a Domain Controller, which is the sim being quietly accurate about how domains are actually built, and the others are the File Server, the Mail Server and the Print Server.

Part one's rack carries four things this tab leaves out, and an incident review asks about all four. Out-of-band management is a card inside the server with a network address of its own, reachable when the machine will not boot, and it is what keeps a 2am reboot from being a drive to the office. A backup system, judged by the restores somebody has tested. A change record naming who altered what and when, which is the first thing anybody opens during an outage. And the door log on the room, answering the same question about people. On any incident that started this morning, ask for the change record early: the answer is usually sitting in it.

Redundancy reads differently here too. Two domain controllers are listed, and on a real network the point of the second one is that sign-ins carry on while the first is down. Failover is something planned and tested, so read each pair on this tab as an inventory and ask what would still be working if the left-hand node stopped. The physical half of that question is back in part one: which PDU a device is fed from, and whether one patch lead is the only thing holding a floor up. The alerts themselves come out of a monitoring product on a real desk, Nagios or PRTG in most shops, and the work there is reading a screen of them and writing the one sentence that goes in the ticket.