“Fix It Without Changing It”? HOW?!
I work as a tech engineer for primary schools. Here I am, about three years ago, just promoted to Network Engineer from a service desk position, sent out to set up new Wi-Fi access points (APs) for a customer. The ticket states that their alarm system is fickle and I should steer clear of it.
No biggie: five APs. That’s about an hour of work, including configuring SSIDs and checking if they are placed conform to Sitesurvey. I send commands into one and… no connection. I check the IP I got from the DHCP server and it’s off.
Off to the patch cabinet, and lo and behold: in a world where 90% of all primary school devices use Wi-Fi, and 10/100 switches are a thing of the past, I find a 1000mbit fiber switch, functioning as a core switch to a stack of 10/100 UTP switches. Our firewall can be found here, as well, but no devices I can think of that could produce a second DHCP server.
So, I set a static IP in the subnet I got from this rogue DHCP server, and I manage to find a Gigaset VoIP box. I log in with the default credentials, and to my shock, it’s actually running a DHCP server and functions as a PPPoE modem toward an Internet connection no one heard of.
This is where the fun starts. I turn off the DHCP function and start configuring my APs. Halfway through, the alarm sets off. The customer is angry because I changed something. I’m flabbergasted.
I decide to go the “make the customer happy” way and go and fix it right this time. So, first things first: how is this alarm system configured? Luckily, VoIP and the alarm were installed by the same provider. Sadly, this provider sold all their alarm customers and all their VoIP customers to another third party. Both were unaware of what was actually installed due to a horrible handover.
The alarm company luckily can tell that they only have systems using IP to manage, and all they should need is a specific open port to the outer world. This goes against the interpretation of our customer, who is 100% sure the alarm uses a phone connection.
The VoIP company doesn’t have any information on the VoIP box I found and decides to just send someone to handle it and document everything to make sure this doesn’t happen again.
Since the VoIP box isn’t in use, doesn’t contain any SIP information, and can’t be found, I decide to remove it from the network. These old switches, lacking proper ARP support, make this an old-school “let’s ping this and pull cables” search. But after eliminating this bugger and resetting the alarm and the actually used VoIP system, I’m finally ready to continue configuring the APs.
Done? No. The party that installed the APs neglected the request to remove the old ones from above the ceiling plates, probably due to a lack of documentation. (Note: we were not in charge of the old APs).
So, I track them down and remove these APs, which are covered in crusty mouse droppings. Never have I been so happy with a bar of soap and hand sanitizer.
When packing my stuff to leave, I hear the school head on the phone with one of our Relationship managers, complaining about the fact that I managed to set off the alarm and shouldn’t have touched it.
IT in primary schools is usually sub-par, but this school took the cake and complained about me after I fixed most of their issues, while not even being there for troubleshooting