“Pwn is a leetspeak slang term derived from the verb ‘own,’ as meaning to appropriate or to conquer to gain ownership. The term implies domination or humiliation of a rival, used primarily in the internet-based video game culture to taunt an opponent who has just been soundly defeated (e.g., “You just got pwned!”).” –Wikipedia
We are in the tail end of completing our Mega Job. All the gear is in, the wiring has been terminated and connected, the programming is done, and we are down to refining and tweaking the system for the homeowner and squashing the little bugs that always creep up on big jobs.
While the job has gone incredibly smoothly for such a massive project, one thing that I did not realize would be such a major headache and source of frustration, aggravation, and time suckage on this job was the managed network. When I initially described our system goals and needs to Pakedge prior to ordering, we were told that a project our size must include a managed network and that it just wouldn’t work without it (or, more correctly, it would crawl, lag, and crash.) With 112 hardwired Ethernet ports, 21,500 square feet of multiple Wi-Fi access points and so many different systems talking back and forth, a managed network would allow us to separate traffic onto different VLANs.
The goal, of course, was to keep the data traffic from critical devices like Kaleidescape, Control4, and Lutron flowing fast and smooth, top down in a network-surfing Ferrari, while non-critical things like guests watching cat videos chugged along on a different road in a VW minibus enjoying the scenery at a lower priority. And, hey, if managed is better and was going to make the project perform like a rock star, then brother, manage me up! Money is no object. Let’s do it! #Managed4Life #Blessed
To be fair, we had never had the need to use a managed network on a project before, so we weren’t aware of the complexity, configuration, and issues that come with a managed network—IGMP Snooping, spanning tree protocol, port flow control, link aggregation… No idea. And while a managed network sounds like a great idea in theory, in practice in the resi world, it has been excruciating, and at this point I don’t know that I would ever do another project using managed switches.
I do want to give props upfront to the amazing tech-support team at Pakedge. I wrote a lengthy blog on the first bout of help we received from Network Jedi, Steve who literally spent eight-plus hours on the phone and Team Viewer with us to get our network’s management settings off on the right track. Steve was patient, knowledgeable, and non-judgmental as he worked through our initial configuration issues. And when I left the house that day/night, I felt like I had been through a lot of growing pain, but it would ultimately be worth it, and now my network would be like a caterpillar transformed into a beautiful butterfly.
However, since that phone call my techs and I have spent more time on the phone and Team Viewer with Pakedge. Lots and lots more time… And while their technical support is always patient, helpful, and awesome, it is still hours of time sitting there watching the screen as they make log into all of the different components and make adjustments and configuration changes to the network that I don’t understand. And instead of working on other items on the project, I’m forced to just sit there with fingers crossed hoping each time that the next leap will be the leap home.
The problem I am running into now is, I frequently don’t know or even have a way of knowing if the problem is A) a broken piece of Pakedge gear B) a bad management configuration in one of the many Pakedge managed devices C) a broken piece of Control4 gear D) a bad programming binding in the Control4 system E) some random curse from the fickle whim of the cruel Internet gods. Something just won’t work, and I’m left scratching my head wondering where to start the troubleshooting. And the house is SO huge that every time I need to check something, it requires a lengthy walk down long hallways and into a closet in some room where I have to pull out a rack, probe around with a flashlight, unplug and reboot something, wait for it to come back online, then test to see if the change fixed the problem, typically a process that takes 10 or more minutes.
And a Control4 processor and Pakedge managed switch aren’t like a Blu-ray player that I can just “throw in a new one real quick” to test. Or even just plug into a different switch port in many cases. Or move a non-working one to a working area.
And you know the Butterfly Effect, the theory that a butterfly flapping its wings on the other side of the world can cause some disaster locally? That’s starting to feel how things are when we make a change somewhere in the network. A priority tweak as it cascades from the router to one switch, to the next switch, to the next, and an adjustment at one side of the house might cause a crash in an unrelated system somewhere else. And with 112 hardwired Ethernet ports, and eight separate systems it can take days before anyone notices that the change “broke” something that had previously been working.
Last week I went up to the job for a simple, “Why are the lights in the theater not turning off?” service call (turned out the new Sony VPL-VW1100ES 4K projector was continuing to send “I’m off” IP commands that triggered the Control4 system to tell the Lutron system to issue the “End Movie” lighting scene) that turned into another eight-hour network, “Where’s Waldo?” hunt down as I tried to figure out why music wouldn’t stream to two of five rooms that appeared to be identically configured. Ultimately we (and I’m using the royal “we” here as it was really all Pakedge, with me offering ocular support and running around the house confirming which ports on the switches certain pieces of gear were connected to) ended up undoing a VLAN setting that we had spent hours doing the week before.
Further compounding the problem is that companies and their tech support departments aren’t designed to troubleshoot with each other. There is no quick and easy way to get Control4 on the phone with Pakedge and me to three-way out this issue—all of us looking at the same project and working together off the same page. Control4 sent me a “Technical Support Bulletin” (now almost three years old), titled “Settings on Managed Switches That Must be Disabled for a Control4 System to Operate” that started with the ominous phrase, “Using a managed network switch (not recommended or supported)…” which I shared with Pakedge. But after making some of the “dumbing down” changes suggested by the bulleting, it actually seemed to make the problems worse!
(Sigh…) (Gulp of scotch…)
As a company we realize the network has become the most crucial component of the modern systems we install, and we can no longer rely on DHCP and correctly terminated RJ45 jacks to resolve our problems. We’ve decided to take a proactive step and send someone to CEDIA’s three-day networking school and take advantage of the manufacturer trainings and face-to-face time at CEDIA Expo. This training, coupled with the support of companies like Pakedge, should have us better prepared to tackle the needs of the next Mega Job.
John Sciacca is principal of Custom Theater and Audio in Myrtle Beach, SC.
