We have been steadily plugging along at our mega install job, getting all the gear racked in, terminating all the wiring, installing all the touchpanels, hanging the projector and screen, and programming and testing all the systems and subsystems. All told, we have 109 Gigabit networking ports on this project.
Because the entire home is going to depend so completely on the network being robust and up 100 percent of the time, I knew this wasn’t something we could screw around with using off-the-shelf routers and switches. Between the Control4 processors and touchscreens, the Kaleidescape movie server, the Lutron HomeWorks lighting system—and its 350 loads and 30 drapes—and everything else, if the network were to go down on this job, it would be a system destroying disaster where literally nothing would work. (Pretty much my current reoccurring nightmare at this point.)
For that reason, I went with Pakedge for the networking backbone. To be honest, prior to this job I’d really only known Pakedge by reputation, but everything I’d heard about that reputation made me feel confident that they were the right solution. Also, their prior PR manager, the ever-sparkly Olivia D, sent me a Pak wireless access point to experiment with in my own home, and I was thoroughly impressed with its quality, performance, and 100-percent uptime.
For this job, we went with a K6 router, 24-port managed PoE switch (SW24P), 24-port managed non-PoE switch (SW24GBM), seven 8-port managed switches in racks around the home (S8Mpd), five dual-band wireless access points (4 W7 and 1 W7O for outside), a C36 WAP controller, and the P8 power manager. (I have grown to love this P8, for its ability to bring the system back on line in the correct order in the event of a sudden power failure. Again, hoping to ward off the reality of the “nothing works!" nightmare.)
I honestly didn’t know a ton about managed switches and all of the configuration and optimization possibilities, but both our reps and Pakedge had assured us over and over prior to the sale that they would be there to hold our hands and give us all of the support we needed to make sure this thing was configured and running perfectly. And prior to starting the networking configuration on this project, the part I was most concerned with was just the correct terminating of all those many, many Cat5/6 wires. I figured once we had that done correctly it would be mostly a matter of plugging them into ports on the Pakedge gear and then standing back and watching the magic happen.
Our most computer-savvy tech is the closest thing our company has to an “IT professional,” and he handled the majority of the network configuration, setting up a VLAN and subnets so the Control4, Lutron, and Kaleidescape systems could have top priority and get out of the muck-and-mire of living down in the Mos Eisley spaceport with the rest of the house’s networking scum and villainy.
Things ran mostly pretty smooth with the network, but we noticed that randomly throughout the day the system would seem to just lock us out. For like these random 10-minute periods, we couldn’t access the internet, we couldn’t get into the router, we’d lose connection to our Control4 processor. At first we chalked it up to just ghosts in the machine, or the slow internet service, or the fact that we were hammering away at the programming, but after a particularly long lockout where we weren’t taxing the system at all, we finally called up Pakedge for a, “Yo! What’s going on here?! Is something broken?!” love in.
Fairly early in the conversation, we mentioned “Sonos” and it was like we just read a particularly nasty incantation from the Necronomicon. The Pak tech immediately said that was definitely our problem and that the system had to be specially configured to handle the constant network chatter and blathering that the Sonos spouted forth. He walked my tech through some setting changes in the Pakedge menus with names like IGMP Snooping and Spanning Tree Protocol. Click-click, and the problems went away like magic. (Though, I can’t understand why a garbage, off-the-shelf router from Wal-Mart handles Sonos fine, while a $1500 enterprise-grade Pakedge switch would get tripped up by it. The only thing I can guess is that when given free rein on a managed switch that is NOT being managed, the Sonos goes all Augustus Gloop at the chocolate factory, eating up all the food and splashing around in the chocolate river and making all sorts of network ruckus and mayhem. Someone feel free to enlighten me in the comments.)
A few days later we were troubleshooting an issue where we were unable to remotely access the Control4 processor. We talked to Control4 and pretty quickly determined things looked OK on their end, and they directed us to call Pakedge for some help in configuration. This was around 11 or so on a Friday morning. So we called Pakedge, and this is where Steve, the Steveinator, network Jedi, enters the picture. What we think is going to just be a simple, “Hey, Steve, can you clicky-fix our remote access issues? Toodles!” call turns out to NOT be that. At all.
Steve took over the computer and started poking around and looking at settings. Then he asked some gentle, probing questions like, “What are you trying to do?” and “Who set this up?” and “Who is the IT expert on your staff?” Fairly quickly it became obvious that somewhere along the line of our initial setup and configuration we veered off the path of “the right way” and have instead plowed miles ahead, and now we were deep—DEEP!—in the murky, scary, backwoods, uh-oh-what's-that-shack-for?! part of “totally messed up.” And it was one of those mess-ups like when you miss a crucial plot point in a linear video game where even though you’ve been playing for an extra 1,000 hours, you need to go back and totally restart and redo that one simple thing you missed.
I’m not sure that you can actually hear someone take a deep, cleansing breath through the emotionless box of a chat window, but I picture that’s what Steve did. I also imagine him taking a super cool, Don Draper drag on a cigarette, blowing a long, lazy stream of blue-grey smoke up into a slowly twirling ceiling fan, powering down a stiff, three fingers of his favorite single malt, cracking his knuckles and saying, “OK. Let’s do this thing.”
What he actually said—in the nicest way possible—was that we had so badly messed up configuring this network that it would be easier if we just factory defaulted everything and completely started over from ground zero. I tried to keep it light and breezy by asking him just where we had gone so wrong and he replied, “You want the whole list? :-)” Ouchey.
I told him that we had to do whatever it took, but that if he burned our network house down, he would have to stay with us until we rebuilt it; we couldn’t leave the project in complete ruins, with absolutely nothing working. He asked me how long I was planning on staying and I told him, “As long as it takes. I’m here for the duration.” (I then texted Dana and told her I wouldn’t be making it home for dinner. And maybe not ever again. It was too early to tell.) Steve assured me that with enough time, anything was possible, and that he could make it right.
So, Steve set about rebuilding our network using Pakedge best practices. From his top-secret Steve Lair in the middle of Network Super Computer Land, Steve moved the mouse around, pointing, clicking, changing, rapidly circling things to show us something important, all the while rebuilding and reconfiguring.
When I walked into that house at 9:00 that morning, nearly 95 percent of everything was done and working. I thought I had a day of tweaking and fine-tuning ahead of me. Yet, several minutes into the Steve Marines Corps style, “I’m gonna tear your down, rip you apart, then rebuild you, make you stronger, better, more of a man than you’ve ever been!” network rebuild, nothing in the house worked. For several hours… At one point I’d been just sitting in a chair watching Steve move the mouse around with nothing working on our end for about three hours.
Undaunted, the Steveinator kept plowing ahead. There were quite a few moments where he would change something, ask us to try it, change something else, and ask us to try it again. After several of these attempts, we couldn’t establish even basic communication between the main rack and the sub-rack. And then we lost the ability to see any of the Control4 components on the network. Then the main processor disappeared and wouldn’t come back. Amazingly, the Kaleidescape system remained rock stable throughout all of this, continuing to stream movies, plowing through the network traffic, and totally shrugging off the mayhem going on as the world burnt down all around it.
Steve typed, “I need a minute to think about this…” in the chat window and then disappeared for a bit. It was like we were in mission control and the Apollo module had just gone behind the dark side of the moon. Would it come back?! Would the people be alive?! I turned to my tech and said, “I’m starting to feel sick. I feel like I’m going to throw up into this sink…”
Finally, we got to that breakthrough moment when systems started popping back on line! Communication was restored throughout the house! After working with us for like eight hours straight, the house was back up and by God online. Lyrical ballads and epic poems were written in Steve’s name. Children were named after him and ships sailed forth to proclaim his might across the land. Angels broke froth in song. I might have hugged my tech.
This experience really convinced me of several things.1).
You absolutely cannot underestimate the power of the Force. Wait. I mean, you can’t underestimate the necessity of a solid AND correctly configured network. With modern systems, if the network is down or kludgey or intermittent, performance will suffer. 2).
You can’t just meddle around in the Big-Boy world of networking. When you start working with real, hardcore, pipe-hittin’ enterprise-grade networking components, you must know what you are doing. We have made a commitment to go through and view all the Pakedge training videos and at least attempt to bring ourselves up to speed on the best practices.3).
When you buy high-end gear, you get high-end performance. With the Pakedge equipment that we purchased, we had all the right tools, just not a full understanding of how to best implement them. I wouldn’t say that we were doing the equivalent of trying to hammer in a nail with a screwdriver (though Steve might…) but we were certainly not employing the systems to their full capacity. Beyond that, the gear gave Steve the tools he needed to make things right. With his expertise, he was able to move from component to component and configure everything for max performance.4).
Most importantly, when you buy high-end gear, you get high-end service and support. This was no call to India where the person on the other end was reading from a pre-built script or trying to blame the problem on something else or wanting to just get us off the line to get to the next call. Steve was a total pro with a really solid understanding on Control4 as well as Pakedge that spent eight hours working on our job. Eight. Hours. He could have easily said, “You guys totally messed this up and it is just beyond my ability to fix your entire system for you.” He could have also groused the whole time about how much we sucked and why didn’t we know what we were doing. But he didn’t. He stayed with us and literally reconfigured everything for us. Explaining what he was doing along the way and keeping in good spirits.
I won’t do another job where networking is this important without using Pakedge because I know that if I run into major drama I can call up and get Steve on the phone and have a SEAL Team 6 Network operator on the other end of the line.
Steve, we owe you the tallest of cold ones. And when I see you at CEDIA, expect a hug.
John Sciacca is principal of Custom Theater and Audio in Myrtle Beach, SC.