The Day Pakedge Steve Became My Network Tech Support Jedi Master

Mar 25

Written by: John Sciacca
3/25/2014 9:46 AM  RssIcon

We have been steadily plugging along at our mega install job, getting all the gear racked in, terminating all the wiring, installing all the touchpanels, hanging the projector and screen, and programming and testing all the systems and subsystems. All told, we have 109 Gigabit networking ports on this project.

Because the entire home is going to depend so completely on the network being robust and up 100 percent of the time, I knew this wasn’t something we could screw around with using off-the-shelf routers and switches. Between the Control4 processors and touchscreens, the Kaleidescape movie server, the Lutron HomeWorks lighting system—and its 350 loads and 30 drapes—and everything else, if the network were to go down on this job, it would be a system destroying disaster where literally nothing would work. (Pretty much my current reoccurring nightmare at this point.)

For that reason, I went with Pakedge for the networking backbone. To be honest, prior to this job I’d really only known Pakedge by reputation, but everything I’d heard about that reputation made me feel confident that they were the right solution. Also, their prior PR manager, the ever-sparkly Olivia D, sent me a Pak wireless access point to experiment with in my own home, and I was thoroughly impressed with its quality, performance, and 100-percent uptime.

For this job, we went with a K6 router, 24-port managed PoE switch (SW24P), 24-port managed non-PoE switch (SW24GBM), seven 8-port managed switches in racks around the home (S8Mpd), five dual-band wireless access points (4 W7 and 1 W7O for outside), a C36 WAP controller, and the P8 power manager. (I have grown to love this P8, for its ability to bring the system back on line in the correct order in the event of a sudden power failure. Again, hoping to ward off the reality of the “nothing works!" nightmare.)

I honestly didn’t know a ton about managed switches and all of the configuration and optimization possibilities, but both our reps and Pakedge had assured us over and over prior to the sale that they would be there to hold our hands and give us all of the support we needed to make sure this thing was configured and running perfectly. And prior to starting the networking configuration on this project, the part I was most concerned with was just the correct terminating of all those many, many Cat5/6 wires. I figured once we had that done correctly it would be mostly a matter of plugging them into ports on the Pakedge gear and then standing back and watching the magic happen.

Our most computer-savvy tech is the closest thing our company has to an “IT professional,” and he handled the majority of the network configuration, setting up a VLAN and subnets so the Control4, Lutron, and Kaleidescape systems could have top priority and get out of the muck-and-mire of living down in the Mos Eisley spaceport with the rest of the house’s networking scum and villainy.

Things ran mostly pretty smooth with the network, but we noticed that randomly throughout the day the system would seem to just lock us out. For like these random 10-minute periods, we couldn’t access the internet, we couldn’t get into the router, we’d lose connection to our Control4 processor. At first we chalked it up to just ghosts in the machine, or the slow internet service, or the fact that we were hammering away at the programming, but after a particularly long lockout where we weren’t taxing the system at all, we finally called up Pakedge for a, “Yo! What’s going on here?! Is something broken?!” love in.

Fairly early in the conversation, we mentioned “Sonos” and it was like we just read a particularly nasty incantation from the Necronomicon. The Pak tech immediately said that was definitely our problem and that the system had to be specially configured to handle the constant network chatter and blathering that the Sonos spouted forth. He walked my tech through some setting changes in the Pakedge menus with names like IGMP Snooping and Spanning Tree Protocol. Click-click, and the problems went away like magic. (Though, I can’t understand why a garbage, off-the-shelf router from Wal-Mart handles Sonos fine, while a $1500 enterprise-grade Pakedge switch would get tripped up by it. The only thing I can guess is that when given free rein on a managed switch that is NOT being managed, the Sonos goes all Augustus Gloop at the chocolate factory, eating up all the food and splashing around in the chocolate river and making all sorts of network ruckus and mayhem. Someone feel free to enlighten me in the comments.)

A few days later we were troubleshooting an issue where we were unable to remotely access the Control4 processor. We talked to Control4 and pretty quickly determined things looked OK on their end, and they directed us to call Pakedge for some help in configuration. This was around 11 or so on a Friday morning. So we called Pakedge, and this is where Steve, the Steveinator, network Jedi, enters the picture. What we think is going to just be a simple, “Hey, Steve, can you clicky-fix our remote access issues? Toodles!” call turns out to NOT be that. At all.

Steve took over the computer and started poking around and looking at settings. Then he asked some gentle, probing questions like, “What are you trying to do?” and “Who set this up?” and “Who is the IT expert on your staff?” Fairly quickly it became obvious that somewhere along the line of our initial setup and configuration we veered off the path of “the right way” and have instead plowed miles ahead, and now we were deep—DEEP!—in the murky, scary, backwoods, uh-oh-what's-that-shack-for?! part of “totally messed up.” And it was one of those mess-ups like when you miss a crucial plot point in a linear video game where even though you’ve been playing for an extra 1,000 hours, you need to go back and totally restart and redo that one simple thing you missed.

I’m not sure that you can actually hear someone take a deep, cleansing breath through the emotionless box of a chat window, but I picture that’s what Steve did. I also imagine him taking a super cool, Don Draper drag on a cigarette, blowing a long, lazy stream of blue-grey smoke up into a slowly twirling ceiling fan, powering down a stiff, three fingers of his favorite single malt, cracking his knuckles and saying, “OK. Let’s do this thing.”

What he actually said—in the nicest way possible—was that we had so badly messed up configuring this network that it would be easier if we just factory defaulted everything and completely started over from ground zero. I tried to keep it light and breezy by asking him just where we had gone so wrong and he replied, “You want the whole list? :-)” Ouchey.

I told him that we had to do whatever it took, but that if he burned our network house down, he would have to stay with us until we rebuilt it; we couldn’t leave the project in complete ruins, with absolutely nothing working. He asked me how long I was planning on staying and I told him, “As long as it takes. I’m here for the duration.” (I then texted Dana and told her I wouldn’t be making it home for dinner. And maybe not ever again. It was too early to tell.) Steve assured me that with enough time, anything was possible, and that he could make it right.

So, Steve set about rebuilding our network using Pakedge best practices. From his top-secret Steve Lair in the middle of Network Super Computer Land, Steve moved the mouse around, pointing, clicking, changing, rapidly circling things to show us something important, all the while rebuilding and reconfiguring.

When I walked into that house at 9:00 that morning, nearly 95 percent of everything was done and working. I thought I had a day of tweaking and fine-tuning ahead of me. Yet, several minutes into the Steve Marines Corps style, “I’m gonna tear your down, rip you apart, then rebuild you, make you stronger, better, more of a man than you’ve ever been!” network rebuild, nothing in the house worked. For several hours… At one point I’d been just sitting in a chair watching Steve move the mouse around with nothing working on our end for about three hours.

Undaunted, the Steveinator kept plowing ahead. There were quite a few moments where he would change something, ask us to try it, change something else, and ask us to try it again. After several of these attempts, we couldn’t establish even basic communication between the main rack and the sub-rack. And then we lost the ability to see any of the Control4 components on the network. Then the main processor disappeared and wouldn’t come back. Amazingly, the Kaleidescape system remained rock stable throughout all of this, continuing to stream movies, plowing through the network traffic, and totally shrugging off the mayhem going on as the world burnt down all around it.

Steve typed, “I need a minute to think about this…” in the chat window and then disappeared for a bit. It was like we were in mission control and the Apollo module had just gone behind the dark side of the moon. Would it come back?! Would the people be alive?! I turned to my tech and said, “I’m starting to feel sick. I feel like I’m going to throw up into this sink…”

Finally, we got to that breakthrough moment when systems started popping back on line! Communication was restored throughout the house! After working with us for like eight hours straight, the house was back up and by God online. Lyrical ballads and epic poems were written in Steve’s name. Children were named after him and ships sailed forth to proclaim his might across the land. Angels broke froth in song. I might have hugged my tech.

This experience really convinced me of several things.

1). You absolutely cannot underestimate the power of the Force. Wait. I mean, you can’t underestimate the necessity of a solid AND correctly configured network. With modern systems, if the network is down or kludgey or intermittent, performance will suffer.

2). You can’t just meddle around in the Big-Boy world of networking. When you start working with real, hardcore, pipe-hittin’ enterprise-grade networking components, you must know what you are doing. We have made a commitment to go through and view all the Pakedge training videos and at least attempt to bring ourselves up to speed on the best practices.

3). When you buy high-end gear, you get high-end performance. With the Pakedge equipment that we purchased, we had all the right tools, just not a full understanding of how to best implement them. I wouldn’t say that we were doing the equivalent of trying to hammer in a nail with a screwdriver (though Steve might…) but we were certainly not employing the systems to their full capacity. Beyond that, the gear gave Steve the tools he needed to make things right. With his expertise, he was able to move from component to component and configure everything for max performance.

4). Most importantly, when you buy high-end gear, you get high-end service and support. This was no call to India where the person on the other end was reading from a pre-built script or trying to blame the problem on something else or wanting to just get us off the line to get to the next call. Steve was a total pro with a really solid understanding on Control4 as well as Pakedge that spent eight hours working on our job. Eight. Hours. He could have easily said, “You guys totally messed this up and it is just beyond my ability to fix your entire system for you.” He could have also groused the whole time about how much we sucked and why didn’t we know what we were doing. But he didn’t. He stayed with us and literally reconfigured everything for us. Explaining what he was doing along the way and keeping in good spirits.

I won’t do another job where networking is this important without using Pakedge because I know that if I run into major drama I can call up and get Steve on the phone and have a SEAL Team 6 Network operator on the other end of the line.

Steve, we owe you the tallest of cold ones. And when I see you at CEDIA, expect a hug. John Sciacca
 
 
 
 
John Sciacca is principal of Custom Theater and Audio in Myrtle Beach, SC.
 
 

11 comment(s) so far...


Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

This is all well and good - I'm overjoyed you were able to get through such a hard ordeal with a well qualified person on the other end of the line. We were one of their early adopters and even tipped off distributors to the quality of product they produced.

Then, things began failing. Access points would just die, power supplies failed routinely, etc. Another customer needed a bunch of static IP's setup. Within their router web GUI setup, I needed to login to a console and issue a long-winded console command for a simple static IP setting. When I couldn't get it to work right (after reading documentation for their CLI console), I called into their tech support - and they took about 25 minutes to set a static IP for ONE device. I think I was put on hold 4 times.

Service reigns supreme, when you are contracted for a jobsite which is detailed on Residential Systems, I guess...

By Dan on   3/25/2014 7:17 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

Great article, John. I always enjoy your writing style - your references to pop culture, both old and new - always make your articles entertaining as well as informative.

By Steve St. Agathe on   3/25/2014 11:09 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

sounds like a lot of drama for a relatively small network. Then the length of time it took to resolve it... wowie.
I'm torn about pakedge. super pricey. while they have a nice complete line that fits the custom integrators needs well... nothing makes their stuff so superior that it justifies the cost. ubiquiti stuff is super awesome and super inexpensive.

By g rod on   3/26/2014 7:24 AM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

Hey, Dan. I'm sorry to hear that you had a poor experience with Pakedge. So far we have had one device -- an 8port switch -- that stopped working when powered via PoE. (It continued to power via local power supply.) Pak immediately sent us a new one and covered shipping both ways. We set up static IP addresses for virtually every device on the network -- over 80 for sure -- and had none of the problems you mentioned.

I would like to dispel the suggestion that we received any special or preferential treatment. It was my tech -- not myself -- that initiated the call to Pakedge's support line. We just randomly got a tech -- Steve -- on the line and he set about helping us. When I want "special" treatment, I will generally go through PR channels and ask them to set up a support session. In this case, we just called-in cold like any other dealer. Also, I've received several Tweets and messages from other dealers echoing the same experiences that I had, so I don't feel this was necessarily beyond the norm for what they provide.

g rod: I think it took so long to resolve it because we had actually gone in and (accidentally) made settings and configurations that exacerbated the problem. The tech had to undo a lot of our things before being able to go forward. Had we just had a fresh out-of-box system, it would have been much quicker. I don't disagree with the price; it is expensive. However, I feel that the backend support (that we desperately needed) was a case of we got what we paid for.

Thanks for reading,
John

By John Sciacca on   3/26/2014 10:16 AM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

hi John; it is a welcome message to hear of such great support. It makes the greatest impression and builds confidence in a product line. Most AV firms are not up to what I call proficiency, we all can learn and sometimes need guidence from the product manufacture, especially in the IT products. What bites me most is that there is not one place that we can go to learn the "electron" if you will, level of code and handling of code by the IT products we use. Add to that products like SONOS, which you mentioned that need special attention because of their nature.

By Bob Lieto on   3/26/2014 12:04 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

John,

I have been following your blogs for some time and find them informative and entertaining.

In an attempt to repay for the information and entertainment you have provided I am going to make some recommendations.

First, let me give some background so you know where I am coming from. My career is in IT as a network engineer (wired & wireless)but my passion is AV and automation. This gives me a position of understanding from the enterprise network side of things how AV and automation systems play in this environment. Knowing this here are my recommendations for you;

1. Either hire a network engineer who has an interest in learning AV and automation to supplement your staff or start your current staff on a track of networking certifications.
2. If you choose the path of training and certifications for your staff look into Cisco CCENT as a base with CCNA as the goal to get at least one staff member to. This will cover you on the wired network. If wireless is something you deploy often in the homes then I would also include CWTS from the CWNP group to build on top of the CCENT and/or CCNA.
3. If you choose to hire an engineer make sure they have at minimum CCENT and are in the process of getting their CCNA.
4. Stay with deploying a product like Pakedge that has been purpose built for what you are doing. It will save you and your business many lumps and bruises and damaged rep. (no I don't secretly work for Pakedge!)

With more and more systems going the IP route it is going to be critical that you and your staff have a good foundation of core networking knowledge and expertise.

Good luck to you and keep up the great blogging.

By Michael on   3/28/2014 1:48 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

Thanks, Michael! Jeremy -- Editor of Resi Systems -- asked me the same question..."Why don't you hire a certified network engineer or IT guy?!" The truth is, like many CI firms, we are a small company; including myself we have 5 employees. We also have very few times where we run up against something of this level where our "in house" knowledge doesn't suffice. Also, I need my techs to wear many hats, and I just don't have enough steady work -- or the room on the payroll -- to have a dedicated network engineer. Further, the hiring pool in my area is pretty limited; we don't have a lot of network engineers with an interest in learning AV and automation just lurking about. (If you are one of those people and you happen to be reading this, please to contact me!) I definitely think that we -- as a company and an industry -- do need to be more proactive in seeking out the industry trainings that are available to us, both on line and at tradeshows like CEDIA. Working with company's like Pakedge that have great products and support and training systems is certainly a boon. Also, I gladly accept "repayment" in beer. :-)Cheers,John

By John Sciacca on   3/28/2014 2:06 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

Michael,
It be nice to contact you from time to time related to the IT part of this business. Drop me an email (jglowacki@nbmedia.com) if you'd be willing.
Thanks, Jeremy (editor of Residential Systems)

By Jeremy Glowacki on   3/28/2014 2:11 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

John,

I am used to seeing your articles in other publications, so I was interested in reading this particular one, especially because of Pakedge. I have a large home in which I had installed CAT 7A Network Cables so that My Home Network would remain SOLID & be able to GROW with the future upgrades of Networking to 40 G bits, 100 G bits & beyond. I too have many devices on my Network that broadcast all their junk that screws-up normal traffic. My Installers, Syzygy, Inc., also went to Pakedge for their SMART NETWORKING GEAR. In particular, my 5 Apple TVs were the blame for most of my problems, before I switched to Pakedge gear, from my normal Cisco stuff. I also have 4 Kaleidescape Players, 1 of which is a Player / Reader. Plus, 5 DirecTV Satellite Receivers, 2 of which are the 5-Channel sweet ones, plus Integra Pre-Amps, SAVANT Remotes, SAVANT iPads, SAVANT IP Phones, various iPhones, Laptops, iPads & other hard-wired Computers, all on my Network. So, needless to say, I have many devices that do need to be Managed. Originally, I had all me Network run by my Apple Airport Extreme, which worked great, until Savant entered my Home Automation picture. Then, all of its IP devices, iPads, Remotes, Phones, etc., started to get interference from my Apple TVs, which apparently broadcast all the time. So, Pakedge was the solution. They divided My Network up into Layers, all controlled by Pakedge, and now, for the most part, everything seems to be operating OK. I still have some dropped streaming problems when watching The Blaze & some delayed responses from some SAVANT Remote Commands; but, at least now my IP Telephones don't echo & delay. So, was the extra $30,000.00 really worth it, for all the Pakedge gear? To me, not really. Instead, I would recommend deleting the SAVANT IP Telephones & changing them all to Panasonic ones, which, of course, are not compatible with My Whole Home SAVANT Automation System & then I would just live with any SAVANT Remote Command Response Delays because they are still there anyway. Also, my network would be much simpler to maintain because I would only have to use the Apple Airport Utility instead of having to live with Pakedge components that I cannot access or understand how to maintain. Instead, now I have to call on my Installers to do anything with the Pakedge gear. So, until Apple adds the layering network feature to their Airport Extremes, which do not currently have that capability, I will stick with my double-edged-sword Pakedge gear. In this case, I did get what I paid for, for now............

By William F. Jakobi on   3/29/2014 5:44 PM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

Guys,

One of the reasons you are having to install enterprise grade networking is because of the protocols (Bonjour, mDNS, DLNA, UPnP) some of the AV and automation gear chooses to communicate on you network. Here is an article I wrote that describes why these protocols have difficulty scaling and the challenges they present on campus and large enterprise networks.

www.securedgenetworks.com/secure-edge-networks-blog/bid/83696/Problems-with-AirPrint-and-AirPlay-on-school-wireless-networks

Recently Apple released, quietly, an update (6.1) to Apple TV's that provides a process for discovery via Bluetooth. This means that if you are in the proximity of Bluetooth discovery (30-50ft) of the Apple TV you will "see" it and learn it's IP. Then when you attempt to mirror to it your unicast transmission can traverse across layer 3 segments (VLANs)where the discovery broadcasts were originally blocked. This release solves one issue but does not solve all of them. It is my understanding that Apple is in the process of finding a solution that would work more efficiently on large networks and minimize the impact on all networks.

Regardless of the environment the network has become the platform that all devices, appliances, etc... deliver their services on. Until we get to the point that SDN (software delivered networking)is smart enough and cheap enough to deliver to the home consumer it is important to become really comfortable and knowledgeable about IP networking.

By Michael on   3/31/2014 8:40 AM
Gravatar

Re: The Day Pakedge Steve Became My Network Tech Support Jedi Master

Adding a comment regarding the SONOS portion of the article:

To answer the question as to why SONOS works on a cheap "Walmart" router, low-end devices do not have STP support, so they pass traffic through without touching it. They aren't compartmentalizing Sonos traffic. High-end devices that support STP need to be configured to handle the traffic correctly.

A typical multi-zone SONOS system install comprises of a SONOS bridge and several zone players. The bridge is typically connected to the router directly, and in this way, the SONOS system acts as its own mini network apart without having to interact with the devices connected to the other router ports (directly or indirectly). In order for the SONOS mini network to work properly and prevent network loops between the bridge and the zone players, SONOS implements STP (spanning tree protocol).

In larger networks with multiple devices and branches, a SONOS system is typically connected directly to the switch instead of the router. In this case, the switch port that the SONOS is connected to must have STP enabled in order to work properly otherwise the Sonos will create a network storm that essentially disables the network. With STP at the switch port disabled, the switch will block the SONOS STP BPDU packets. When the zone players are not able to see BPDUs, they cannot detect there is a shared network between the zones and will flood the network. In most managed switches, STP is disabled by default. In order to have SONOS work properly, STP must be configured such that it is enabled for the switch port that SONOS is connected to.

You can read more about Sonos and STP here:
forums.sonos.com/showthread.php?t=16973

By Pakedge Device & Software, Inc. on   4/1/2014 2:00 PM

Your name:
Gravatar Preview
Your email:
(Optional) Email used only to show Gravatar.
Your website:
Title:
Comment:
Security Code
CAPTCHA image
Enter the code shown above in the box below
Add Comment   Cancel 

 

Archive
<October 2014>
SunMonTueWedThuFriSat
2829301234
567891011
12131415161718
19202122232425
2627282930311
2345678
Monthly
October, 2014 (16)
September, 2014 (21)
August, 2014 (13)
July, 2014 (15)
June, 2014 (12)
May, 2014 (12)
April, 2014 (14)
March, 2014 (15)
February, 2014 (14)
January, 2014 (24)
December, 2013 (11)
November, 2013 (12)
October, 2013 (15)
September, 2013 (19)
August, 2013 (18)
July, 2013 (19)
June, 2013 (12)
May, 2013 (18)
April, 2013 (17)
March, 2013 (13)
February, 2013 (16)
January, 2013 (26)
December, 2012 (9)
November, 2012 (11)
October, 2012 (16)
September, 2012 (11)
August, 2012 (15)
July, 2012 (13)
June, 2012 (8)
May, 2012 (9)
April, 2012 (10)
March, 2012 (7)
February, 2012 (11)
January, 2012 (14)
December, 2011 (5)
November, 2011 (12)
October, 2011 (8)
September, 2011 (4)
August, 2011 (7)
July, 2011 (11)
June, 2011 (12)
May, 2011 (8)
April, 2011 (6)
March, 2011 (9)
February, 2011 (10)
January, 2011 (9)
December, 2010 (4)
November, 2010 (7)
October, 2010 (4)
September, 2010 (9)
August, 2010 (7)
July, 2010 (8)
June, 2010 (12)
May, 2010 (6)
April, 2010 (8)
March, 2010 (10)
February, 2010 (11)
January, 2010 (1)
December, 2009 (6)
November, 2009 (6)
October, 2009 (11)
September, 2009 (6)
August, 2009 (4)
July, 2009 (4)
June, 2009 (5)
May, 2009 (7)
April, 2009 (9)
March, 2009 (7)
February, 2009 (9)
January, 2009 (8)
December, 2008 (9)
November, 2008 (6)
October, 2008 (7)
September, 2008 (7)
August, 2008 (3)
July, 2008 (5)
May, 2008 (1)
April, 2008 (4)
March, 2008 (3)
February, 2008 (5)
January, 2008 (2)
November, 2007 (1)
October, 2007 (4)
September, 2007 (2)
August, 2007 (3)
July, 2007 (4)
June, 2007 (6)
May, 2007 (6)
April, 2007 (4)
March, 2007 (4)

 

Skip Navigation Links.
Skip Navigation Links.
download

Don’t Let Little White Lies Become Big Dark Lies
Read More»
By Todd Anthony Puma

To Defy Moore's Law of Electronics
Read More»
By John Sciacca

Automation, as We Know it, is Changing
Read More»
By Heather L. Sidorowicz

Reach Higher Value Shoppers with Google’s RLSAs
Read More»
By Brendan Morrissey

NY Realtor Sees Re-Sale Value in Dedicated Home Theaters
Read More»
By Jeremy J. Glowacki

Slide Show
Slide Show

Audio Video Systems, Plainview, NY
Slide Show

Station Earth, Fergus, Ontario, Canada
Slide Show

Audio Command Systems, Westbury, NY
Slide Show

Evolution Video and Sound, Calgary, Alberta, Canada
seize the time to get free $10 Free Cash Vouchers for swtor credit at swtor2credits facebook
Read More»
l is enough." Al
Read More»
, as is the case with Ri
Read More»
, as the young hurler will try t
Read More»
m 2007-2010. The staff
Read More»
staff throughout the entire season, prin
Read More»
of the cab that got hit and took the brunt of the
Read More»
d eight assists while shooting 6 for 14
Read More»
an Patrick Kabongo anno
Read More»
-0 in Quebec Major Junior Hocke
Read More»