Own the Network Before it ‘Pwns’ You

Jun 10

Written by: John Sciacca
6/10/2014 7:40 AM  RssIcon

“Pwn is a leetspeak slang term derived from the verb ‘own,’ as meaning to appropriate or to conquer to gain ownership. The term implies domination or humiliation of a rival, used primarily in the internet-based video game culture to taunt an opponent who has just been soundly defeated (e.g., "You just got pwned!").” –Wikipedia

We are in the tail end of completing our Mega Job. All the gear is in, the wiring has been terminated and connected, the programming is done, and we are down to refining and tweaking the system for the homeowner and squashing the little bugs that always creep up on big jobs.

While the job has gone incredibly smoothly for such a massive project, one thing that I did not realize would be such a major headache and source of frustration, aggravation, and time suckage on this job was the managed network. When I initially described our system goals and needs to Pakedge prior to ordering, we were told that a project our size must include a managed network and that it just wouldn’t work without it (or, more correctly, it would crawl, lag, and crash.) With 112 hardwired Ethernet ports, 21,500 square feet of multiple Wi-Fi access points and so many different systems talking back and forth, a managed network would allow us to separate traffic onto different VLANs.

The goal, of course, was to keep the data traffic from critical devices like Kaleidescape, Control4, and Lutron flowing fast and smooth, top down in a network-surfing Ferrari, while non-critical things like guests watching cat videos chugged along on a different road in a VW minibus enjoying the scenery at a lower priority. And, hey, if managed is better and was going to make the project perform like a rock star, then brother, manage me up! Money is no object. Let’s do it! #Managed4Life #Blessed

To be fair, we had never had the need to use a managed network on a project before, so we weren’t aware of the complexity, configuration, and issues that come with a managed network—IGMP Snooping, spanning tree protocol, port flow control, link aggregation… No idea. And while a managed network sounds like a great idea in theory, in practice in the resi world, it has been excruciating, and at this point I don’t know that I would ever do another project using managed switches.

I do want to give props upfront to the amazing tech-support team at Pakedge. I wrote a lengthy blog on the first bout of help we received from Network Jedi, Steve who literally spent eight-plus hours on the phone and Team Viewer with us to get our network’s management settings off on the right track. Steve was patient, knowledgeable, and non-judgmental as he worked through our initial configuration issues. And when I left the house that day/night, I felt like I had been through a lot of growing pain, but it would ultimately be worth it, and now my network would be like a caterpillar transformed into a beautiful butterfly.

However, since that phone call my techs and I have spent more time on the phone and Team Viewer with Pakedge. Lots and lots more time… And while their technical support is always patient, helpful, and awesome, it is still hours of time sitting there watching the screen as they make log into all of the different components and make adjustments and configuration changes to the network that I don’t understand. And instead of working on other items on the project, I’m forced to just sit there with fingers crossed hoping each time that the next leap will be the leap home.

The problem I am running into now is, I frequently don’t know or even have a way of knowing if the problem is A) a broken piece of Pakedge gear B) a bad management configuration in one of the many Pakedge managed devices C) a broken piece of Control4 gear D) a bad programming binding in the Control4 system E) some random curse from the fickle whim of the cruel Internet gods. Something just won’t work, and I’m left scratching my head wondering where to start the troubleshooting. And the house is SO huge that every time I need to check something, it requires a lengthy walk down long hallways and into a closet in some room where I have to pull out a rack, probe around with a flashlight, unplug and reboot something, wait for it to come back online, then test to see if the change fixed the problem, typically a process that takes 10 or more minutes.

And a Control4 processor and Pakedge managed switch aren’t like a Blu-ray player that I can just “throw in a new one real quick” to test. Or even just plug into a different switch port in many cases. Or move a non-working one to a working area.

And you know the Butterfly Effect, the theory that a butterfly flapping its wings on the other side of the world can cause some disaster locally? That’s starting to feel how things are when we make a change somewhere in the network. A priority tweak as it cascades from the router to one switch, to the next switch, to the next, and an adjustment at one side of the house might cause a crash in an unrelated system somewhere else. And with 112 hardwired Ethernet ports, and eight separate systems it can take days before anyone notices that the change “broke” something that had previously been working.

Last week I went up to the job for a simple, “Why are the lights in the theater not turning off?” service call (turned out the new Sony VPL-VW1100ES 4K projector was continuing to send “I’m off” IP commands that triggered the Control4 system to tell the Lutron system to issue the “End Movie” lighting scene) that turned into another eight-hour network, “Where’s Waldo?” hunt down as I tried to figure out why music wouldn’t stream to two of five rooms that appeared to be identically configured. Ultimately we (and I’m using the royal “we” here as it was really all Pakedge, with me offering ocular support and running around the house confirming which ports on the switches certain pieces of gear were connected to) ended up undoing a VLAN setting that we had spent hours doing the week before.

Further compounding the problem is that companies and their tech support departments aren’t designed to troubleshoot with each other. There is no quick and easy way to get Control4 on the phone with Pakedge and me to three-way out this issue—all of us looking at the same project and working together off the same page. Control4 sent me a “Technical Support Bulletin” (now almost three years old), titled “Settings on Managed Switches That Must be Disabled for a Control4 System to Operate” that started with the ominous phrase, “Using a managed network switch (not recommended or supported)…” which I shared with Pakedge. But after making some of the “dumbing down” changes suggested by the bulleting, it actually seemed to make the problems worse!

(Sigh…) (Gulp of scotch…)

As a company we realize the network has become the most crucial component of the modern systems we install, and we can no longer rely on DHCP and correctly terminated RJ45 jacks to resolve our problems. We’ve decided to take a proactive step and send someone to CEDIA’s three-day networking school and take advantage of the manufacturer trainings and face-to-face time at CEDIA Expo. This training, coupled with the support of companies like Pakedge, should have us better prepared to tackle the needs of the next Mega Job.

John Sciacca is principal of Custom Theater and Audio in Myrtle Beach, SC.
 

5 comment(s) so far...


Gravatar

Re: Own the Network Before it ‘Pwns’ You

Wow. Every issue here would be irrelevant with a WhyReboot network where we WOULD call the manufacturer for you if it ever came to that. Not to mention if you use enterprise gear you have more visibility into the system which allows for much better troubleshooting and visibility. That, and no need to tie up a computer with Teamviewer because with VPN you could have direct access to the local network gear. This sounded like a nightmare but I'm glad it seems you got ahold of it in the end! Definitely take the CEDIA networking course and if you or your team have any questions after please don't hesitate to contact myself or any of the other guys working on the courses.

By Bjørn Jensen on   6/10/2014 1:07 PM
Gravatar

Re: Own the Network Before it ‘Pwns’ You

John I didn't know you ran into issues with managed solutions. There are many proactive solutions to your problems. 1) Pakedge can store configuration files for each device so every install has the same setup/config and can even pre-load this prior to shipping. 2) We can get you and your techs together to do a one-on-one training to better educate and prepare you for such installations. Lets set this up when you have time. 3) We do system design on the sales side, you give us the scale of the project and we can recommend the proper devices. Not all projects call for VLANs, it is case by case as the homes vary. Unmanaged, Managed, and even VLAN configured switches call for different installs based on whats going into the job. 4) The NP36 monitoring device will be huge in these homes, Pakedge is completely connected in so many ways when this is added to a network, making configurations, power cycling PoE/AC ports among other things easier. It also helps create a SSL connection to the router without Dyndns or port forwarding. Even beyond that you can dial into gear with a web interface (Printers, Switches, Routers.. etc) using the device portal option on the NP36. I think you and you're company are headed in the right direction and with the proper guidance none of these headaches will be a problem. The most important thing beyond us getting you to a standardized way to roll things out is to educate and support you.

Please let me know anytime you need something as you have my direct contact information now.

By Jason on   6/10/2014 8:58 PM
Gravatar

Re: Own the Network Before it ‘Pwns’ You

Jason,
Thanks for the offers of support! I'll definitely take you up on the one-on-one training.

Also, please don't misconstrue my frustration with the managed switches as a dissatisfaction with Pakedge. If anything, it has shown me how critical terrific back-end tech support is, especially when dealing with a complex network!

Hopefully what we learn from this job will only make the next one go that much smoother!
John

By John Sciacca on   6/11/2014 6:37 AM
Gravatar

Re: Own the Network Before it ‘Pwns’ You

Hi John,

Very glad to hear you're interested in sending someone to participate in our networking course here at CEDIA Headquarters. However, I assume you mean to send them to Indianapolis and not London, in which case as the instructor, I wanted to provide a few points of clarity. The course is actually 5 days long and includes information on integration/automation as well as networking and the link you should be using is cedia.net/train/in-person-training/boot-camps/networking-boot-camp rather than the one in your blog as it refers to a course we offer through our offices in the UK.

If you have any additional questions or need further assistance feel free to email me at srissi@cedia.org

Cheers,
Steven Rissi

By Steven Rissi on   6/13/2014 9:50 AM
Gravatar

Re: Own the Network Before it ‘Pwns’ You

Steven,
You're right! Though I'm sure he would LOVE to go to London instead, it will be the week-long training in Indy!
Thanks for the clarification!
John

By John Sciacca on   6/13/2014 11:03 AM

Your name:
Gravatar Preview
Your email:
(Optional) Email used only to show Gravatar.
Your website:
Title:
Comment:
Security Code
CAPTCHA image
Enter the code shown above in the box below
Add Comment   Cancel 

 

Archive
<July 2014>
SunMonTueWedThuFriSat
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789
Monthly
July, 2014 (15)
June, 2014 (12)
May, 2014 (12)
April, 2014 (14)
March, 2014 (15)
February, 2014 (14)
January, 2014 (24)
December, 2013 (11)
November, 2013 (12)
October, 2013 (15)
September, 2013 (19)
August, 2013 (18)
July, 2013 (19)
June, 2013 (12)
May, 2013 (18)
April, 2013 (17)
March, 2013 (13)
February, 2013 (16)
January, 2013 (26)
December, 2012 (9)
November, 2012 (11)
October, 2012 (16)
September, 2012 (11)
August, 2012 (15)
July, 2012 (13)
June, 2012 (8)
May, 2012 (9)
April, 2012 (10)
March, 2012 (7)
February, 2012 (11)
January, 2012 (14)
December, 2011 (5)
November, 2011 (12)
October, 2011 (8)
September, 2011 (4)
August, 2011 (7)
July, 2011 (11)
June, 2011 (12)
May, 2011 (8)
April, 2011 (6)
March, 2011 (9)
February, 2011 (10)
January, 2011 (9)
December, 2010 (4)
November, 2010 (7)
October, 2010 (4)
September, 2010 (9)
August, 2010 (7)
July, 2010 (8)
June, 2010 (12)
May, 2010 (6)
April, 2010 (8)
March, 2010 (10)
February, 2010 (11)
January, 2010 (1)
December, 2009 (6)
November, 2009 (6)
October, 2009 (11)
September, 2009 (6)
August, 2009 (4)
July, 2009 (4)
June, 2009 (5)
May, 2009 (7)
April, 2009 (9)
March, 2009 (7)
February, 2009 (9)
January, 2009 (8)
December, 2008 (9)
November, 2008 (6)
October, 2008 (7)
September, 2008 (7)
August, 2008 (3)
July, 2008 (5)
May, 2008 (1)
April, 2008 (4)
March, 2008 (3)
February, 2008 (5)
January, 2008 (2)
November, 2007 (1)
October, 2007 (4)
September, 2007 (2)
August, 2007 (3)
July, 2007 (4)
June, 2007 (6)
May, 2007 (6)
April, 2007 (4)
March, 2007 (4)

 

Skip Navigation Links.
Skip Navigation Links.
download

My Special Project: Immersive Video in a $250,000 Tractor
Read More»
By Heather L. Sidorowicz

What Net Neutrality Means for AV Integrators
Read More»
By Lindsey Adler

6 Reasons Why the Dedicated Remote Will Never Die
Read More»
By Todd Anthony Puma

The Pros and Cons of Owning a CI Showroom
Read More»
By John Sciacca

The Internet of Things: Tech Support Concerns in Today’s Connected Home
Read More»
By Patrice Samuels


CEDIA Awards Ceremony 2012
Wisdom Audio Factory Tour
Unexpected Atlanta
Triad DesignerSeries Step-by-Step Install
Top 10 Things to See at CEDIA 2010

+ All Slide Shows
Re: Sales Representative Wanted
Read More»
Re: Was CES time well spent?
Read More»
Re: Was CES time well spent?
Read More»
Re: Would You Miss Satellite Radio Too?
Read More»
Re: What is Your Recurring Revenue Stream?
Read More»
Re: Was Utz Baldwin the right choice for CEDIA CEO?
Read More»
Re: Would a DTV Delay Matter to You?
Read More»
Re: FIFA 15 Player Ratings: FUT 14 ps4 coins site where you can pay by moneybooker
Read More»
Re: As I started off out Buy FIFA 14 Coins playing it just emerged to me
Read More»
As I started off out Buy FIFA 14 Coins playing it just emerged to me
Read More»