I do not understand quite how it was so hard for me to get myself a decent job in IT, to build up my experience to get to where I am today when there is people in the industry, probably earning more than me, doing what they are doing and doing their jobs with the pitiful excuse for knowledge that they have. I am not saying, that I know everything, I definitely don't - some people will know that from some of the SOS messages I post up here for when I need help with stuff. A bit of back ground on me - I am an IT Engineer, predominantly I support small to medium business' who have no IT departments (as they have me) but I have a few clients where they are of a size where they have an IT department, and I support their IT departments and provide them with consultation, strategic help etc. at 15:30 yesterday one of my sites went down, completely down, geographically its 80 miles from me so I can't just nip over and check. I was told that one of the DSL lines had gone down, but also the server was acting funny. Then I was told that BT recognise that there is a fault on the line and they are working to fix it - I kind of left it that I couldn't check out their server until I could dial in - the server was fully roaring, like going full pelt, as I could hear it over the phone. This morning, they still had no DSL, I was on another site where all hell had also broken loose, so it took me 2 hours this morning sorting the site I was on, whilst chatting to my other site trying to at least work out what was going on. I fix my original site this morning, I then spend an hour solidly talking to my guy on site, the head of IT, his junior is off today and tomorrow, I frequently ask "so no one has done anything in any of your comms cupboards - as there is something really not right" - "no, no one has been in any of the cupboards". So I get to the point, where I can no longer handle things getting lost in translation, and I just tell them I am going to go there and sort it out. I get there, the server is roaring away, the internet facing router cpu is maxed out, so I get down to trying to work out what is up with the server, and what is up with the router, I take the router out of the network, and I rig it up to my laptop, at the same time I am rebooting servers, the servers error on boot and say ILO is malfunctioning - at the same time, the router which I have taken out of the network is connecting just fine with only me hooked up with my laptop. odd. at the back of my mind I am starting to get a little nag that I know what is up. So this site, there is a sever room in the basement, there is a couple of switches here, a baseline switch for server etc. and a poe switch for phone system, these are both then connected to the firewall that is connected to the router that I have taken out of the network (so they weren't connected to it, but normally they would be). The baseline switch also is connected to a fibre link that links some floors a couple stories up and another fibre link that runs to the top floor. So I start to integrate the router back into the network, as I know there is no fault on the line as I was browsing BT on my laptop. I put the phone system in, fine. I put the rest of the network in, not fine. the router maxes out again, to the point I can't even get on it, ssh access or anything. So there is something not right. I pull the power to the switch - the second I do this, all the servers shut up and stop roaring....Broadcast storm. I am not a man who likes running up and down stairs, I pulled the fibre links, isolated the network to just the one switch, I integrate this back with the network, fine, normal service. I bounce all the servers, they come back up, fine, internet access, happy days. I power up some machines on the floor, log in as some users, fine, I got drives, an ip, internet access the works. I connect the second floor fibre line, not fine, servers start roaring again, router crashes, bad times. I start again, I omit connected the 2nd floor fibre, I connect the top floor fibre, fine, everything is still happy. So, I know the second floor cabinet is really upsetting everyone. Anyone guess what I found in the second floor cabinet? This is a brand new site, they have just moved there, they didn't have enough patch cables, yesterday, an order for heaps of lovely new patch cables arrived - for some reason, in the junior's infinite wisdom, they felt the need to go and patch 10 patch cables into one switch, looped back, out one port, into the port next to it, 10 times over... WHY? WHY when I say "has anyone done anything?" do I get told no, when the answer is yes. Even when I ask "has anyone even been in any comms cupboards" I get told no. This had a knock on effect, as there is still staff on their legacy site, who connect to this site over a static link, and access the servers on this site - so they couldn't work either as the link was down. So because of one persons actions, two sites couldn't work for 1.5 days. GRRRR. Annoying, I am rushed off my feet this week, I didn't need to loose 3 hours to train journey's to go and fix a problem that didn't even need to be a problem. How does anyone who has a job or any kind of input into the IT decisions of a company not know that plugging a patch lead into the same switch it's coming out of is bad news. especially if you do it 10 fold. That's it I am done, thanks for reading /rant.
Ouch, that is pain! A word needs to be had, both with the junior, his manager and his manager's manager. Make it clear that the Junior has made a bad mistake which cost productivity, but that the manager is at fault for not keeping an eye on a Junior (after all, we can all make mistakes!) Also, time to rig a case open switch to the cupboard doors and onto a headless machine to log when someone's been in and out.
Also a word needs to be had as to why anyone has access to the cabinet who shouldn't have access or doesn't know what they are doing. Edit: Also you'd be surprised (depending on the company) how far an official report into the cause of a network outage goes. Anyway in the end, bill for your time, train tickets. win?
If you want to make the serious money, learn yourself up on all of the ITIL buzzwords and terms and go back in to the same place and demonstrate how proper change control and configuration management would have averted their issue in the first place
I normally don't bother with STP for small business' with only 1-2 switches, but I'm starting to second guess that decision now I think the first thing anyone going into IT or telecom needs to learn is: In a production network, keep your hands in your damn pockets unless you know exactly what you're doing. If I was the manager there, I would give strong consideration to passing along that tip while showing the junior fellow out.
Whilst I see this sort of thing (i.e. this level of utter stupidity) on a regular basis, I have never seen that done. That guy should be fired on the spot, as should the engineers who installed the past 3 network cabinets I've pulled apart, rebuilt and rewired this year. I am truly sorry you had to deal with such blatant stupidity but honestly, it doesn't surprise me. It's not the IT/Telecomms Engineers I have to worry about, it's the idiot employees or MDs thinking they know something about networks because a BT technical help desk worker in India explained to them how to install a HomeHub and connect a laptop to it over the phone one evening, sticking their heads into network cabinets and changing cables around on enterprise level hardware And yes, I'm talking about one site and one person in particular Another thing you have to look out for is a member of staff who's been left in charge for the day calling a different, uncontracted engineer to the site to look at a problem, and that (idiot) engineer removing hardware from your meticulously installed managed network, not bothering to test those items, instead replacing hundreds of pounds worth of equipment then billing the company for said equipment. Then you arrive on site hours after he's left and commence asking questions about why equipment that should still be in the main cabinet are lying on the floor in the server room and why there's a bill for the better part of £500 on the boss's desk. Yes, again I'm talking from experience. I think that guy actually did lose his job over that...
Say you had a roughly $250,000.00 dollar piece of equipment(of the computer/server type) that was overheating. Would you put a bag of ice on top of it? Yup! It happened... Or the time someone destroyed the same type of equipment because they didn't want to wear a sweater at work because the room was too cold...
You can never bee too sure. I've started using red patch cables (with a 'hands off' warning sign) for any mission critical links. A supposedly competent vendor tech disconnected the trunk to the router and moved it to another port so he could plug in a patch cable in that was too short. It was an easy fix when he called me, but he seemed almost clueless as to what vlans or trunk ports were
Red is reserved solely for crossovers in all of my installations, I tend to use blue or yellow to denote mission critical patches and black for regular switch to panel patches. I try my best to avoid grey patch cables in new or "I was installed by an idiot, tear me down and start over" installations. I have no idea why, I suppose I just don't like grey cables...
That kind of thing should get posted on http://www.reddit.com/r/talesfromtechsupport too! All the tech support guys there will give you mega cuddles
This is why I quit general IT services - half of the industry are totally and utterly incompetent. I lost a big client because another company managed to pitch to them while we were in a bit of very awkward downtime that even had Microsoft stumped (never did find out wtf was going on). They told the client all sorts of promises about what they could do, and all sorts of promises about where we had been going wrong. I got so disillusioned by the accusations that I threw the towel in and went along with it, knowing it would implode. Sure enough, within 75 days they had severed the contract with the new company for failing to deliver on their SLA and are now paying through the nose for another company who are only vaguely better.
Times change as do budgets. Most IT departments are grossly understaffed and over half the staff is not qualified. People managing those departments need a shake but so do the budget people. Company I work for recently halved its IT budget. I read an article saying business uptake of windows 8 was poor. Well in our company would never get the budget for a windows 8 upgrade.
We had an issue once, where a technician accidentally kicked a free-standing server next to a rack cabinet. Now, when I say kick, I don't mean a little bump that might have caused a tiny bit of data corruption - no, in this case the kick was so hard, all the hard disks inside were completely shattered. Turning the machine on again produced a horrific grinding and crunching sound (or so I have been told, wasn't actually there). There were recent backups, but this was a critical system. Putting IT departments aside for a moment, I think that IT support from companies can be even worse. Take for example when Virgin Media installed a 100Mbps fibre link to the site. They fitted the fibre and their switch correctly, but forgot to enable autonegotiation. The result was a brand new Internet connection working at only 10Mbps both ways. Bearing in mind that the switch is technically still their property, it should have worked from the start. And don't get me started on RM...
I've actually found BT business/Openreach, who deal with their EFM connections and stuff, to have really good technicians (I refuse to call them engineers, it's an insult to real engineers) that are all properly sound guys. Contrary to the consumer business of course...
They did what? I have, what is technically known as, **** all knowledge about networks of that level and even I know that's utterly retarded
I feel for you. We've very often had this with customers - where you specifically, painstakingly describe something back and forth until you're almost describing individual pixels, to make absolutely sure there's been nothing lost in communication, because what they're telling you just doesn't add up. And then we've gotten to the machine in person and found they were just pants-on-head retarded and thought "up" meant "at the bottom" or "square connector" meant "round connector" or something similarly inexcusable. But that's regular Joe Public, we expect that. Frankly for it to happen inside a business, in the hands of amateur techies specifically entrusted with caretaking IT equipment, is horrific. I'd be making calls and trying my best to get somebody's head to roll for that.
