185

I’m relatively new at a job, but have established myself as the “guy” when it comes to a certain web service we use. Our own website integrates with the service. I don’t really work on the site, just with the service.

Today we had something of a show stopping issue that resulted in users of the site not being able to use the service. Naturally I was called in to help, but one of the site developers got snippy with me personally, insinuating I wasn’t doing my job and accusing me of poor communication. Later on in a conference call with at least 8 other people, he passive-aggressively called me out on not having resolved the issue.

After a long day, working late and from home, I finally determined the root of the problem to not be in the service, but in the code on the site and specifically in code that the snippy developer wrote.

TL;DR: the guy who was pointing his finger at me was in fact the cause of the problem

I’m generally an easy going person and have no need to tell this guy off; however, I’m hoping for some suggestions on how to tactfully express to him that I’d prefer he was 100% absolutely sure that it’s not his code before he takes that tone with me again.

Community
  • 1
Ryan
  • 1,411
  • 2
  • 9
  • 5

11 Answers11

377

Just send an email to everybody involved saying that, after hours, you found (and fixed) the bug. "It is so and so in AB." Maybe with a link to something. Don't mention any person.

That should be enough. You don't have to point a finger at anybody.

The others will look into this and soon they will find out who was/is responsible for that bug, and next time they will think twice before they mess with you.

Southpaw Hare
  • 1,237
  • 1
  • 15
  • 21
Edgar
  • 4,317
  • 2
  • 13
  • 22
  • 47
    I'd do this too assuming this was a once-off incident. If it is part of a pattern then defensive escalation might be warranted. For now I'd just keep the notes of what happened ready so that if this happens a few times I can compile a fairly damning report of just how much time you're spending cleaning up this guys mess. – Tim B Mar 09 '18 at 09:46
  • 153
    Agree with this. Be seen as the guy who gets s**t fixed and let others play the blame game if they want - send a neutral email explaining where the problem was and those who know the system can figure out who caused it, you don't need to make that point – Will Appleby Mar 09 '18 at 09:47
  • 2
    Agreed too. From my own experience, the guy more interested into finding someone to blame than solving the issue is usually the one to blame. Assuming your managers aren't fools, they already know who messed up. – Berthim Mar 09 '18 at 10:19
  • 66
    “Because the others will look into this“ depends, many people, especially higher ups, don't look into resolved issued. – DonQuiKong Mar 09 '18 at 11:50
  • 7
    There's a saying in business - "Own the mistake". Even if it wasn't actually yours, you did put in the effort to resolve it, so you should absolutely own the extra time and effort that you put in to fix it.

    No need to point the finger at this guy for making the error in the first place - just own the fact that you took care of it.

     – Zibbobz Mar 09 '18 at 14:16
  • 1
    I agree with @DonQuiKong. Others may or may not bother to look into it. But it sounds like mentioning where the bug was will probably make it clear that it was something he did, not you. But I also agree with the other comments that the management/stakeholders will be more impressed by that fact that you put in the extra time to get it fixed than by whose fault it was. – Sean the Bean Mar 09 '18 at 14:29
  • 88
    To satisfy the need of vengeance I'd slide a "sorry for all the holdup, the reason it took me so long is that the bug wasn't in the code that I am working on but it turned out to be in this other bit". – Džuris Mar 09 '18 at 15:36
  • 1
    Good question, and this answer is spot on. Just would like to add a few cents. In my experience, when people get that aggressive it does usually indicate they are at fault. So all the people that witnessed this know that is a strong possibility, so sending out a communication on what the root cause was, how that was fixed, and how it will be prevent in the future without aiming at his head would be appropriate and everyone would draw the proper conclusion and adjust their perceptions accordingly. – Thomas Carlisle Mar 09 '18 at 16:36
  • 51
    All of this was super helpful, thank you everybody for chiming in. I think I was a little worked up over the incident so some level-headed responses are just what I needed. I did exactly what was suggested in this answer (which is echoed in many of the other answers, I'm sorry I can only approve one as "correct"): I sent an email linking to my pull request, so the evidence is there for everyone to see. I feel good about not pointing fingers and letting others, if they care to, figure out where the real problem lies. Thank you again. – Ryan Mar 09 '18 at 17:07
  • 6
    @Džuris If CV is being used, I'd finish with "[...] working on; it was introduced in revison #9987". – Mindwin Remember Monica Mar 09 '18 at 17:25
  • 2
    You can't let it go by without pointing out that it wasn't your code but his code. Doing this without pointing the finger at who was responsible will leave people with a bad impression of you. My comment would have been different if the other guy hadn't questioned your abilities. – Pieter B Mar 10 '18 at 11:18
  • 4
    No one is going to look into it. Someone will be assigned to fix it but that is it. The manager will not care where it originated from and would probably even think that you were responsible for it. Even if he is technically sound, it's not going to matter to him who caused the issue, only that it is resolved. The best you can do is try to make it obvious who was responsible, but without explicitly using his name. Another option is to respond to the behavior of the person as a central issue and then reference how he blamed you for something but in fact it was he who was the cause of it. – Mars Mar 10 '18 at 17:09
  • 1
    Still I would have a quiet word with the colleague and let him know that it's not cool to start pointing fingers and dissing me in front if the team, and that next time he does that he can expect to have a serious problem on his hands. – Sentinel Mar 11 '18 at 09:10
  • 2
    What I would do is to include "it was introduced in version #123 of the website" and link that text to the web interface of the source control that clearly states who committed this. If you have such a web interface, of course. – Josef Mar 12 '18 at 10:03
  • 1
    @DonQuiKong They absolutely will be looking into this if it's a show-stopper defect. Business will probably order a report at the very least. – Korthalion Mar 12 '18 at 10:31
  • 1
    @PieterB You can say it wasn't your code without explicitly pointing out whose code it was, though. Josef's suggestion is how I would do that, personally (that is, note and link in the bug tracker which revision of which code caused the problem. - Plus, this is just good practice for documentation purposes, anyway, even if it had been code you wrote yourself.) – reirab Mar 12 '18 at 18:04
  • If this person (or anyone else) acts the same way again in a meeting, you might just say that you will update everyone as soon as you have found and fixed the problem. That, of course, means that you may have to publicly admit to any mistakes you have made, but that should not be a problem. – sdenham Mar 13 '18 at 15:42
  • To really make it clear, even to the laziest of recipients, include something to the effect of "this bug was introduced in commit r456, which did ". The guy making the false accusations will know it's his code in the diff, and anyone else who wants to dig deeper can easily check the version-control system and see who made that commit. – aroth Mar 18 '18 at 04:46
122

Avoid the blame game

There's always going to be issues, mistakes happen, circumstances arise that means that service X is no longer happy with service Y.

What everyone needs to concentrate on is solving the issue without getting drawn into the blame game. Let the subject matter expert look into and diagnose the problem and work things out from there.

The problem isn't (officially) with someone else - it's a problem with service Y who has its own subject matter expert to sort the issue out.

After that, there's a "lessons learned" to try and figure out what caused the issue.

Not "who". That's the key thing here. If people want to take the blame, that's the honourable thing to do.

In this case, don't retaliate, don't get drawn into the blame game, be thankful that the issue was identified and sorted out.

Community
  • 1
  • Are you suggesting to completely ignore the finger-pointing or to tell the other guy to please not do that either? – lucidbrot Mar 09 '18 at 09:21
  • 49
    That's the problem with the "avoid the blame game" meme. What usually happens is one person blames someone and then when someone says "hey, that's not me", the first person comes back and says "let's not get into the blame game". Lots of people in deep denial with flip-charts say nice and fluffy things about communication and "lessons learned" and stuff, but the reality is that you should do the right thing (not blaming others for your mistakes) because it's the right thing (not in expectation of reward), and suck it up as best you can when others don't behave because work pay you money. –  Mar 09 '18 at 10:21
  • 1
    @user6697063 I get your point here. but the game blame is pretty immature playground stuff. It's tempting to blame people for mistakes happening, but the mature thing is to concentrate on the problem itself and learn from whatever mistake caused it. Whoever caused the problem should be mature enough to admit it (if only to themselves) and move on from there. MY current team doesn't follow the blame culture, we just get on and get stuff sorted out. –  Mar 09 '18 at 13:11
  • 36
    @Snow You say its pretty immature playground stuff, but if OP's manager gets it stuck in their head that OP can't be relied on, that affects their career. It's a nice rule in general but if your abilities are publicly questioned I think there needs to be some kind of pushback, otherwise it could become a recurring issue. – Jeff Lambert Mar 09 '18 at 15:15
  • 4
    A different way of framing it is what's to "blame" is the process. Either multiple people are "to blame", or you have a process where a mistake by any one person can cause a serious issue in production in which case the whole team is "to blame". Blaming an individual for a mistake isn't going to stop them (or anyone else) from making another mistake in the future. Figuring out how to improve the process so it is harder for the inevitable mistakes to become production failures, however, is both a constructive dialogue and of business value. – Derek Elkins left SE Mar 09 '18 at 16:33
  • 4
    @snow My usual approach, if I'm blamed by people in the habit of pointing fingers, is just to say where I think the fault lies, let them say "let's not play the blame game", and then get on with life. That way other people can decide. The problem of letting it pass is that you can end up getting a reputation as a doormat among people looking to pass off their mistakes and, when it reaches critical mass, people who don't know what's happening assume that a piece of work is poor quality even if they don't know you, and it becomes lore. Usually an initial challenge is enough to stop that spiral. –  Mar 09 '18 at 17:58
  • 9
    @DerekElkins After years of working with someone who used to say that, no. Sometimes it is a person's fault. Because especially in software, there are major design flaws, just plain stupid lines of code, and complete and totally wrong understandings of the code you wrote. While we don't need public shaming of any kind, they do need to acknowledge (at least mentally) they screwed up and learn from it. No process can fix everything. Sometimes you just plain have to have competent people, and people don't get competent without taking responsibility for their bad decisions. – jpmc26 Mar 09 '18 at 23:21
  • 2
    @Snow that's nice if it works for you, but that doesn't work everywhere. In OP's situation, s/he's getting called out publicly in front of people not on the team and getting blamed. If s/he comes back with some neutrally-worded postmortem, a common result is people not closely involved (like a high-level manager) will get the idea OP messed up but at least fixed it. – Chan-Ho Suh Mar 10 '18 at 18:38
  • 1
    I blame you @Snow – Neo Mar 10 '18 at 19:45
  • Blame games destroy morale and lead to a toxic culture. Never play along with it. – Brandon Mar 17 '18 at 16:06
100

Some people have a tendency to aggressively put the blame on others in order to divert attention from themselves. Since you are also relatively new, it must have been very convenient for him to do what he did.

Here's my advice. Write a mail, describe what the issue was, how you reached the root cause, what was the eventual fix for it and how it can be prevented in the future. Include all the stakeholders in this mail.

At the end of this mail, write something on these lines,

XYZ,

Can you please add these steps in the appropriate documentation or Standard-Operating-Procedures document for this piece of code?

This way you are "not pointing a finger" at him, but you are explicitly calling out that he is the owner of this piece of code and hence responsible for it. Calling out is important because not everyone (especially the higher management) will open any codebase links to see whose commit it was.

A bit harsh, but he deserves it.

Akshay Arora
  • 1,213
  • 1
  • 7
  • 10
  • 5
    Best answer until now IMO. The idea of asking for documentation is quite good. – Rui F Ribeiro Mar 09 '18 at 15:33
  • 3
    I like the idea of adding note for "XYZ....." –  Mar 10 '18 at 05:25
  • In most places I've worked, someone does indeed look at the code, especially if there were actual user issues / outages. If not, they would tend to ask someone who has. I don't think calling out is needed, and I'd prefer not to do it, unless it comes up again and I'm getting hassled about it. – bytepusher Mar 10 '18 at 13:58
25

I've build a career off of not playing the blame game, but the truth is, humans listen to the loudest voice. However, if you get involved in defending yourself, he's going to pull you down to his level and beat you with experience.

If you can find one, you need a champion. Ideally, this should be your manager, but sometimes, it is another well respected developer. They will go to bat for you to silence the loud one. All you have to do is have a private discussion with them about the facts (not the blame) of what you did to solve the problem and how they would like you to proceed so that, next time something is wrong with the service, the problem can be found and fixed faster. That may involve writing a little testing utility that tests the service directly (without using the other developers code), or some logging or something, so the "us versus them" issue can be very quickly determined. If they know who was actually at fault, they can clear your name without you getting in direct conflict with the loud one.

I've always bent over backwards to avoid putting other developers on the defense. I've said things like "I'm having trouble duplicating the problem, can you please let me know what calls you are making to the service and what you are getting back so I can test the service correctly"? If the developer bothers to give you log traces from the actual calls and responses, ask what they expected to get back. However, most of the time, the developer will just show you their code. In that case, you can sometimes spot the problem. Even if you do, still don't call them out. They have to find the problem themselves. Have them run the code through a debugger, and innocently ask what a certain variable contains at a certain line of code. I could go on, but you get the idea.

Guy Schalnat
  • 587
  • 3
  • 7
  • 5
    Best advice here. Dealing with code is much like dealing with problems with posts here on SE; if you let things get personalized, everybody stops listening to everyone else. – T.E.D. Mar 12 '18 at 21:22
18

It's a very fine tradition in the IT industry (and in some others, like aviation) that when someone finds a problem, everyone works together to find a solution, and ideally to find the root cause so that processes can be improved, but no-one gets personal blame or suffers penalties for making the mistake. The result is that people are open and honest about their mistakes, rather than trying to cover them up, which is to everyone's benefit.

It looks as if there are people in your shop who haven't bought in to this culture, and that's something that needs management attention.

Michael Kay
  • 1,762
  • 9
  • 15
  • 15
    OP says there was an 8-people call, with 2 or more developers present. One of topics discussed was poor communication.All that while production was down. Not until the evening was OP able to effectively analyze the problem. Blame game is not the most severe problem of this "culture". – kubanczyk Mar 09 '18 at 19:33
12

I believe you should have a talk with your manager on how issues are handled, especially priority issues that impact user experience.

In this case you have 2 systems communicating with each other and suddenly the communication stops working. When communication fails, it is important both parties look at both systems. Everyone put the focus on your service, leading them to not investigate things on their end. The biggest waste of time in resolving an issue is trying to find out what is wrong with a part of it that is actually running as should.

These are, however, learning experiences. I bet you now know perfectly how to troubleshoot your service. Try to set up a troubleshoot plan based on your experience, and make one of the first steps to be sure it's actually your service that is failing ("Does the service work being called from another page?", "Is the service failing partially or entirely?"). You are THE web service guy, you can have a little confidence in your work.

Try to let go of the fact that it was actually his code that failed. Trying to call him out on that is a bit petty. He shouldn't have put the focus on you so much from the begin with. See it as a general issue within the company with troubleshooting an issue and handle it as such with your manager. You also don't know if it was actually his code, he could have also copied it from another section written by a colleague.

LVDV
  • 249
  • 2
  • 8
  • 1
    Yup, in an ideal world this web service would be covered by a test suite and running that suite would be your first port of call. If the tests all pass, then you're looking at (a) client error, (b) service error not caught by tests, i.e., a testing failure, (c) service working as intended but not as expected by the client, i.e., a documentation failure. – Ed Daniel Mar 09 '18 at 12:38
  • 1
    That's a good call, I'll have to look into some kind of test rig for the service. It's a pretty popular and robust service, so honestly I don't think it's ever an issue of bugs with them, more of a is-it-configured-properly thing. Regardless, it's my domain so it would be great to cover my bases for the next time this happens. But on the plus side I did learn more about that domain, along with a bunch of "AB's" code that interacts with my domain, so I'm better off for the experience. – Ryan Mar 09 '18 at 17:09
  • Copying code you don't understand is inappropriate, even if it is not buggy. And if it is in the same project, copying it instead of calling it is inappropriate. – WGroleau Mar 17 '18 at 04:41
4

I think the suggestion to talk with your manager is a good one. Your manager needs to know that you were blamed unfairly.

But on top of that, you are being tested. Your initial instinct is right; you have to respond, or it will get worse. I'd email him directly and let him know that the problem was in his code, and that for this time, you haven't told anyone but your manager, whom you had to tell in order to protect yourself. And lastly let him know that if he does it again, you will go public.

William Jockusch
  • 295
  • 1
  • 3
  • 1
    This will just escalate the problem, I disagree with your advice – reggaeguitar Mar 12 '18 at 18:37
  • 2
    A private attack is still an attack, and yes it will escalate the situation. Corrective action should follow the chain of command. – Phil M Mar 12 '18 at 22:02
  • 1
    More of a "proportionate response". I've had situations like this; if one doesn't respond, it has a tendency to get worse. The chain of command thing might work. Maybe or maybe not. What I can say is that from personal experience, I've seen it fail twice and succeed once. The trouble is that while waiting to find out, things can get quite a bit worse. – William Jockusch Mar 13 '18 at 02:09
4

As a software tester, I'm familiar with this situation. And also as a software tester, why wasn't this bug picked up in testing? I'm assuming you're a developer?

Yes, it sucks that you've been blamed by the guy that caused the issue. However, this is NOT what the business is going to care about. All they want is a resolution where no more show-stoppers make it to the live site.

I would wait until the next conference call if you have them regularly, and mention that you've analysed the root cause of the issue and objectively state you findings. If asked who wrote that code say who it was, otherwise do not point your finger at the guy who did it. You should be more mature/professional than that and you will get a lot more respect if you handle the situation in such a way.

As a resolution to stop this happening again, query testing as to how this slipped through in an non-accusatory way, and work with them to implement a new step or test that would have picked the issue up.

Finally, report all this to your manager. Show stopper defects are a big deal and his higher ups will be breathing down his neck too.

Korthalion
  • 333
  • 2
  • 8
2

I'm a little late to the question here, but alongside the very sound advice in Edgar's answer, there is second part to this.

If you send out a communication that you found the problem and note where it was solved, then the other developers will probably be aware where the issue lies (this is good), but management will probably not.

Doing it like this means that you've done this guy a favour - he publicly called you out, you found the issue, fixed it and didn't drop him in it with management. Building up trust like this with your co-workers will stand you in good stead in the long term.

Slight side note - this obviously depends slightly on the nature of the fault you find in their work. If what you find is so egregiously wrong that it indicates incompetence on their part, you may want to quietly escalate this to your manager - they will want to know and you are building trust with them as well.

Paddy
  • 645
  • 3
  • 6
  • If you don't expose him, others probably will. Unless they are all his "buddies," in which case you are history in spite of your innocence. – WGroleau Mar 17 '18 at 04:44
0

I suggest you sleep on it before responding to the personal aspects of the situation. If you fixed the problem and got things running again then you are still “the man”. After getting some rest it will be easier to be gracious.

Kevin Olree
  • 311
  • 1
  • 5
0

While the main issue was already addressed at length in other awswers - the other guy harassing you - I'd like to point out another perspective.

The fix was done in the code the other guy owned, yet more often than not the service could have prevented a bigger issue by being more conservative at handling requests from clients. Does it validate inputs? Does it reports any runtime errors? Is there a testing environment (this applies to consumer as well)? Sometimes an issue in the service was only waiting to surface, so I'd say just because a fix was done in one place it doesn't mean that is the whole story. Also, for issues like this, there's more than just code. There's process as well.

André Werlang
  • 168
  • 8