pozorvlak | Code review gamification

Entry tags:

Code review gamification

Here's a daft idea I had last night, on which I'd appreciate feedback:

Suppose your project struggles to get code reviews done in a timely or thorough fashion¹. It might be possible to improve matters by making code review into a game: you score points for every bug you find, for suggesting successful solutions, or for completing your code review quickly. More points for finding subtler bugs, submitting working patches or completing reviews extra-quickly. At the end of the month award prizes (or perhaps just bragging rights) to whoever's amassed the most Code Review Points.

Has anyone tried this? Has anyone got any good ideas for (semi-)automating the system? Does anyone think it's a terrible idea? If so, why?

¹ You are doing code reviews, right? :-)

Flat | Top-Level Comments Only

Some places have tried something similar. The problem is that it falls under economics and game theory and human nature. The premise was that testers who find bugs get paid per bug found. Simple, right? Wrong, what happened was that coders and testers would collude to create bugs to be "found" and they would split the bounty. I suspect that something fairly similar would happen here. Especially, if there was money involved or some kind of consumable item.

Yeah, I was definitely thinking that the points should be meaningless (like Reddit/Slashdot karma) rather than money. Hence "gamification".

One way to stop collusion would be to deduct any points earned from the person whose code is being reviewed, so the net point allocation is zero. I'd shied away from that idea, because I thought it would discourage coders from trying risky or ambitious things, but maybe it's not such a bad idea.

One potential problem is "What counts as a bug?" For instance, there was some code I looked at it in a previous job (written by colleague). I moved one line outside a loop (since it only needed to be run once), and that made the code a lot faster; I think the new time was 30 seconds rather than 20 minutes. However, the old code was functionally correct, it was just slow. So, suppose that my change had made it run in 18 mins rather than 20 mins, and this particular routine was only used once a month; would it still be fair for me to take points away from my colleague?

I think that code reviews are supposed to be cooperative rather than competitive. If I look at something and suggest an improvement, my colleague will be more likely to accept my suggestion if they don't get punished at the same time.

One potential problem is "What counts as a bug?"

Yeah, that's a major obstacle to automation. The bug that prompted this thought was more subtle than yours: it wouldn't have affected speed or performance of the compiler itself, but would have adversely affected performance of generated code (though how much, I couldn't tell you - highly branchy code would be more badly affected).

I moved one line outside a loop (since it only needed to be run once), and that made the code a lot faster

Look, no optimising compiler! :-)

I think that code reviews are supposed to be cooperative rather than competitive.

Excellent point. That argues in favour of not deducting points for bugs, and preventing collusion through social pressure - which shouldn't be a problem in a small enough team for the basic idea to work.

From a business perspective, what is a 'subtle' bug, and how much should I care about them? Should I care more about a subtle bug which will occur 1-in-a-million times than an obvious bug that occurs 1-in-10 times*?

Another issue may be that it could take time away from your regular work. Again from a business perspective it may be better to spend a few hours coding a new feature than trying to spot obscure bugs which will likely never manifest (or can be dealt with on a client-by-client basis if they do).

[* Okay, the answer to that is probably 'it depends'. Imagine the bug entirely crashes the software (or worse). If the section of code is executed every day then probably not - it'd take two-and-half thousand years to come to light in any single instance. If it is executed every minute then maybe - it'd probably happen every two years. If it is executed 10,000 times a second then almost certainly - you'd see the problem every minute or so! A one-in-ten bug that only activates on software restart and only has a cosmetic effect may be less important to fix.]

[Oh, and I am aware that the idea of one-in-a-million bugs is a slightly odd one. As most code is deterministic, most bugs are surely one-in-one occurrences, given the correct starting conditions. One-in-a-million is an attempt to express how common these conditions may be in an arbitrary phase-space.]

It's also important to look at how many people use the application. I can't find a source right now, but one of the Microsoft blogs basically said "If there's a one-in-a-million bug then it will show up every day for one of our customers".

I'm pretty sure a friend of mine must have read the same blog as he was telling me about something very similar.

Statistics can be fascinating - I assume you've heard of Littlewood's Law:

'Littlewood defines a miracle as an exceptional event of special significance occurring at a frequency of one in a million. He assumes that during the hours in which a human is awake and alert, a human will experience one event per second, which may be either exceptional or unexceptional (for instance, seeing the computer screen, the keyboard, the mouse, this article, etc.). Additionally, Littlewood supposes that a human is alert for about eight hours per day.

As a result a human will in 35 days have experienced under these suppositions about one million events. Accepting this definition of a miracle, one can be expected to observe one miraculous occurrence within the passing of every 35 consecutive days – and therefore, according to this reasoning, seemingly miraculous events are actually commonplace.'

Sadly with bugs, one experiences the opposite :-)

By "subtle", I meant "not obvious from the code, and difficult to test for". And yes, you should care about subtle bugs if their impact will be high enough.

Another issue may be that it could take time away from your regular work.

That's an argument against code reviews in general, and it doesn't wash. The cost and difficulty of fixing a bug goes up more-than-linearly with the time since its introduction, so it's best to catch it as early as possible. Code reviews are one tool for doing this, and they have the additional benefit that they familiarise at least one other person with what you're doing. Also, note that I suggested awarding additional CRPs for quick reviewing.

As most code is deterministic, most bugs are surely one-in-one occurrences, given the correct starting conditions.

You haven't written much multithreaded code, have you? :-)

¹ You are doing code reviews, right? :-)

... no. I'm having a hard time getting my collaborators to even use issues tracking, and some of them are resilient to version control.

I expect this'll change, at least somewhat, in the fall.

... yeah :-( I've never worked at an organisation that does routine code reviews. Though I've been very impressed with the code reviewing done by the redo project, and to a lesser extent by the Factor project.

Sounds a bit like a work equivalent of chore wars! http://www.chorewars.com/

That's genius!

So, the aim is to increase the employees' buy-in to the code review process? A teacher friend of mine used to use a technique known as "Doggies" to encourage her class to pay attention, ask questions and generally participate. "Doggies" were cardboard tokens (with a cartoon dog on) that she made and handed out during a lesson for good answers, positive attitude, etc, etc. At the end of the lesson she had a lucky-dip bag containing random sweets, stationery items, etc. One "Doggie" entitled the holder to one dip in the bag. She found it increased concentration no end, because the class enjoyed the game aspect of the prizes. Of course, her decisions to award Doggies were somewhat arbitrary, and depended on the pupil's "normal" behaviour.

Not sure that this necessarily works in your case. If you have a timeliness problem, sounds like you need better project management and better understanding of the need for code reviews and suchlike from your fellow coders. It's a classic management problem, how to get people to do a task that they don't particularly enjoy. I know I go on about this a lot, but go and read the work of W Edwards Deming (start wth deming.org) and go on from there. I can give you a quiok primer if you want.

So, the aim is to increase the employees' buy-in to the code review process?

Not really: I'm assuming that they're on-board with the decision to have code reviews, but for whatever reason aren't actually doing the reviews quickly or thoroughly enough. If they're not onboard with the decision, they'll resist whatever you do.

One "Doggie" entitled the holder to one dip in the bag.

Interestingly, you don't actually need the concrete reward: an abstract reward is often enough to motivate people. Consider Slashdot/Reddit karma.

Not sure that this necessarily works in your case.

Well, I'm being hypothetical here: my project doesn't actually do code reviews :-(

It's a classic management problem, how to get people to do a task that they don't particularly enjoy.

Yes, exactly. I'm suggesting a possible approach for achieving this.

tancerze erotyczni [url=http://detsad154.ru/forum/profile.php?mode=viewprofile&u=113641 ] wieczory panienskie lodz[/url] tancerze erotyczni Warszawa

tancerze erotyczni Bydgoszcz [url=http://www.primorsky.ru/forum/profile.php?id=287457 ] panienski wieczor Wroclaw[/url] Torun panienski wieczor

wieczor panienski Poznan [url=http://forum.vnn.im/member.php?u=122536 ] panienskie wieczory Poznan[/url] wieczory panienskie

tancerze erotyczni lodz [url=http://www.freshenup.co.uk/forum/profile.php?id=254698 ] tancerze erotyczni torun[/url] tancerze erotyczni

wieczor kawalerski pomysly [url=http://butul.nainternetu.com/forum/index.php?action=profile;u=486523 ] panienskie[/url] tort na panienski

tancerki na wieczor kawalerski [url=http://robotyka-forum.cba.pl/index.php?action=profile;u=2037 ] zadania na wieczor panienski[/url] prezent na wieczor panienski

prezent na kawalerski wieczor [url=http://www.monstermatch.net/forum/profile.php?mode=viewprofile&u=20422 ] pomysly na wieczor panienski[/url] striptiz

babski wieczor [url=http://forums.xx-system.net/community/index.php?action=profile;u=78228 ] wieczor panienski warszawa[/url] kawalerski

striptiz [url=http://www.cruiseshipdjs.info/member.php?199185-WieczorR68 ] panienskie wieczory Wroclaw[/url] tancerze erotyczni

wieczor panienski krakow [url=http://www.renga-renk.tk/index.php?topic=968 ] wieczor panienski zdjecia[/url] wieczor panienski warszawa

wieczor panienski wroclaw [url=http://novflat.ru/forum/index.php?showtopic=16176&st=0&gopid=73686&#entry73686 ] tancerze erotyczni Poznan[/url] wieczor paniejski

wieczor [url=http://bbiforum.com/index.php?topic=13930 ] wieczor panienski pomysly[/url] zadania na wieczor panienski

panienski wieczor Poznan [url=http://forum.aero-kalijati.com/index.php?action=profile;u=16040 ] panienski wieczor Poznan[/url] panienski wieczor Wroclaw

wieczor kawalerski wroclaw [url=http://wiecory-panienskie.semistrona.net.pl/ ] panienski wieczor Poznan[/url] pomysl na wieczor kawalerski

Flat | Top-Level Comments Only

Code review gamification

Incentives

Re: Incentives

Re: Incentives

Re: Incentives

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

wieczory panienskie