Modifications to the Scoring system

Eyewire General
#1

Thanks everyone for giving us lots of feedback on the scoring system.  We’d like to make it as fun for everyone as possible while still aiding our scientific goals.  We aren’t ready to do a total overhaul yet, we’ve still got a lot of data to analyze, but we are prepared to make a few small changes which should improve the experience in the short term.


First, there’s been a lot of feedback that getting 0 as a score is no fun.  At the same time, we don’t want to give lots of points to people for doing things wrong and/or too quickly.  To that end, we’re going to have 2 parts to the score.  The first part is based on the amount of time you spend on a task.  It will look roughly sigmoidal with respect to time.  If you click done immediately, you will get 0 points for that.  We will chose a characteristic time length for a task.  Currently we are thinking about something in the 1.5-2 minute time frame.  From what we can tell, these tasks take about that long to do anyway, so that shouldn’t really be an issue for most of you.  Let us know if it’s taking you much more or less time to do tasks than that.  The points that you get for doing the task will max out around that characteristic time.  The goal of this part of the score is to encourage people to do more tasks!

The second part of the score will be very similar to the current scoring system.  We try to measure how accurately you did the task based on how everyone else did it and we score you based on that.  The feedback that we’ve gotten on this piece is that there is too much variation in the scores you get.  Sometimes you get 12 points, sometimes you get 800.  It order to calm this down a bit we are going to normalize the number of points you get to a more reasonable range.  We haven’t totally settled on the numbers yet, but we hope that it will still be rewarding if you do well, but slightly less random how many points you get for it.

We’ll be pushing out these changes later today.  Please let us know what you think of the changes, and keep on wiring!
#2

Interesting… all other things being equal, the penalty of completing a task quickly could be cancelled out by the points gained by completing more tasks, and v.v. So (again, all things being equal) the penalty for people with slower computers and slower network connections could be cancelled out by the gain they get for spending more time on a task. That’s approximately fair, but could require some balancing.


Looking forward to seeing normalization of points per task!
#3

I’m a brand-new user… I’m confused about something.


I did a bunch of the puzzles under “play” yet I still have 0 tasks completed.  Why?

How do I get to the “tasks”?
#4

@monkeywidget If you are just starting, you’re probably still doing the tutorial tasks.  The goal is to try to help you learn how to do the eyewiring.  There are 15 tutorial tasks, and once you are done with them you start on the real deal!

#5

OK, that can’t be right… now small tasks with nothing to do are scoring 100 points.

#6

@robertb I was about to say the same: 100 points for not adding anything?? It still requires some tweaking I guess!

#7

I was seeing 65 for doing nothing and quite a range for doing something.


Needs some balance me thinks.
#8

I think the score += sigmoid(time) needs scaling, probably reduce the multiplier and stretch the time out. It seemed like 100 = order of a minute or so, and I think it should be much less than that.

#9

I take varied lengths of time depending on the slides I’m looking at.  It typically takes me 1 minute per task when I’m not multitasking on my computer. 

I feel like I’ve been getting points way too easily this morning.  I like points.  However, my concern is that without receiving 0 points for tasks that are not correct, I’m left with no frame of reference for what is correct.

#10

It seems OK to me. On tasks with nothing to do, I get a few points based on how much time I spend on the task (the time parameters seem pretty good now, thanks!). On tasks with more to do, I get more points. I’d still like to see a detailed accounting of each score, though.

#11

We scaled the time bonus back a bit.  It now maxes out at 20.  It was a bit high there for a while.

#12

So it’s back to this thread. It seems today that no matter what I do I can’t seem to score more than 20 points a task. (Of course less for the nothing to do here, slivers that take 10 slides to go through and that’s it, those give out like 6 to 13 points, my guesses.) But, and this is why I now care about this again, I just spent like 20 mins, give or take on one task that (maybe this is an indicator) looped all over the place with in the task, I took a capture of it as I was going to submit it to the gallery with the title ‘twister’ because it looks like the thing is playing a 3D version of the popular party game twister. And that one scored at 20 points, which tells me something is a little off, again.

#13

Well, that’ll teach me to get all uppity about the scoring. Just had one that totally blows what I said above out of the water. BUT!.. on such a huge task why would it give out such a pitance? Was that one of the ‘trailblazer’ units? I’ve noticed that they stopped telling me this even though you would think that there would be a lot of that now that there is a new cell to work on. So any ideas what is going on with this? Thanks again.

#14

So I think what might be going on is something that we’ve mentioned before.  Either you were the first person to get the task (a trailblazer) in which case we have no frame of reference on which to score you, or the few people who did it before you did it wrong, in which case we have a bad frame of reference on which to score you.  (It’s also possible that you did it wrong, but we’ll assume that’s not the case for now :slight_smile: ).  We are still looking into new scoring systems which will better be able to handle situations like this.  Sorry that it can be kind of frustrating at times.

#15

Well, I appreciate the reply. A few thoughts. It would be nice to know I was the trailblazer and that is why I get minimal points instead of wondering if I messed up or something is wrong with the system. I also understand the bad frame of reference and or bad wiring on my part (;-p).


But, the why I am back here again, it sure seems like I get 40 sliver, trailblazer, do-nothing etc tasks in a row, one or two decent ones and then right back to the frustrating bits and bobs tasks. I’m not complaining really, these are just my observations about the system. Thanks for all the hard work guys.
#16

In addition, the tasks that span the whole set of slides and have like two (estimate) holes in it are probably the worst. I know they have to be done but do they all have to be done by me?

#17

Is there a maximum point value per task?

#18

We haven’t implemented such a thing yet.  We are still considering it.