The aim of this study is to test whether an automatic, reference-free quality score can rank annotators by how carefully and accurately they work. Your masks, drawn at different paces, are what we compare that automatic score against.
Twenty-four real pathology nuclei and dermoscopy lesions patches are to be annotated. For each one, you'll draw the boundary with the brush or lasso, no pre-drawn outline to start from. How long this takes is individual and depends on your own experience. Speed and accuracy both score. At the end you will send your results to me via an email button with all the necessary information.
Two quick examples of the raw photos in this bench, no markings, just what the tissue/lesion actually looks like. In the real task, the canvas starts blank. Your job is to draw the boundary yourself with the brush or lasso tools, entirely from scratch.
Take your time to look at the texture and edges here. The real patches are the same size and quality as what you see now. When a patch loads for real, look for the true edge of the nucleus or lesion, and trace it as closely as you can.
Here is what a correctly traced boundary looks like on each of the two photos from before, one for pathology and one for dermoscopy, for reference only, not something to copy exactly since every patch is different. This is the exercise for both domains: an outline that follows the true edge of the object as closely as you can manage.
Practice on these images before the real task starts, pathology on the left, dermoscopy on the right. Nothing you draw here is scored or submitted, it's only so you're comfortable with the tools first.
No wrong answers, this just helps us interpret the scores afterward.
These help us design the scoring/gamification side better for real medical annotation work.
This task didn't use points, badges, or a leaderboard. We're considering them for a separate future study, and your honest preference helps design that, it has no effect on the results you just submitted.
We're also considering giving annotators an automated starting mask (an AI-generated pre-segmentation) in a future version, instead of always starting from a blank canvas. This task didn't have that, so these are about a hypothetical future version too.
Take your time. Aim for the most accurate boundary you can manage. There's no time limit.
| # | Set | Mode | Match | Time | Hint | Points |
|---|
Each annotator downloads a results file when their run ends. Drop every file you've collected here. Nothing leaves this page, it's all read locally in your browser.