Many years back, we conducted some analysis on how the number of citations for a paper evolves over time. We noticed that while the raw number of citations tends to be a bit difficult to estimat...
http://www.behind-the-enemy-lines.com/2024/01/the-pip-auc-score-for-research.html
Today is October 18th. It is 41 years since Greece voted for Andreas Papandreou with a 48% vote percentage to be elected as prime minister, fundamentally changing the course of history for Greece...
http://www.behind-the-enemy-lines.com/2022/10/tell-these-fucking-colonels-to-get-this.html
Last week we wrote in the Compass blog how we estimate the geographic footprint of an agent . At the very core, the technique is simple: Use the addresses of the houses that an agent has boug...
http://www.behind-the-enemy-lines.com/2021/11/geographic-footprint-of-agent-or-one-of.html
The New York Times has an article about Mechanical Turk in today's print edition: "I Found Work on an Amazon Website. I Made 97 Cents an Hour ". (You will find a couple of quotes from yours trul...
http://www.behind-the-enemy-lines.com/2019/11/mechanical-turk-97-cents-per-hour-and.html
A few weeks ago we had a discussion about citations, and how we can compare the citation impact of papers that were published in different years. Obviously, older papers have an advantage as they...
http://www.behind-the-enemy-lines.com/2018/11/distribution-of-paper-citations-over.html
TL;DR: There are about 100K-200K unique workers on Amazon. On average, there are 2K-5K workers active on Amazon at any given time, which is equivalent to having 10K-25K full-time employees. On a...
http://www.behind-the-enemy-lines.com/2018/01/how-many-mechanical-turk-workers-are.html
(This is my answer to a question posted on Quora ) Mechanical Turk is a platform for work. Workers get paid, which makes now Amazon a payment processor. Payment processors are moving money on b...
http://www.behind-the-enemy-lines.com/2017/01/why-was-my-amazon-mechanical-turk.html
In Game 4, of the 5-game series between AlphaGo and Lee Sedol, the human Go champion, Lee Sedol managed to get his first win . According to the NY Times article: > Lee had said earlier in ...
http://www.behind-the-enemy-lines.com/2016/03/beat-machine.html
In my last post, I examined the number of "active requesters" on Mechanical Turk, and concluded that there is a significant decline in the numbers over the last year. The definition of "active r...
http://www.behind-the-enemy-lines.com/2016/02/a-cohort-analysis-of-mechanical-turk.html
It seems that after years of neglect, Mechanical Turk starts losing its appeal. In our latest measurement, we see Mechanical Turk losing 50% of its requesters in a YoY measurement. A few days...
http://www.behind-the-enemy-lines.com/2016/02/the-decline-of-amazon-mechanical-turk.html
A few months back, I launched demographics.mturk-tracker.com , a tool that runs continuously surveys of the Mechanical Turk worker population and displays live statistics about gender, age, incom...
http://www.behind-the-enemy-lines.com/2015/06/an-api-for-mturk-demographics.html
The Center for Data Science at NYU invites applications for a post-doctoral fellowship in statistical methodology relating to evaluating rater quality for a new research program in the applicati...
http://www.behind-the-enemy-lines.com/2015/06/postdoc-position-for-quality-control-in.html
I am often asked about statistics and data about the global population of "crowdsourcing" workers, going beyond Mechanical Turk. I am happy to say that from now on I will be able to point everyo...
http://www.behind-the-enemy-lines.com/2015/05/the-world-bank-report-on-online-labor.html
One of the most common question that I receive is whether I have new data about the demographics of Mechanical Turk workers. The latest data that I had collected were back in 2010, and it was not...
http://www.behind-the-enemy-lines.com/2015/04/demographics-of-mechanical-turk-now.html
One of the components that I use in my class is student presentations. While I like having students present, I had always a hard time grading the presentations. Plus, many students seemed t...
http://www.behind-the-enemy-lines.com/2014/06/my-peer-grading-scheme.html
We had an interesting discussion a few days back about online job markets, and why they are not a HUGE success so far, when other, comparatively less important products are getting huge valuatio...
http://www.behind-the-enemy-lines.com/2014/04/online-markets-selling-products-vs.html
I have been following with interest the discussion about the future of education. *** Some people criticize existing educational institutions, indicating that they offer little in terms of...
http://www.behind-the-enemy-lines.com/2014/01/future-of-education-fighting-obesity-or.html
A common question that comes up when discussing research in crowdsourcing, is how it compares with similar efforts in other fields. Having discussed these a few times, I thought it would be good...
http://www.behind-the-enemy-lines.com/2014/01/crowdsourcing-research-what-is-really.html
For those not familiar with the term, the Lake Wobegon effect is the case when all or nearly all of a group claim to be above average, and comes from the finctional town where "all the women ar...
http://www.behind-the-enemy-lines.com/2013/10/badges-and-lake-wobegon-effect.html
A public service announcement, to advertise CrowdScale (http://www.crowdscale.org/ ) a cool workshop at HCOMP 2013 that focuses on challenges that people face when applying crowdsourcing at ...
http://www.behind-the-enemy-lines.com/2013/09/crowdscale-workshop-at-hcomp-2013.html
I am a big proponent of outsourcing work using online labor markets . Over the last decade I outsourced hundreds of projects, ranging from simple data entry to big, complex software products. I ...
http://www.behind-the-enemy-lines.com/2013/07/online-labor-markets-why-they-cant.html
In crowdsourcing, redundancy is a common approach to ensure quality. One of the questions that arises in this setting is the question of equivalence. Let's assume that a worker has a known proba...
http://www.behind-the-enemy-lines.com/2013/07/majority-voting-and-information-theory.html
I just read that Facebook started thinking about brand safety , and will restrict ads from appearing next to content that may be controversial (e.g., adult-oriented content). I was rather s...
http://www.behind-the-enemy-lines.com/2013/06/facebook-implements-brand-safety-doing.html
Over the last year, Amazon embarked into a big effort: All holders of an Amazon Payments account (which includes all the Mechanical Turk worker) had to verify their accounts, by providing their ...
http://www.behind-the-enemy-lines.com/2013/06/mechanical-turk-account-verification.html
One of the key problems in crowdsourcing is the issue of quality control. Over the last few years, a large number of methods have been proposed for estimating the quality of workers and the quali...
http://www.behind-the-enemy-lines.com/2013/06/project-troia-quality-assurance-in.html