Human Computation / Crowdsourcing
Benchmarking of crowdworker's performance mapping task quality and efficiency on crowdsourcing platforms leading to development of a recommendation engine - CrowdUtility
CrowdUtility Recommendation Engine
Architectural diagram of CrowdUtility - a recommendation engine for crowdsourcing platforms.
Efficiency & Quality Variance
Crowd workers (unlike in a typical organization), exhibit varying work patterns, expertise, and performance - with little or no control that can be imposed on them. Requesters (e.g. enterprises) also exhibit diverse requirements in terms of the size, complexity and timings of the tasks, as well as SLAs (performance expectations). Clearly, the heterogeneity makes the choice of a platform suited for a given task difficult for the user.
"How can business owners decide which crowdsourcing platform to choose to meet enterprise SLAs?"
Focusing only on Externally Observable Characteristics (EOCs), without any internal knowledge of the platforms, this research built an end-to-end recommendation engine starting from requirements specification, to platform recommendation to execution of the tasks.
Benchmarking
Collect performance data of crowdworkers
Build models
Create statistical models that characterize each platform over time
Recommendation
Compute recommendations based on input task requirements and existing behavior models
Dispatch Tasks
Send tasks to the recommended platform for execution.
Update models
On completion of tasks, new performance data is fed back to update the models in real-time.
Research Program
Connected studies conducted progressively answered different aspects of the central research question.
Benchmarking Crowdworker Performance
"How does crowdworkers performance vary across time of the day, day of the week, geography, cost, number of tasks, task complexity?"
Baseline crowdworker performance
Development of CrowdUtility recommendation engine
"How to build a recommendation engine that dynamically assigns tasks to platforms based on cost, accuracy, and deadline constraints?"
CrowdUtility Recommendation Engine, Task Scheduler
Understanding Crowd Workers Dynamic Performance Variability across Crowdsourcing Platforms
Background
Crowd workers exhibit varying work patterns, expertise, and quality - which, in turn, lead to wide variability in the performance of platforms. Very little work, however, has been done in terms of understanding these variations and leveraging the information for better selection of platforms.
Research Questions
- Identify parameters along which two crowdsourcing platform's performance differ?
- Understand the patterns / interactions across these parameters.
Research Approach
Key Findings
Geography
Crowdsourcing platforms varied in the availability of workers across geographies
Day of the Week
One platform was faster on weekdays while the other was faster on weekends
Incentive
As payment increases, we observed faster task completion times - at the cost of accuracy.
Task Complexity
As task complexity increased, task accuracy dropped on both crowdsourcing platforms
Baseline studies on worker performance benchmarking directly informed the recommendation engine parameters developed in the second phase.
What We Learned
Learnings from this program
Externally Observable Characteristics
EOCs were sufficient to model crowdsourcing platform performance characteristics
Research Artifacts
Empirical assets and frameworks generated to guide future enterprise-wide design and engineering direction.
Benchmarking data
Performance data of two crowdsourcing platforms
Classification models
Characterizing the performance of the two crowdsourcing platforms
Task Schedulers
That decide the batch size and schedule tasks
Proof of Concept
A working prototype of the end to end CrowdUtility system
Additional Studies in this Research Program
This representative deep dive is one part of a much larger program of research.
CrowdUtility: A Recommendation System for Crowdsourcing Platforms
Task Scheduling algorithms
Research Impact
The structural, organizational, and methodological contributions generated by this program.
Product Contributions
- A working prototype of the end to end CrowdUtility system
Organizational Alignment
- PoC built and tested with a client
- 4 patents
- 7 publications
Methodological Value
- Using externally observable characteristics to model
- Task schedulers