Contents:
1. What is the quality review tag?
2. How does it work?
3. How are tagged submissions shown?
4. Managing tagged submissions
5. How do I bulk reject these submissions?
6. Individual review option
7. Frequently asked questions
What is the quality review tag?
The quality review tag is an automated feature that flags participant submissions requiring additional attention and groups them for bulk rejection. This tag helps you identify submissions that might not meet your study standards, enabling informed decisions about approval or rejection.
How does it work?
Automatic detection: Our system automatically applies the quality review tag to certain submissions based on various completion criteria and quality indicators.
Bulk actions: You can now bulk reject tagged submissions, streamlining your review process for potentially low quality submissions.
How are tagged submissions shown?
On your study’s Submissions page, you’ll see:
- An amber flag icon next to the 'time taken' that highlights submissions completed significantly faster than expected.
A tooltip explaining:
“This submission is eligible for rejection due to being exceptionally fast. Reject exceptionally fast submissions from the bulk actions menu.”
Managing tagged submissions
The quality review tag serves as a helpful indicator to identify submissions completed unusually quickly. However, it's important to note that this feature is designed to assist, not dictate, your review process.
As a researcher, you maintain complete control over which submissions to accept, reject or return within our guidelines. The tag simply highlights submissions that might warrant closer examination, but the final decision always rests with you based on your study requirements and quality standards.
We recommend reviewing tagged submissions to ensure they align with your specific study criteria before making any decisions. Some exceptionally fast submissions may still meet your quality standards, while others might not.
How do I bulk reject these submissions?
All submissions marked as being completed exceptionally fast will be grouped together.
You can now bulk reject these in two quick steps:
- Click the ‘Bulk actions’ drop-down on the right side
- Confirm the rejections by clicking ‘Reject all exceptionally fast submissions’
Individual review option
You can still review each tagged submission individually if you prefer:
- Review each submission and assess the quality of the work
- Based on your assessment, choose to approve or reject the submission
a. If you decide all submissions should be rejected follow the steps above
b. To handle tagged submissions individually, first process those you plan to approve. This reduces the list of exceptionally fast submissions, allowing you to use the bulk action for the remaining submissions you wish to reject.
Make sure to use the bulk reject option rather than the standard reject option. Using standard rejection will count towards your rejection limit, while the "exceptionally fast" bulk rejection will not.
Frequently asked questions
Q: How should I calculate the length of my study?
To determine your study's duration, test it with friends, family, or colleagues to get realistic timing estimates before launch. Alternatively, run a small pilot study on Prolific to gather accurate timing data from representative participants before recruiting your full sample. This helps you adjust your time estimate based on actual participant behavior.
The estimated completion time must be a whole number of minutes (e.g., 5 or 6 minutes, not 5.5). This ensures clarity for participants browsing studies and sets appropriate expectations about time commitment.
Q: Should I automatically reject all tagged submissions?
Not necessarily. The tag is a quality indicator, but you can still review submissions based on your study requirements. Some tagged submissions may still meet your standards.
Q: Will participants know why their submission was rejected?
Participants receive a standard notification that their submission was rejected. The specific detection criteria are not disclosed to maintain system integrity. Participants have been informed not to contact researchers, as this decision cannot be overturned or mediated by you. If a participant contacts you about this, you may either not respond or direct them to contact Prolific Support directly.
Q: How accurate is the quality detection?
Our system uses multiple behavioural and response indicators to identify potentially problematic submissions. While highly accurate, we recommend reviewing tagged submissions to ensure they don't meet your study requirements before rejecting them.
Q: What criteria does the system use to flag submissions?
The system analyzes various completion patterns and response characteristics. We don't disclose specific criteria to maintain the effectiveness of the quality detection system.
Q: What happens to my study completion rate if I reject tagged submissions?
Rejected submissions don't count toward your study completion rate, so rejecting low-quality submissions can actually improve your overall data quality metrics.
Q: What happens to the places I bulk return?
The same process applies as with normal returns—the places will be refilled automatically.
Q: Can I turn off quality detection for my study?
The quality review tag is applied automatically to maintain platform standards. However, you're never required to act on tagged submissions - the decision to reject them remains entirely yours.
Q: How does this differ from manual quality control?
This automated system helps identify submissions that warrant closer review, complementing your own quality assessment rather than replacing it.
Q: Can I block these participants from future studies?
Yes, you can manually create a blocklist to prevent these participants from joining any future studies. Check out our 'Custom blocklist' article on how to do this.
Q: I’ve accidentally approved participants instead of bulk rejecting them
Unfortunately, once a participant has been approved, this action cannot be reversed. Please review submissions carefully before approving them, especially when dealing with potentially low-quality submissions. We recommend checking all flagged submissions thoroughly before taking any approval actions.
Q: I have a full set of data for a participant but they have been flagged. Should I reject them?
As a researcher, you maintain complete control over which submissions to accept, reject or return within our guidelines. The tag simply highlights submissions that might warrant closer examination, but the final decision always rests with you based on your study requirements and quality standards.
We recommend reviewing tagged submissions to ensure they align with your specific study criteria before making any decisions. Some exceptionally fast submissions may still meet your quality standards, while others might not.
Need further help?
Click here to contact us