Skip to content
Guidance: using forms for independent evaluations
We mainly encourage these to be done using (a version) of our regular evaluation form.
However, these forms are geared towards our commissioned evaluators, not specifically for independent evaluations. This shouldn’t matter too much, though. Below, some tips on how independent evaluators should consider this form and fill it out.
Introduction/about our evaluation process
‘We ask evaluators to’...
Write an evaluation, up to the standards of a high-quality referee report for a traditional journal. Consider as well as .
We understand that independent evaluators may find it difficult to reach these standards
Please try to address any specific considerations mentioned in the our bespoke evaluation notes, and any specific requests from the evaluation manager.
If you are evaluating a paper we’ve previously evaluated, you should find these notes linked in the “evaluation summary and metrics” (at unjournal.pubpub.org), usually under “What we asked evaluators to consider”. For the ~newest packages, we’re also working to add a “Issues meriting further evaluation” field
If you are evaluating a paper on our ‘priority list’ (coming soon) we will try to include some suggestions as well.
Also, if you are evaluating a paper not on the above lists, please before proceeding
2. Give .
3. Identify and assess the ‘most important claim’ of the research.
4. Answer a short questionnaire about your background and our processes.
You don’t have to do all of this, but it’s generally helpful and we appreciate what you can do, as well as your comments on how to make the system better and more understandable.
Compensation
As noted, we compensate invited evaluators for prompt evaluations. Independent evaluations are not compensated, but they will be eligible for prizes and other benefits.
Project/Paper details
Please link the version of the paper or project you evaluated ... e.g., a link to a publicly hosted paper on Arxiv, NBER, etc.
Your details
We need this to give you public credit for your work (when we link or host it), and/or to make you eligible for our ‘best independent evaluation prizes’ and other rewards.
Evaluation report
Here’s where you paste, link, or upload the substance of your report. See the guidelines noted in the survey.
Structured ratings/metrics
We consider these a very important part of our standard evaluation process, roughly as important as the evaluation report written content.
Perhaps this section is slightly less important for the independent evaluations, and we don’t plan to integrate these ratings with those from our commissioned evaluations. However, still should be very useful and important to help readers understand your assessment of this research, it's comparative strengths and weaknesses in different areas and relative to other work. It should be helpful for you to learn how our evaluations work and for us to get a sense of how well independent evaluators understand these, and benchmark and calibrate them.
So please complete these carefully... But if you need to skip some sections because they were relevant or you don't understand them, that's OK. (And please give us feedback.)
please rank this paper relative to all serious research in the same area that you have encountered in the last three years.
Perhaps you have not read so much ‘serious research in the same area’. Don’t let this discourage you. Give your best assessment according to your beliefs about what the research you would be likely to find in this area.
Journal ranking tiers
Naturally, you may have even less experience with these “journal ranking tiers”. Again, give your best impression and leave us your thoughts and feedback. And do have a look at the which have deeper links to journals we suggest might fit within these tiers.
Claim identification and assessment
I expect the section to be particularly relevant and useful for the independent evaluations. (But as always, give us your feedback to help us improve this process.)
Final section/survey
Not all of the questionnaire survey questions are relevant to independent evaluators, but most of them are. Please complete this if you feel comfortable doing so, it seems likely to be helpful to us.
As noted, can opt into join our evaluator pool and we highly encourage you to fill out the linked form at which helps us understand your interests, and expertise for future, potential commissioned and paid assignments.
As mentioned before, if you do strong independent evaluation work, this will make it more likely we will contact you for such assignments.
Load content from globalimpact.gitbook.io?
Loading external content may reveal information to 3rd parties. Learn more
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.