Skip to content

Commit b66d168

Browse files
enh: add new peer review lead role to the how to guides (#348)
* enh: peer review lead * enh: update review guide with more specifies and dashboard * enh: cleanup * Update how-to/peer-review-lead.md Co-authored-by: Leah Wasser <leah@pyopensci.org> * Update how-to/peer-review-lead.md Co-authored-by: Leah Wasser <leah@pyopensci.org> * Apply suggestion from @lwasser --------- Co-authored-by: Eliot Robson <eliot.robson24@gmail.com>
1 parent 251a00a commit b66d168

File tree

7 files changed

+83
-11
lines changed

7 files changed

+83
-11
lines changed

how-to/author-guide.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Reviewer Guide <reviewer-guide>
99
Editor Guide <editors-guide>
1010
Editor-in-Chief Guide <editor-in-chief-guide>
1111
Triage Team Guide <review-triage-team>
12+
Peer Review Lead <peer-review-lead>
1213
:::
1314

1415
```{toctree}
@@ -271,7 +272,7 @@ are usually not accepted by JOSS. Be sure to review JOSS's
271272
before writing up a paper about your package.
272273
```
273274

274-
## Post review - welcome to the pyOpenSci community!
275+
## Post review - welcome to the pyOpenSci community
275276

276277
Congratulations! Once your package has been accepted into the pyOpenSci ecosystem, you'll be invited to join our [community Slack](https://join.slack.com/t/pyopensci/shared_invite/zt-39qitgkqb-gZTIo79xCJhS5kSxW1yNfg) where you can connect with other package maintainers, get help with maintenance questions, and stay updated on community developments.
277278

how-to/finding-reviewers.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
1+
(finding-reviewers)=
12
# Finding Reviewers
23

3-
Sometimes the most challenging part of our pyOpenSci open peer review process
4+
Sometimes, the most challenging part of our pyOpenSci software peer review process
45
is finding reviewers. This page provides tips and tricks to help you find the
56
right people to review a scientific Python package.
67

@@ -35,13 +36,13 @@ up using our [reviewer signup form](https://docs.google.com/forms/d/e/1FAIpQLSeV
3536

3637
## Criteria for Choosing Reviewers
3738

38-
Here are criteria to keep in mind when choosing a reviewer. You might need to
39+
Here are the key criteria to consider when selecting a reviewer. You might need to
3940
piece this information together by searching `PyPI`, `Conda` / `Conda-forge`
4041
and the potential reviewer’s GitHub page and general online presence (personal
4142
website, social media profiles).
4243

4344
* Has not reviewed a package for us within the last 6 months.
44-
* Some package development / contribution experience.
45+
* Some package development/contribution experience.
4546
* Some domain experience in the field of the package or data source.
4647
* No [conflicts of interest](coi).
4748

@@ -55,26 +56,26 @@ complexity of the package.
5556
* **Openness** - reviewers should also have demonstrated interest in open
5657
source or Python community activities, although blind emailing is fine.
5758

58-
Each submission should be reviewed by _two_ package reviewers. Although it is
59+
Two package reviewers should review each submission. Although it is
5960
fine for one of them to have less package development experience and more
6061
domain knowledge, the review should not be split into two parts. Both
6162
reviewers need to review the package comprehensively, from their particular
62-
perspectives. In general, at least one reviewer should have prior reviewing
63-
experience, and of course inviting one new reviewer expands our pool of
63+
perspectives. If possible, at least one reviewer should have prior reviewing
64+
experience, and of course, inviting one new reviewer expands our pool of
6465
reviewers.
6566

66-
Reviewers should ideally have some subject matter expertise associated with
67+
At least one reviewer of the two should have some subject matter expertise associated with
6768
the package functionality. It is ok and even welcome if one reviewer has more
68-
technical expertise and the other focuses on usability and is less technical.
69-
Read through the Guidelines for Reviewers Section to learn more about finding
69+
technical knowledge and the other focuses on usability and is less technical.
70+
Read through the [Guidelines for Reviewers Section](reviewer-guide-sphinx) to learn more about finding
7071
and selecting reviewers.
7172

7273
(review-mentorship)=
7374
## Peer Review Mentorship
7475

7576
pyOpenSci encourages those who are newer to review to become involved in our
7677
open peer review process. As such, we offer a reviewer mentorship program
77-
where we pair a new reviewer with someone in the community that has previous
78+
where we pair a new reviewer with someone in the community who has previous
7879
review experience.
7980

8081
It is useful for reviewers to not only review the technical content of a

how-to/peer-review-lead.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Peer Review Lead Guide
2+
3+
:::{tip}
4+
There are several resources that will help you with this role.
5+
Make sure that you have access to the:
6+
7+
* shared Google folder that contains a list of editors and reviewers who have recently signed up to participate in our review process
8+
* Check out our [peer review status dashboard](https://www.pyopensci.org/metrics/peer-review/peer-review-status-dashboard.html) for the state of peer review
9+
* [Check out our editorial dashboard](https://www.pyopensci.org/metrics/peer-review/editorial-dashboard.html) to get a sense of how many of our editors are busy, and who might be available to lead a review.
10+
* Check out the current review status for the [current state of peer review](https://www.pyopensci.org/metrics/peer-review/current-review-status.html)
11+
:::
12+
13+
14+
The Peer Review Lead is responsible for keeping the software review process moving forward. They are responsible for:
15+
16+
* Keeping the review process moving forward by checking in on stalled reviews and supporting the editorial team.
17+
* Ensuring a diverse and active editorial board.
18+
* Onboarding and offboarding new editors.
19+
* Making updates to the [pyOpenSci Software Peer Review Guide](https://www.pyopensci.org/software-peer-review/) as needed.
20+
* Updating [software peer review policies](https://www.pyopensci.org/software-peer-review/our-process/policies.html) as needed.
21+
* Helping editors find reviewers as necessary.
22+
23+
This role will also help manage conflicts that may arise in the software peer review process.
24+
25+
:::{note}
26+
The pyOpenSci Executive Director has historically held the lead role in peer review. However, we are transitioning this role to a stipend position, which a community volunteer with excellent organizational and communication skills will hold.
27+
28+
Ideally, this person has experience with our peer review process as an editor or has been part of our community for some time and is familiar with the process.
29+
:::
30+
31+
## How to keep peer review moving forward
32+
33+
A volunteer editorial team runs software peer review; it also relies on volunteer community reviewers. As with any volunteer-led effort, it is common for the review process to stall or slow down.
34+
35+
As Peer Review Lead, you should monitor the status of reviews and help editors move stalled reviews forward. Our [review status dashboard](https://www.pyopensci.org/metrics/peer-review/current-review-status.html) will help you stay informed about the overall review process.
36+
37+
Ideally, you should check in on peer reviews weekly or every other week, depending on the volume of reviews submitted. We anticipate that your time allocation in this role will be 4-8 hours a month. Some months may require additional hours if we are working on new policies. Others you may simply be checking in on the review process.
38+
39+
Keep an eye on a few things when you check in:
40+
41+
### 1. Dashboard: check the date the comment date for every review
42+
43+
Before you begin, ensure the peer review dashboard is up to date. At the top of the dashboard, you will see the **Last Updated** date. The metrics repository has a cron job that runs weekly to update review status metrics. If the date at the top of the page is older than 1-2 weeks, likely, a pull request, [similar to this one](https://github.com/pyOpenSci/metrics/pull/147) needs to be merged. Go ahead and merge that PR if one exists. If the dashboard date is old and there is no open PR to merge, our cron job has likely failed. In that case, please leave a note in our Slack `#pyos-infrastructure` channel. Our infrastructure lead can investigate and resolve any issues that arise with our build.
44+
45+
:::{image} /images/peer-review-metrics-dashboard.png
46+
:alt: Dashboard last updated date
47+
:::
48+
49+
### 2. Check in on prereview checks
50+
51+
Next, check in on the prereview checks by sorting the [all-open-reviews table](https://www.pyopensci.org/metrics/peer-review/current-review-status.html#all-open-reviews) table on the **Active_Status** column. Sometimes our Editor in Chief becomes overwhelmed with too many submissions at once. It's easy to forget that they are also able to [delegate the prereview checks to another editor on the team](pre-review-checks) if they fall behind. You can remind them and even help them find other editors to step in if needed.
52+
53+
### 3. Identify and check in on stalled reviews
54+
55+
The [All Open Reviews](https://www.pyopensci.org/metrics/peer-review/current-review-status.html#all-open-reviews) section of this page will also help you keep tabs on the current activity status of our reviews.
56+
57+
Check out the **The last date a comment was left on a review.** You can sort the table by this column to see the date that someone last commented on the issue.
58+
59+
If a review has been quiet for over a month, it's a good idea to check in on things. You can either leave a note on the issue to see if anyone responds (this is ideal, as you can then track when someone responds to the issue or if you were the last person to respond!). If momentum is not gained by leaving a note for the editor and reviewers on the issue, then follow up with the editor in Slack after 2 weeks to see if they need some support. If Slack doesn't work normally, email works as a last attempt to connect with the editor.
60+
61+
Also, check out the days open column. Keep an eye out for reviews that have been open for longer than 6 months. In some cases, you may want to check in with the editor to see how things are going and whether there is a way to move the review forward.
62+
63+
In some cases a review hasn't moved forward because the editor is struggling to find reviewers. [This page](finding-reviewers) will help you with some tips on helping an editor find reviewers. Sometimes this is as easy as posting in our pyOpenSci #software-review channel. Other times we might need to run a call for reviewers.
64+
65+
### 4. Check the reviews seeking editors section
66+
67+
The [**Editors Needed** section of the dashboard](https://www.pyopensci.org/metrics/peer-review/current-review-status.html#editors-needed) is useful for you to check in and see how the Editor in Chief (EiC) is doing. In some cases, the Editor in Chief needs support in onboarding a new editor. If packages are seeking editors for more than a month, it's time to check in. Sometimes, all that is needed is for the package to have an editor, but the YAML at the top of the issue hasn't been filled out properly. This is an easy fix - just add the editor to the `editor:` field at the top of the issue. In other cases, we may need to run a call for more editors. In that case, you can consult
68+
* Post in the software-review channel to see if anyone in our Slack community is interested in stepping into an editorial role
69+
* Please reach out to the Executive Director about posting on social media to find a new editor to join our team. [The editorial signup form can be found here](https://forms.gle/VEUxEzN6YmeWSb6t8), and the responses (names, emails, and domains only) to that form can be provided to you by the Executive Director upon request. It's not publicly shared to maintain the privacy of those who sign up to volunteer with us.

how-to/reviewer-guide.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
(reviewer-guide-sphinx)=
12
# Guide for Reviewers
23

34
```{epigraph}
275 KB
Loading

images/dashboard-last-updated.png

101 KB
Loading
78.3 KB
Loading

0 commit comments

Comments
 (0)