POLICY: Lines of code in a package

[lwasser]

JOSS has a policy that a package should have atleast 1000 lines of code to be a package. We have never enforced that policy, but we also have recently gotten packages with 10-12 lines of code, which doesn't quite seem appropriate.

Let's talk more here about whether we want to uphold a policy that supports some minimal amount of code to warrant a review.

[sneakers-the-rat]

10-12 lines of code

wow i'm curious. i would have figured the problem would be the other direction.

seems like something we could have the bot do, run cloc over the repository and paste the output into a comment.

i think an approximate rule makes sense, like we wouldn't want to encourage people to stuff an extra hundred lines of nonsense into the package in order to meet scope, but something like "packages should have roughly 1000 lines of non-comment non-whitespace code at a minimum, at the editor's discretion"

[eliotwrobson]

Agree with @sneakers-the-rat, the EiC always has final say so I'm not worried about excluding packages. At below 1k lines I have serious questions about how useful a given package could be, and whether the project is mature enough for a full review. Doing this with a bot that raises an alert would definitely be a useful flag for editors to look at.

[MicahGale]

In general given the agreement with JOSS I would argue there is a responsibility to meet their standards, which perhaps just outright deferring to JOSS standards may suffice. I do like the JOSS criteria for "substantial scholarly effort":

Reviewers should verify that the software represents substantial scholarly effort. As a rule of thumb, JOSS’ minimum allowable contribution should represent not less than three months of work for an individual.

[lwasser]

A few notes to consider on the JOSS requirement... (and i'm not disagreeing with anyone here just providing context!)

we have atleast one package that had under 1000 lines of code that we reviewed. I remember this one because there was some push back when we moved it to JOSS in that it was short by a small amount of lines. There also was some confusion around scope but it still was rejected by JOSS in the end even tho we accepted it. It was a lot of back and forth over a few months.

One of the thing we offer is that if there is a super useful package, we will review it even if it's less than 1000 lines of code.

I do really appreciate the language "substantial scholarly effort criteria" to avoid people creating a tiny package and submitting multiple different tools to us.

Perhaps we adopt something similar to joss - while still being aware that a value that we provide is we review software that is NOT in scope for JOSS but is clearly useful in support of open science workflows.

[sneakers-the-rat]

i agree that there are lots of useful, small packages (arguably we should encourage small packages! For some reasonable definition of small, north of leftpad or is-even), and sloc is maybe generously described as a pretty dang noisy measurement of both effort and impact.

maybe we can just have a blanket policy like "our policies may be more lenient or more strict than JOSS's policies. When a package is within PyOS's policies, but outside JOSS's policies, it is not eligible for "fast track" review"

It seems like we already have this language in the peer review docs https://www.pyopensci.org/software-peer-review/partners/joss.html

JOSS has more limited scope for packages that it will review. For instance, while pyOpenSci will review and accept API wrappers, JOSS won’t.

If your package is accepted by pyOpenSci and in scope for JOSS, JOSS will fast track your package through their review process given it was already reviewed by us.

And the editor guide: https://www.pyopensci.org/software-peer-review/how-to/editors-guide.html#optional-instructions-for-submitting-to-joss

If the package fits within the JOSS Scope

So maybe we already have accounted for divergent scopes? and that lets us both have our own process and also remain respectful/etc. of our beloved JOSS, whom we would never wish any ill effect.

The more I think about it, the more I think we can do better than a sloc minimum. I agree with basically everything Leah (and others too!) are saying, w.r.t. goals of the rule

to avoid people creating a tiny package and submitting multiple different tools to us.

which is sorta necessary to maintain a free and voluntary review process. but I'd actually love to review and catalogue a really solid (e.g.) 250 line package that does exactly one thing well in a way that nothing else does.

We already have the notion of a landscape analysis, or authors explaining the use/position/etc. Of their submission, and the overlap/differences to other packages (if any exist). Maybe our standard could include something like "if you're submitting something really small, plz spend some extra time in the landscape/context section explaining the previously unmet need your package fills - small packages can still be significant, but we discourage "salami slicing" packages for the sake of preserving editor and reviewer volunteer labor"

[MicahGale]

Reading the discussion from @sneakers-the-rat and @lwasser my main question is: is this an issue of formal policy revision, or simply review checklist (e.g., issue templates) revision. To me it sounds more like a template revision where maintainers may self-select out based on the template, or editors reject packages earlier due to having address the size and effort question earlier. At the end of the day I think it will be editor discretion, and the policy should reflect that (which it seems to mostly do so already).

[lwasser]

I think where we've landed here is along the lines of what you describe @MicahGale . It's more than just a template update as it creates more space for editors to manage submissions and make calls on whether they are in scope for pyOpenSci. (So it's language in the package scope section of our guide that empowers editors and also creates supportive guardrails for our review team without being overly prescriptive.