BIP3 revisions #2037
BIP3 revisions #2037
Conversation
…m to the ML themselves
…o assignment Based-on: aecb84f
| provided it is the original work of its authors and the content is of high quality, e.g. does not waste | ||
| the community's time. No content may be generated by AI/LLMs and authors must proactively disclose | ||
| up-front any use of AI/LLMs. | ||
| the community's time. No content may be generated by AI/LLMs. |
There was a problem hiding this comment.
Would prefer that authors be required to proactively disclose out of respect for the community's time (and that of the editors).
There was a problem hiding this comment.
If someone uses AI to iterate on the language of a BIP (e.g. to make it more concise), you will get a high "LLM score", but that does not mean that the entire BIP was AI-generated. AI can level the playing field for people whose first language isn't English.
Disclosure of AI use makes sense, but maybe we can replace the sentence 'No content may be generated by AI/LLMs.' with something like 'AI/LLMs may assist with (language) refinement, but normative content must originate from the authors.'*
*Full disclosure, AI helped me generate the alternative sentence ;)
There was a problem hiding this comment.
The issue is not the use of spelling and grammar checkers, which have been in use for a long time. The issue is coming from overuse of LLMs to generate the content itself.
The priority is not wasting the community’s time.
Experienced human review is the scarce resource and bottleneck, and with AI that situation doesn’t appear to be improving, at least not yet.
Who wants to review LLM content or AI hallucinations? Right, no one. Humans want to review work from humans who have shown merit.
Yes, we’ll try to have AI review the AI-generated content and code, to level the playing field... but for now, we’ll probably need to rely even more on human heuristics like reputation, proof of work, and personal recommendations, to decide what deserves expensive human attention for the proposals that get past the AI filters.
See also the discussion at https://stacker.news/items/1287863.
There was a problem hiding this comment.
Humans want to review work from humans who have shown merit.
People evidently do review BIP PRs filed by accounts with no prior contributions. Ultimately, the merit of the BIP is what matters.
There was a problem hiding this comment.
Yes, I wrote "want" -- not "do".
The BIP editors do review, and you are encouraged to help by also contributing review, but that sentence summarized our general preference and mine.
There was a problem hiding this comment.
Several reviewers have been suggesting that #2006 be reverted with what I find to be convincing arguments:
- The rule only hurts good actors, bad actors would never disclose either way
- The main motivation for the rule was to curb low-quality proposals as they waste time, but proposals are already required to be of high quality
- Competent authors can use AI-tools to save time and create high-quality proposals
Therefore, this paragraph will likely not be present in BIP 3 either way.
| ideation phase, e.g., by generating substantial public discussion and commentary from diverse contributors, by | ||
| independent Bitcoin projects working on adopting the proposal, or by the authors working for an extended period toward | ||
| improving the proposal based on community feedback, will be assigned a number by a BIP Editor. A number may be | ||
| ideation phase, will be assigned a number by a BIP Editor. |
There was a problem hiding this comment.
Noting here that I would like to think about this more. It seems to me that the bar for numbers might be too low and that standards on quality/proof of work, soundness, completeness, and originality may need reinforcing in light of the ease of lobbing LLM-generated slop or hallucinations over the fence at us.
There was a problem hiding this comment.
Given that this keeps the criteria as before and only drops the examples of how “progressed beyond ideation” may present, this seems like a disimprovement to me. Clearly, a list of examples is not meant to be considered exhaustive, perhaps you could suggest more examples how progressing beyond the ideation phase might present.
| For each new BIP pull request that comes in, an editor checks the following: | ||
|
|
||
| * The idea has been previously proposed by one of the authors to the Bitcoin Development Mailing List and discussed there | ||
| * The idea has been previously proposed to the Bitcoin Development Mailing List and discussed there |
There was a problem hiding this comment.
Probably best to keep this as-is to avoid other people sniping/jumping on an idea.
Further, not only "The idea" but also the proposed draft for review on the author's own repository.
We're seeing new, out-of-the-blue github accounts (often with no established record or proof of work in the space) opening a PR directly to the BIPs, skipping the steps, and then using the "fait accompli" to protest the PR being closed.
There was a problem hiding this comment.
I don’t feel strongly about this one. I recall three cases in which a PR was opened by someone other than the person initially proposing an idea, twice consensual, once by surprise. It seems to me that this sort of thing sorts itself out.
I’d rather reiterate explicitly that both the idea and an early draft should be sent to the mailing list, or even require at least the latter as BIP 2 does.
murchandamus
left a comment
murchandamus
left a comment
There was a problem hiding this comment.
Thanks for your review.
| BIPs are intended to be a means for proposing new protocol features, coordinating client standards, and | ||
| documenting design decisions that have gone into implementations. A BIP may be submitted by anyone, | ||
| documenting design decisions that have gone into implementations thereof. | ||
| BIPs should cover topics that at least potentially have a need of standardization, involving multiple projects, and not merely configuration (such as default settings or per-node policies, except where the latter may overlap with standardized interactions). |
There was a problem hiding this comment.
Very similar language regarding proposals being relevant is already found in the workflow section below: “The idea must be of interest to the broader community or relevant to multiple software projects. Minor improvements and matters concerning only a single project usually do not require standardization and should instead be brought up directly to the relevant project.”
The rest of this suggestion also seems superfluous, as policy that is only relevant to one project is considered non-relevant per the quoted section, and policy that is relevant to multiple projects or otherwise of interest to the broader community could clearly be useful here.
| provided it is the original work of its authors and the content is of high quality, e.g. does not waste | ||
| the community's time. No content may be generated by AI/LLMs and authors must proactively disclose | ||
| up-front any use of AI/LLMs. | ||
| the community's time. No content may be generated by AI/LLMs. |
There was a problem hiding this comment.
Several reviewers have been suggesting that #2006 be reverted with what I find to be convincing arguments:
- The rule only hurts good actors, bad actors would never disclose either way
- The main motivation for the rule was to curb low-quality proposals as they waste time, but proposals are already required to be of high quality
- Competent authors can use AI-tools to save time and create high-quality proposals
Therefore, this paragraph will likely not be present in BIP 3 either way.
| Authors may want additional help with the BIP process after writing an initial draft. In that case, they may assign | ||
| one or more Deputies to their BIP. Deputies are stand-in owners of a BIP who were not involved in writing the | ||
| document. They support the authors in advancing the proposal, or act as a point of contact for the BIP in the absence of the | ||
| one or more Deputies to their BIP. |
There was a problem hiding this comment.
Commit message:
“Remove assumption Author was involved in writing the original document”
The edited line describes Deputies, not Authors.
From Email:
- Authors/Deputies assumes the champion (Author) was involved in writing
the document. This is a deviation from the current process where the
champion can be reassigned by editors if the current one is MIA.
That’s one of the reasons to introduce Deputies: it allows us to appoint Authors or Deputies depending on the situation.
| Authors may choose to submit BIPs in MediaWiki or Markdown[^markdown] format. | ||
|
|
||
| Each BIP must have a _Preamble_, an _Abstract_, a _Copyright_, and a _Motivation_ section. Authors should consider all issues in the | ||
| Each BIP must have a _Preamble_, an _Abstract_, a _Motivation_, a _Specification, and a _Copyright_ section. |
There was a problem hiding this comment.
We have Informational BIPs that don’t have a Specification section, and it seems to me that Informational BIPs do not require a Specification section. The BIP Types section specifies that Specification BIPs require a Specification, and heavily implies that Process BIPs do, too (“Process BIPs are like Specification BIPs, …”).
| * Reference Implementation — Where applicable, a reference implementation, test vectors, and documentation must be | ||
| finished before the BIP can be given the status "Complete". Test vectors must be provided in the BIP or | ||
| as auxiliary files (see [Auxiliary Files](#auxiliary-files)) under an acceptable license. The reference implementation can be provided in the BIP, as an auxiliary file, or per reference to a pull request that is expected to remain available permanently. | ||
| as auxiliary files (see [Auxiliary Files](#auxiliary-files)) under an acceptable license. The reference implementation can be provided in the BIP as an auxiliary file, reference to a pull request (expected to remain available permanently), a new code repository, or similar. |
There was a problem hiding this comment.
A specific branch seems more obvious to me than a new repository, but sure. This should be phrased in a way that makes clear that it’s expected to remain available permanently either way.
| repository](https://github.com/bitcoin/bips). | ||
|
|
||
| When either a new BIP idea or an early draft is submitted to the mailing list, BIP Editors and other community members should comment in regard | ||
| When either a new BIP idea or an early draft is submitted to the mailing list, BIP Editors and/or other community members should comment in regard |
There was a problem hiding this comment.
It already says “should”, but just “or” instead of “and” is also fine with me.
| For each new BIP pull request that comes in, an editor checks the following: | ||
|
|
||
| * The idea has been previously proposed by one of the authors to the Bitcoin Development Mailing List and discussed there | ||
| * The idea has been previously proposed to the Bitcoin Development Mailing List and discussed there |
There was a problem hiding this comment.
I don’t feel strongly about this one. I recall three cases in which a PR was opened by someone other than the person initially proposing an idea, twice consensual, once by surprise. It seems to me that this sort of thing sorts itself out.
I’d rather reiterate explicitly that both the idea and an early draft should be sent to the mailing list, or even require at least the latter as BIP 2 does.
| * Motivation, Rationale, and Backward Compatibility have been addressed | ||
| * Specification provides sufficient detail for implementation | ||
| * The defined Layer header must be correctly assigned for the given specification | ||
| * The defined Type and Layer headers must be correctly assigned for the given specification |
| - A BIP in Draft or Complete status may no longer be closed solely on grounds of not making progress for three years.[^rejection] | ||
| - A BIP in Draft status may be updated to Closed status if it appears to have stopped making progress for at least a | ||
| year and its authors do not assert within four weeks of being contacted that they are still working on it. | ||
| year and its authors do not assert within four weeks of being contacted that they intend to continue working on it. |
| [^OtherImplementations]: **What is the issue with "Other Implementations" sections in BIPs?** | ||
| In the past, some BIPs had "Other Implementations" sections that caused frequent change requests to existing BIPs. | ||
| This put an onus on the BIP authors, and frequently led to lingering pull requests due to the corresponding BIPs’ | ||
| This put a burden on the BIP authors and editors, frequently leading to lingering pull requests due to the corresponding BIPs’ |
4 participants
This addresses several issues identified in my review of BIP 3. Other unaddressed issues are being posted to the mailing list for further discussion.