Standards for evaluating a new Committer to the project

[rusackas]

Neither Apache nor the Superset PMC seem to have written standards in place for the bar one should meet in order to be considered as a committer. This discussion should evaluate the thresholds and standards we use to measure who would be a good fit. I'll provide markdown herein that we should transfer to the Superset Wiki upon finding alignment. This may be updated to reflect the comments in the discussion as it unfolds.

---

Standards for evaluating a potential Committer to the Apache Superset project

There are several ways to be a contributor to the Superset project and its surrounding community. While special exceptions may apply (see below) the general standards for Committers are based on code contributions and related involvement. Potential Committers will be held to these standards when put up for a [VOTE] thread on the [email protected] mailing list, as is standard Apache procedure.

Must haves:

Minimum Technical contributions

• A track record of at least six months since their first commit was merged.
• A history of either significant code contribution volume AND/OR significant contribution impact.
• Quality code contributions that are simple, well-tested, and well-designed patches.

Minimum Community contributions

• An apparent or stated intent to contribute to the Superset project in an ongoing manner, in order to help represent, guide, and grow the Superset project and its developer community.
• Generally provides and promotes civil discourse and constructive conversations, acting in a helpful and diplomatic capacity in community conversations.

Additional "Nice to have" attributes:

• Ongoing, supportive engagement in the Superset community's various fora (Mailing list, Github, Slack, etc.)
• A history of improving engineering best practices via documentation, tests, style guidelines, etc.
• A history of reviewing PRs, demonstrating adherence to best practices, and seeking to promote stability and maintainability.
• A history of responding to Issues in a constructive manner to either resolve them directly or help steer them on a path toward resolution.
• Testing and voting on releases
• Participation in discussions about roadmaps, architecture, developer experience, and long-term plans
• Assistance in improvements to the website, documentation, and the wiki
• Participation in community events such as meetups, talks, etc
• Improve project infrastructure in order to increase the efficiency of committers and other contributors

Special exceptions for non-code contributions:

On a case-by-case basis, committership (i.e. write access to the Github repo) may be warranted for individuals who may not meet the above code contribution standards, but instead will be considered on the basis of other community-driven qualifications. The granularity of repo permissions currently is limited, meaning those who wish to help triage and manage Issues, Discussions, Projects, labels, and other parts of day to day repo operations, cannot be given those capabilities unless they are made a Committer. These contributors may provide sufficient merit via other efforts including (but not limited to) input on Design and Product decisions involving the community, driving issues and questions toward resolution on the mailing list, Slack and other fora, verifying release candidates, giving talks, organizing community events, and other forms of evangelism and community engagement. Superset, like all Apache projects, has a strong focus on the project community, and Committers may be considered for this merit when being put forth in a nomination on the mailing list.

Additions or revisions may be made to the above guidelines by making a proposal seeking lazy consensus to the dev mailing list, and applying any agreed-upon changes to this wiki page.

---

Additional inputs welcome on the above... I'm not sure if there should be a minimum number of PRs, for example. Would a committer who did a few large/impactful commits (executing on a SIP, for example) be sufficient? Are there other weights and measures we should consider here?

I should also add that I don't think being a Committer should be TOO high of a bar to reach for. PMC should have significantly higher standards. The hope here is that more committers will garner more sustainability of the repo as people gain the power to tag/merge/close/assign Issues and PRs.

[john-bodley]

Though I applaud the inclusion of,

A long-term commitment to the Superset project.

I think this is somewhat subjective and hard to quantify. How would we evaluate such a condition?

[rusackas]

Simple: tattoos

[rusackas]

In reality, I'll just remove this from the "must haves" and roll it into a bit of a general preamble, based on some of the ones @ktmud provided.

[ktmud]

Thank you for starting the discussion! Do we want to add a requirement related to doing code reviews or clarify the standards related to code quality? The biggest privilege of being a committer is having write access---via stamping and merging other people's PRs, so I would hope we can show some emphasis on code quality.

Here are the standards used by some other Apache project for reference:

Apache Spark and Apache Hadoop

1. Sustained contributions to Spark: Committers should have a history of major contributions to Spark. An ideal committer will have contributed broadly throughout the project, and have contributed at least one major component where they have taken an “ownership” role. An ownership role means that existing contributors feel that they should run patches for this component by this person.
2. Quality of contributions: Committers more than any other community member should submit simple, well-tested, and well-designed patches. In addition, they should show sufficient expertise to be able to review patches, including making sure they fit within Spark’s engineering practices (testability, documentation, API stability, code style, etc). The committership is collectively responsible for the software quality and maintainability of Spark. Note that contributions to critical parts of Spark, like its core and SQL modules, will be held to a higher standard when assessing quality. Contributors to these areas will face more review of their changes.
3. Community involvement: Committers should have a constructive and friendly attitude in all community interactions. They should also be active on the dev and user list and help mentor newer contributors and users. In design discussions, committers should maintain a professional and diplomatic approach, even in the face of disagreement.

Apache Flink

Committer candidates are community members who have made good contributions over an extended period of time and want to continue their contributions.

Community contributions include helping to answer user questions on the mailing list, verifying release candidates, giving talks, organizing community events, and other forms of evangelism and community building. The “Apache Way” has a strong focus on the project community, and committers can be recognized for outstanding community contributions even without any code contributions.

Code/technology contributions include contributed pull requests (patches), design discussions, reviews, testing, and other help in identifying and fixing bugs. Especially constructive and high quality design discussions, as well as helping other contributors, are strong indicators.

Apache Hive

• help out users and other developers on the mailing lists, in JIRA, and in IRC
• review and test the patches submitted by others; this can help to offload the burden on existing committers, who will definitely appreciate your efforts
• participate in discussions about releases, roadmaps, architecture, and long-term plans
• help improve the website and the wiki
• participate in (or even initiate) real-world events such as user/developer meetups, papers/talks at conferences, etc
• improve project infrastructure in order to increase the efficiency of committers and other contributors
• help raise the project's quality bar (e.g. by setting up code coverage analysis) as much as possible, keep your activity sustained rather than sporadic

Important points to notice:

• it took a while for him to become a committer: we'd like to make sure that all committers are truly dedicated to the role
• after becoming a committer, he began fulfilling the role by actively reviewing and committing many patches from others (even more than those he continued to author himself!), and sustained that energy over time
• we're using a narrow quantitative measure here (patch count) purely for the purpose of visualizing activity level over time; what we're really interested in are quality and value brought to the project across a wide range of activities (for example, the committer in this case also volunteered to serve as release manager for multiple releases of Hive, starting even before becoming a committer)

[michael-s-molina]

Thank you very much @rusackas for the discussion!

Thanks, @ktmud for the helpful references.

I think it's important to have standards that are clear, measurable, and at the same time consider subjective but important aspects of being a committer.

One possible idea would be to create two main categories with statements that we consider important and set a minimum score for both categories.

Objective/measurable requirements (minimum score is X/100):

• At least X PRs merged
• Submitted and implemented a SIP
• A track record of at least X months since their first commit was merged.
• At least X PRs reviewed (gray checkmark)
• ...

Subjective requirements (minimum score is X/100)::

• Submits simple, well-tested, and well-designed code.
• Constructive and friendly attitude in all community interactions.
• In design discussions, maintain a professional and diplomatic approach, even in the face of disagreement.
• Engages in the Superset community in various fora
• ...

After defining our statements, it would be just a matter of assigning weights to each statement and computing the score for each category. A committer should pass in both categories. It's not a perfect system, but it's clear, measurable, and considers subjective aspects 😉

[eschutho]

Submits simple, well-tested, and well-designed code.
Constructive and friendly attitude in all community interactions.

+1 on the inclusion of code quality and working well with others

[geido]

Thanks for the reference @ktmud that's really helpful.

Objective/measurable requirements (minimum score is X/100):

At least X PRs merged
Submitted and implemented a SIP
A track record of at least X months since their first commit was merged.
At least X PRs reviewed (gray checkmark)

@michael-s-molina I agree a lot with a predictable and consistent scoring. I think it's the best approach to keep subjectivity out of the equation. The only point I think it's unnecessary here is the SIP submission.

Subjective requirements (minimum score is X/100)

However, for the subjective requirements I believe it's going to be hard to objectively score them as they are subjective indeed. I think these can mentioned in the guidelines, so that each PMC can consider them when giving their personal vote.

[michael-s-molina]

The only point I think it's unnecessary here is the SIP submission.

Just to check if we're in sync here. In the example above the SIP is not mandatory, but it gives points to a potential committer if they submitted and implemented one. For example, let's say we only have those 4 statements in the objective category, the minimum score is 75 and each statement has an equal weight of 25.

This would be considered suitable (75/100):
[X] At least X PRs merged
[ ] Submitted and implemented a SIP
[X] A track record of at least X months since their first commit was merged.
[X] At least X PRs reviewed (gray checkmark)

This too (75/100).
[ ] At least X PRs merged
[X] Submitted and implemented a SIP
[X] A track record of at least X months since their first commit was merged.
[X] At least X PRs reviewed (gray checkmark)

I'm not sure if you thought it was mandatory. Let me know.

However, for the subjective requirements I believe it's going to be hard to objectively score them as they are subjective indeed.

The way I'm thinking about this is when someone wants to nominate a new committer, they would include their own personal evaluation of the subjective statements in the nomination email.

For example, let's say I want to nominate John Doe as a new committer. My nomination email would be something like this:

I would like to nominate John Doe ...

Objective/measurable requirements (75/100):
[X] At least X PRs merged
[ ] Submitted and implemented a SIP
[X] A track record of at least X months since their first commit was merged.
[X] At least X PRs reviewed (gray checkmark)

Subjective requirements (75/100):
[X] Submits simple, well-tested, and well-designed code.
[X] Constructive and friendly attitude in all community interactions.
[X] In design discussions, maintain a professional and diplomatic approach, even in the face of disagreement.
[ ] Engages in the Superset community in various fora

If a voter does not agree with the score (in a way that would make the score to be less than the minimum), they would just downvote the nomination as we do right now.

[rusackas]

Hi all,

Another little wrinkle in this discussion crossed my mind today. In this project, we have an annoying limitation that folks can't be "assigned" to issues or PRs unless they're a committer or have commented in the concerned thread. In the past, we've let in a few folks act in a Product Management capacity at Superset-centric orgs. Should there be a potential second door for these sorts of folks who are contributing to the project in non-code ways? For example, I'm seeing a bunch of Issues that I'd love to assign to @lauderbaugh, but I can't since he's not a committer. Curious everyone's thoughts on this matter.

[rusackas]

I've made a whole bunch of edits to the markdown block above, based on everyone's input in this thread. Particular shoutout to @ktmud to see how other communities handle these matters. If the above looks good to all involved, I'll propose it to the dev list for lazy consensus, and then either adjust or put it on the wiki!

[michael-s-molina]

@rusackas Reading the text gives me the impression that any person with only the Must Have requirements would be eligible. In my opinion, to be eligible someone has to achieve a minimum of objective and subjective requirements.

[rusackas]

Ahh, fair point. I'll take another shot at re-wording those headers....

[rusackas]

Let me know if the current format seems clearer. Happy to keep dialing this in, and appreciative of your feedback!

[geido]

@rusackas I tend to disagree with this

On a case-by-case basis, committership (i.e. write access to the Github repo) may be warranted for individuals who may not meet the above code contribution standards, but instead will be considered on the basis of other community-driven qualifications.

We should rather find ways to get them involved in the places where their input is beneficial. Giving the ability to merge PRs to someone who doesn't have coding skills or does not know the codebase is a risky path imho.

[rusackas]

Sadly, between the lack of granularity of Apache roles, and the same lack in Github permission tiers, there's no way to grant someone access to creating/editing labels, closing issues, managing project boards, etc., without granting them the Merge button along with it. I recognize the responsibility that comes with this, which is why I think only some candidates would be a good fit here. Thus the "case by case" basis - I think it's fair game to downvote candidates that aren't worthy of all the rights, and the discipline that comes with it.

That said, since this is up for lazy consensus as penned, and it's up for lazy consensus right now, I'm hesitant to edit this out unless others share the feeling. If you feel strongly, and/or others agree, I don't particularly mind if you object on the email thread, I can remove that paragraph, and we can begin the process anew.

[rusackas]

Consensus was reached on the mailing list, and the proposed content was added to the wiki. I'll lock this thread, and if anyone sees a need to make changes to the guidelines, let's just start a fresh discussion. Thanks!