chore: set up ruff as a new linter/formatter

[mistercrunch]

SUMMARY

Ruff is great, and it's fast. All the cool kids are using it.

#edit -putting my latest comment as PR body:

This PR is meant as an experiment around whether it would make sense to add this package as part of the stack. I'll try and summarize my findings and recommendation after playing with it.

relevant: https://docs.astral.sh/ruff/faq/

Early impression are:

on the positive:

• super fast and centralized in a comprehensive CLI (pretty much fully replaces pycln, pyupgrade, isort, black)
• simplifying the code quality stack quite a bit. Many of those tools themselves are adopting ruff (pylint uses in its repo for instance). There's fair amount of cognitive load in dealing with disjointed tools with their own cli, config and quirks.
• powerful / convenient features like --add-no-qa --ignore-noqa and --fix
• it's taking over the python code-quality world fast, federating everything in a nifty, comprehensive toolkit

on the negative:

• while it covers at lot, it's intricately incomplete, pylint is the less covered source where it cannot replace it fully at this time, for the others, it seems like ruff is "enough of a replacement" to justify rip & replace - here I'm thinking disabling duplicate rules with pylint might speed pylint sifnificantly
• I hit some infinite linting loops and deadends in some area, pushing me to disable black, isort and pyupgrade. Our codebase is large, and while ruff is 99% compatible with all this, 1% is still a significant thing
• this won't replace pylint as the coverage is still too low and pylint is powerful. I was thinking we may be able to disable rules in pylint that are covered and speed it up in the process

Overall I think the recommendation is likely to be "we should do this now" or "we should probably revisit in a little while", will update here once I form an opinion.

[villebro]

Wait what, black is no longer the new black? I took a look at ruff, pretty great perf and overall promises. But let's test this carefully, as even many mature linters fall apart on the Superset codebase..

[john-bodley]

It seems like perf is a big sell,

⚡️ 10-100x faster than existing linters (like Flake8) and formatters (like Black)

however I think Black only takes a few seconds to runs and thus any speed up is irrelevant from a UX perspective in my mind. I've never once thought "you know the main problem with Black is that it's too slow".

@mistercrunch would you mind including some links as to why we should be adopting this over Black et al.? Sure some of the auto-formatting looks great—including being more opinionated about blank lines—however on the flip side the one import per line (with the addition of the # noqa: F401 comment) seems—per the package name— a tad ruff.

[mistercrunch]

Hey I should have written more originally when submitting this draft, but this is meant as an experiment around whether it would make sense to add this package as part of the stack. I'll try and summarize my findings and recommendation after playing with it.

Early impression are:

on the positive:

• super fast
• mostly replaces a lot of things (pylint, pyupdate, flake8, black, parts of pylint), simplifying the over stack quite a bit. Many of those tools themselves are adopting ruff (pylint uses in its repo for instance)
• powerful / convenient features like --add-no-qa
• it's taking over the python code-quality world fast, federating everything in a nifty, comprensive toolkit

on the negative:

• while it covers at lot, it's intricately incomplete, pylint is the less covered source where it cannot replace it fully at this time
• hit some infinite linting loops and deadends in some area, pushing me to disable black, isort and pyupgrade. Our codebase is large, and while ruff is 99% compatible with all this, 1% is still a significant thing

Overall I think the recommendation is likely to be "we should do this now" or "we should probably revisit in a little while "

[mistercrunch]

@villebro @john-bodley - my recommendation is to move forward with this, speed and centralization of the tooling make things really nice. Big wins include:

• speed - a fast CLI, you never wait on it to do anything, unlike pylint (takes minutes to lint the codebase) and seconds to operate in your IDE on a single file
• one modern/comprehensive CLI with all the knobs and buttons you need. A lot of the other CLIs we're replacing are clunky and intricately different, don't always agree with each other, ...
• quality of life features: --add-noqa and --ignore-noqa and having a --fix make things easy
• momentum - this thing is getting better everyday

[hughhhh]

+1 to @mistercrunch comments on upgrading to this package.

@john-bodley for formatting >99.9% compatible to black reading the release statements

Optimized for adoption. Ruff’s formatter is designed as a drop-in replacement for Black. On Black-formatted Python projects, Ruff achieves >99.9% compatibility with Black, as measured by changed lines. When formatting the Django codebase, for example, Ruff and Black only differ on 34 out of 2,772 files. Focusing on compatibility enables projects to adopt the Ruff formatter with minimal disruption.
https://astral.sh/blog/the-ruff-formatter#

[mistercrunch]

This one will conflict with everything (touching 574 files), I'd love to merge quickly(!)

[mistercrunch]

Great resources here too -> https://docs.astral.sh/ruff/faq/

For those interested in the story behind the project -> https://www.youtube.com/watch?v=LCva0NOM2-o

[betodealmeida]

Let's do this!

[mistercrunch]

[andy-clapson]

@mistercrunch did you try out uv ever? same folks and pretty wicked too.

[mistercrunch]

@andy-clapson first I hear of it, but OMG if this delivers on its promises THE FUTURE IS HERE. Oh is it the same dude!? Yes!

We depend on pip-compile-multi, which isn't really well supported anymore. If uv has some support for managing/inheriting from multiple reqs file like we have in requirements/ it'd be really tempting to try and take that jump.

[mistercrunch]

For the record, I opened an issue asking about our use cases -> astral-sh/uv#5487

[andy-clapson]

Yeah, we started using it a few months back and it's extremely fast.
pip-compile-multi is new to me, though - I'm less sure about that.