chore: add pylint to pre-commit hook

[mistercrunch]

I got bit by the old pylint failing in CI again recently. pylint is too slow for IDEs
so I turn it off since it made vim too sluggish to my taste. Feels about
right as a pre-commit hook, I'd rather get that too take a moment than
waiting for a CI cycle.

Not sure why it wasn't there in the first place, too slow?

We should also consider moving to ruff and streamlining all this, but ruff
doesn't support all the pylint rules [yet].

[mistercrunch]

NOTE: this other action here https://github.com/apache/superset/blob/master/.github/workflows/pre-commit.yml#L38 will cover for this one.

[john-bodley]

@mistercrunch per here it seems like this pre-commit hook will only work locally and will be ineffective in a remote CI setting.

[rusackas]

Seems good to me, but I think @john-bodley might be the right stakeholder approver here.

[mistercrunch]

@john-bodley I think it should be fine as long as I install it in the setup-backend reusable action, it'll install the right/same version as what should be on every dev env too.

I just realized it's only pylinting the superset/ folder, not the tests/ or scripts/. It's really resource-intensive though, brings my M2 macbook to its knees. I think I'll give ruff a shot as it's blazing fast and catches (and autofixes) a bunch of things that other tools don't. They are crushing it on their roadmap and eventually should fully replace pylint/isort/black/pydantic/ .... Coming up in another PR soo

[mistercrunch]

I think turning this on as pre-commit is a clear win if we're going to enforce pylint in CI. It's a bit slow, but pre-commit is a good place to pay the cost of verification (CI is too late, and pylint IDE integrations are sluggish in my experience)

[mistercrunch]

Can I get the @john-bodley seal of approval :)?

[john-bodley]

@mistercrunch regarding your comment,

We should also consider moving to ruff and streamlining all this, but ruff
doesn't support all the pylint rules [yet].

There's a few WIP PRs—by @betodealmeida (I believe) and myself—which have custom Pylint extensions so I'm not sure if/when we'll actually can switch off Pylint as I believe Ruff doesn't support custom lint rules.

[mistercrunch]

@john-bodley I don't think we should deprecate pylint, it seems like it will be a long time before ruff gets to covering most of pylint -> astral-sh/ruff#689 . This means it'll probably take years before it's mostly covered. While pylint is slow, it's a great tool with decades of evolution, and I doubt ruff will even attempt to get to 80%+ there. Unclear how much value is in the long tail (?) But not knowing we should totally keep it going.

One thing that we could consider though is disabling the specific (200+) pylint rules where ruff already covers currently. This could allow pylint to perform better as it's pretty slow currently. This could accelerate commit hooks execution (usually that's not bad as is currently) but significantly speed up pre-commit run --all, which runs in CI and should probably run when doing big rebase/merges.