Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Update linters and essential configurations #743

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pre-commit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
# Skip terraform_tflint which interferes to commit pre-commit auto-fixes
- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
with:
python-version: '3.9'
python-version: '3.10'
- name: Execute pre-commit
uses: pre-commit/action@9b88afc9cd57fd75b655d5c71bd38146d07135fe # v2.0.3
env:
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
tests/results/*

# Python
__pycache__/
*.py[cod]
node_modules/
83 changes: 82 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ repos:

# Dockerfile linter
- repo: https://github.com/hadolint/hadolint
rev: v2.12.1-beta
rev: v2.13.1-beta
hooks:
- id: hadolint
args: [
Expand All @@ -66,3 +66,84 @@ repos:
- id: prettier
# https://prettier.io/docs/en/options.html#parser
files: '.json5$'


##########
# PYTHON #
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With pre-commit, the primary ordering should be by whether something is faster or slower. Putting instant checks at the beginning improves responsiveness, with the slow ones stuffed at the end. The second factor is formatters vs. linters. When a bunch of formatters change some type of files, they should be executed before the linters that check the same files.

The ecosystem is something that could be used to bundle similar checks within those groups. But it's definitely not something that would be top level.

##########

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.4
hooks:
- id: ruff
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation. This moment when you add more checks but not yamllint 😂

args: [--fix]
types_or: [python, pyi]
- id: ruff-format
types_or: [python, pyi]
args: [--config, format.quote-style = 'single', --config, line-length = 100]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a good reason(s) to deviate from linter's default settings.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not because of the defaults (for which sometimes there are good reasons to change) but because other tools wouldn't pick this up, it should be avoided (with a few well-defined exceptions like the one I've got for MyPy).

For this reason, every low-level tool must be configured via its own autoloaded config file name. Otherwise, everybody would have to duplicate this config in their editors, IDEs, other tools and sometimes CI, and would have toe know upfront that this is necessary. Being surprised by unexpected behaviors when they don't. I have some notes on this in #742.

I wouldn't say that setting such a long line setting has a merit, though. I'm a firm believer that in the name of readability/inclusivity/maintainability, code should be readable in columns. Just like any text, typography can govern column width for code too. This is directly connected to readability (which is why magazines and newspapers don't print lines from one side of the page to the others, and some books / catalogs use columns). Typographically optimal columns are between 50 and 75 chars (opinions vary, but mostly close to this range): https://baymard.com/blog/line-length-readability. We have a long-standing convention of 79 that exceeds it a bit, but isn't as critical. It allows for columns in editors for working on multiple files while still using enlarged fonts. It also allows for side-by-side diffs. The lines fit on the screen in these settings as well as allow for column-based top-down reading (which is another thing humans do with text naturally, by habit).


- repo: https://github.com/pycqa/isort
rev: 5.13.2
hooks:
- id: isort
name: isort
args: [--force-single-line, --profile=black]

- repo: https://github.com/asottile/add-trailing-comma
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one might be fine to combine with Ruff. But you'll have to put it first and also check if they don't cancel each other out in the way they're configured.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't

I wouldn't be so sure. I've seen corner cases with various formatters where they'd do this very occasionally under certain conditions. It's good that we haven't observed such a behavior here. I'm just saying that this should be kept in mind for the future in case it does happen.

If Ruff does not implement a similar capability, it stands to reason that it's okay to keep this formatter. If it does, it might not make sense.

But if you decide to keep this one, make sure to move it along with any other small narrow-scoped formatters before the Ruff run.

rev: v3.1.0
hooks:
- id: add-trailing-comma

- repo: https://github.com/pre-commit/mirrors-autopep8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's usually unnecessary to use multiple generic code formatters, as they tend to step on each other's toes, resulting in infinite reformatting of the same things back and forth. Choose one and stick with it. It's typically fine with a few smaller ones that change small non-intersecting aspects of files, though. But they may have to be ordered correctly.

rev: v2.0.4
hooks:
- id: autopep8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Ruff in formatting mode, this is probably unnecessary. I'm yet to compare their output, but the recommendation is going to be the same as with pylint for now.

args:
- -i
- --max-line-length=100
Comment on lines +101 to +103
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of interest: why do you sometimes use args: […] and sometimes split args arrays to multiline? Such inconsistency is a bit confusing as it adds impression that this is done for a reason.

Copy link
Collaborator Author

@MaxymVlasov MaxymVlasov Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sometimes it depends on how hook configured - IE

[--config, format.quote-style = 'single', --config, line-length = 100]

will be actually more readble as

[
  --config, format.quote-style = 'single',
  --config, line-length = 100
]

not

- --config
- format.quote-style = 'single'
- --config
- line-length = 100

But I don't remember why I can't use multiline array with [], maybe I just not set it in first place when set it a long time ago.
Second reason - our God of code reusage across projects - copy-paste :D
Nobody never reevaluated it, so it is as it is till now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is rather subjective. I'd argue that [] is the least readable way.

Actually, for --long-args, it's best to use = not a whitespace. This is useful not only in this config but in Python code too. Since you're integrating autoformatters, they will produce such weird constructs with args disconnected from their values unless you use equals that bundles them into the same string.


# Usage: http://pylint.pycqa.org/en/latest/user_guide/message-control.html
- repo: https://github.com/PyCQA/pylint
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I experimented with configuring this linter for a while, but I'm hitting a few bugs within.

You're also integrating Ruff which reimplements most if not all of pylint rules in it.

It doesn't really make sense to have multiple linters report the same violation multiple times within each pre-commit run.

With that in mind, I recommend abandoning pylint for now and focusing on making sure all the corresponding rules in Ruff are enabled instead.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess it's a worthwhile suggestion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I'm planning to research replacing many things with Ruff in my projects but just didn't get to do a full comparison.

Copy link
Collaborator Author

@MaxymVlasov MaxymVlasov Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, ruff has rules that should be manually enabled? Didn't know it

From what I saw - pylint in its current configuration provides additional checks which are not covered by ruff in its current configuration.
At the same time, it totally makes sense to check is ruff and we-make-styleguide both fully cover pylint out of the box or at least able to be configured in such way.
If you do such research - that's would be lovely, but till I want to have all them in place, as better to catch all possible issues now, than deal with them in 6-12months when they detection will be implemented in ruff etc.
pylint check is relatively fast, so I want to preserve it for now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pylint check is relatively fast

😆 this is the slowest checker I know. This is because it goes beyond static analysis and does some dynamic checking as well.

As for ruff, I wanted to research it more. And yes, there's rules that are disabled by default in all the linters I've ever met (flake8, pylint, ruff). Some updates to pylint sometimes move the rules from default to extensions too — this is why I prefer an explicit config to make sure that it doesn't suddenly stop checking something just because of a version bump.

While I do like pylint in general, I tried it locally on this codebase and there's bugs in it that block really adding it in full capacity. Hence, the idea to delay adding it. I may end up debugging the issue for my other projects. But in general, many people stick with flake8 because it's not as slow and easier to extend. We do run it in ansible-test, though.

My initial strategy for adopting Ruff would be enabling as much as possible there, and only if some rule is not ported, then run flake8/pylint (with plugins) only checking those rules, to reduce the overhead. It's good for DX when the same problem isn't reported by 5 different checkers.

rev: v3.3.3
hooks:
- id: pylint
args:
- --disable=import-error # E0401. Locally you could not have all imports.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid putting linter settings into CLI args at all costs. This is incompatible with literally everything that would run said linters. Always keep them in tool-specific configs.

- --disable=fixme # W0511. 'TODO' notations.
- --disable=logging-fstring-interpolation # Conflict with "use a single formatting" WPS323
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop this. Logging actually shouldn't be using f-strings.

- --disable=ungrouped-imports # ignore `if TYPE_CHECKING` case. Other do reorder-python-imports
- --disable=R0801 # Similar lines in 2 files. Currently I don't think that it possible to DRY hooks unique files boilerplate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't use numeric rule codes in pylint. It has human-readable names that make much more sense than random digit sequences.

Also, instead of disabling the rules, many pylint and flake8 rules can remain enabled because they allow tweaking their settings. I know for a fact that this duplication rule allows increasing the number of lines to take into account.

Copy link
Collaborator Author

@MaxymVlasov MaxymVlasov Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
I can only pray that pylint maintainers will add human-readable names in their error msgs, and not require users to google if they exist and how they named if so

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have this configured by default 🤷‍♂️ You should've just used my config.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, Ruff does not allow using those names in ignores, while the rules may have human-readable names. There's a feature request about it.


- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.14.1
hooks:
- id: mypy
additional_dependencies:
- types-PyYAML
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why four rather than two spaces indentation at this line? 😕

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

args: [
--ignore-missing-imports,
--disallow-untyped-calls,
--warn-redundant-casts,
Comment on lines +124 to +126
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing should be disabled globally. Suppressions must be granular, which can be implemented in the config and in-module.

]

- repo: https://github.com/wemake-services/wemake-python-styleguide
rev: 1.0.0
hooks:
- id: wemake-python-styleguide
args:
- --allowed-module-metadata=__all__ # Default to ''
- --max-local-variables=6 # Default to 5
- --max-returns=6 # Default to 5
- --max-imports=15 # Default to 12
# https://wemake-python-stylegui.de/en/latest/pages/usage/violations/index.html
- --extend-ignore=
E501 <!-- line too long (> 79 characters). Use 100 -->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configurable rules shouldn't be ignored really.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It covered by other linters, so deduplicate similar errors
  2. Idk why it not provide link to docs which describes possibilities how to deal with violation as many hooks do 🤔 It's not easy to google them. This one https://www.flake8rules.com/rules/E501.html doesn't include any suggestions about configuration -_-

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WPS can show shortlinks to the rules, you just didn't enable it 🤷‍♂️

This is the setting: https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-max-line-length.

But anyway, it doesn't make sense to have such things as CLI args. I can send in a good config structure where it's more apparent what should go where.

WPS115 <!-- Found upper-case constant in a class. All occurrences are false positive -->
WPS226 <!-- Found string literal over-use - append > 3 -->


exclude: |
(?x)
# Ignore deprecated hook
(^src/pre_commit_terraform/terraform_docs_replace.py$
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should instead go into the flake8 config.

)
3 changes: 3 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,7 @@
"code-block-style": false
},
"markdown.validate.enabled": true,
"python.analysis.extraPaths": [
"./src"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, include the tests dir.

],
}
7 changes: 2 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,12 @@ build-backend = 'hatchling.build'
name = 'pre-commit-terraform'
classifiers = [
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
Comment on lines -12 to -16
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's best to list envs tested in CI here.

'Programming Language :: Python :: 3 :: Only',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
]
description = 'Pre-commit hooks for terraform_docs_replace'
description = 'Pre-commit hooks for Terraform'
dependencies = []
dynamic = [
'urls',
Expand Down
4 changes: 2 additions & 2 deletions src/pre_commit_terraform/__main__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
"""A runpy-style CLI entry-point module."""

from sys import argv, exit as exit_with_return_code
from sys import argv
from sys import exit as exit_with_return_code

from ._cli import invoke_cli_app


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will probably go away with a proper config.

return_code = invoke_cli_app(argv[1:])
exit_with_return_code(return_code)
11 changes: 4 additions & 7 deletions src/pre_commit_terraform/_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,9 @@
from sys import stderr

from ._cli_parsing import initialize_argument_parser
from ._errors import (
PreCommitTerraformBaseError,
PreCommitTerraformExit,
PreCommitTerraformRuntimeError,
)
from ._errors import PreCommitTerraformBaseError
from ._errors import PreCommitTerraformExit
from ._errors import PreCommitTerraformRuntimeError
from ._structs import ReturnCode
from ._types import ReturnCodeType

Expand All @@ -28,8 +26,7 @@ def invoke_cli_app(cli_args: list[str]) -> ReturnCodeType:
raise
except PreCommitTerraformRuntimeError as unhandled_exc:
print(
f'App execution took an unexpected turn: {unhandled_exc !s}. '
'Exiting...',
f'App execution took an unexpected turn: {unhandled_exc !s}. Exiting...',
file=stderr,
)
return ReturnCode.ERROR
Expand Down
1 change: 0 additions & 1 deletion src/pre_commit_terraform/_cli_subcommands.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from . import terraform_docs_replace
from ._types import CLISubcommandModuleProtocol


SUBCOMMAND_MODULES: list[CLISubcommandModuleProtocol] = [
terraform_docs_replace,
]
Expand Down
4 changes: 2 additions & 2 deletions src/pre_commit_terraform/_errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ class PreCommitTerraformBaseError(Exception):


class PreCommitTerraformRuntimeError(
PreCommitTerraformBaseError,
RuntimeError,
PreCommitTerraformBaseError,
RuntimeError,
):
"""An exception representing a runtime error condition."""

Expand Down
13 changes: 8 additions & 5 deletions src/pre_commit_terraform/_types.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
"""Composite types for annotating in-project code."""

from argparse import ArgumentParser, Namespace
from typing import Final, Protocol
from argparse import ArgumentParser
from argparse import Namespace
from typing import Final
from typing import Protocol

from ._structs import ReturnCode


ReturnCodeType = ReturnCode | int


Expand All @@ -16,12 +17,14 @@ class CLISubcommandModuleProtocol(Protocol):
"""This constant contains a CLI."""

def populate_argument_parser(
self, subcommand_parser: ArgumentParser,
self,
subcommand_parser: ArgumentParser,
) -> None:
"""Run a module hook for populating the subcommand parser."""

def invoke_cli_app(
self, parsed_cli_args: Namespace,
self,
parsed_cli_args: Namespace,
) -> ReturnCodeType | int:
"""Run a module hook implementing the subcommand logic."""
... # pylint: disable=unnecessary-ellipsis
Expand Down
28 changes: 17 additions & 11 deletions src/pre_commit_terraform/terraform_docs_replace.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import os
import subprocess
import warnings
from argparse import ArgumentParser, Namespace
from argparse import ArgumentParser
from argparse import Namespace
from typing import Final

from ._structs import ReturnCode
from ._types import ReturnCodeType


CLI_SUBCOMMAND_NAME: Final[str] = 'replace-docs'


Expand All @@ -18,14 +18,20 @@ def populate_argument_parser(subcommand_parser: ArgumentParser) -> None:
'replace the entire README.md file each time.'
)
subcommand_parser.add_argument(
'--dest', dest='dest', default='README.md',
'--dest',
dest='dest',
default='README.md',
)
subcommand_parser.add_argument(
'--sort-inputs-by-required', dest='sort', action='store_true',
'--sort-inputs-by-required',
dest='sort',
action='store_true',
help='[deprecated] use --sort-by-required instead',
)
subcommand_parser.add_argument(
'--sort-by-required', dest='sort', action='store_true',
'--sort-by-required',
dest='sort',
action='store_true',
)
subcommand_parser.add_argument(
'--with-aggregate-type-defaults',
Expand All @@ -51,8 +57,9 @@ def invoke_cli_app(parsed_cli_args: Namespace) -> ReturnCodeType:

dirs = []
for filename in parsed_cli_args.filenames:
if (os.path.realpath(filename) not in dirs and
(filename.endswith(".tf") or filename.endswith(".tfvars"))):
if os.path.realpath(filename) not in dirs and (
filename.endswith('.tf') or filename.endswith('.tfvars')
):
dirs.append(os.path.dirname(filename))

retval = ReturnCode.OK
Expand All @@ -64,13 +71,12 @@ def invoke_cli_app(parsed_cli_args: Namespace) -> ReturnCodeType:
if parsed_cli_args.sort:
procArgs.append('--sort-by-required')
procArgs.append('md')
procArgs.append("./{dir}".format(dir=dir))
procArgs.append('./{dir}'.format(dir=dir))
procArgs.append('>')
procArgs.append(
'./{dir}/{dest}'.
format(dir=dir, dest=parsed_cli_args.dest),
'./{dir}/{dest}'.format(dir=dir, dest=parsed_cli_args.dest),
)
subprocess.check_call(" ".join(procArgs), shell=True)
subprocess.check_call(' '.join(procArgs), shell=True)
except subprocess.CalledProcessError as e:
print(e)
retval = ReturnCode.ERROR
Expand Down
Loading