-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Description
Summary
Following the recent type improvements in #3369 and #3370, I'd like to propose updating the recommended type annotation style for Luigi parameters.
Background
In #3370, I added @dataclass_transform support and a .pyi stub file. At that time, I was influenced by the existing mypy plugin (#3315) which uses the s: str = Parameter() style, so I followed the same pattern in the documentation examples.
However, after further investigation, I found that the s: Parameter[str] = Parameter() style works better across all major type checkers.
However, after further investigation, I found that the s: Parameter[str] = Parameter() style works better and doesn't require any of these workarounds - the __get__ and __set__ descriptor protocol handles everything.
Current Style
class MyTask(luigi.Task):
s: str = luigi.Parameter()
n: int = luigi.IntParameter()Proposed Style
class MyTask(luigi.Task):
s: Parameter[str] = luigi.Parameter()
n: IntParameter = luigi.IntParameter()Type Checker Comparison
Assignment Error at Class Definition
| Type Checker | Style 1 (s: str) |
Style 2 (s: Parameter[str]) |
|---|---|---|
| pyright | ❌ Error | ✅ No error |
| mypy | ❌ Error | ✅ No error |
| pyrefly | ❌ Error | ✅ No error |
| ty | ❌ Error | ✅ No error |
Type Inference (Both styles work correctly)
| Access | Style 1 | Style 2 |
|---|---|---|
task.s |
str ✅ |
str ✅ |
task.n |
int ✅ |
int ✅ |
Task.__init__ |
(s: str, n: int) ✅ |
(s: str, n: int) ✅ |
Task.s |
str |
Parameter[str] (more accurate) |
Wrong Type Detection in init (Both styles work correctly)
Both styles correctly detect this error:
task = MyTask(s="hello", n="42") # Error: n should be int!
| Type Checker | Style 1 | Style 2 |
|---|---|---|
| pyright | ✅ Detects error | ✅ Detects error |
| mypy | ✅ Detects error | ✅ Detects error |
| pyrefly | ✅ Detects error | ✅ Detects error |
| ty | ✅ Detects error | ✅ Detects error |
Why Style 2 is Better
- No type errors - Works out of the box with all major type checkers
- No
__new__override needed - Style 1 requires__new__to returnTto trick type checkers (feat: Add generic type support to Parameter classes #3369) - No
.pyistub file needed - Style 1 requiresparameter.pyifor proper type inference (feat: add parameter.pyi for type hinting parameters #3370) - No mypy plugin needed - Style 1 relies on custom mypy plugin (Implement mypy plugin for
luigi.Task#3315) - More accurate class access -
Task.scorrectly showsParameter[str] - Same type safety -
__init__type checking works identically
Simplification Opportunity
If we adopt Style 2 as the recommended approach, we could potentially:
- Remove the
__new__overrides added in feat: Add generic type support to Parameter classes #3369 - Simplify or remove
parameter.pyiadded in feat: add parameter.pyi for type hinting parameters #3370 - Deprecate the mypy plugin (Implement mypy plugin for
luigi.Task#3315) in favor of native@dataclass_transformsupport
Suggested Actions
- Update documentation examples to use
Parameter[T]style - Consider simplifying the type infrastructure (remove
__new__hacks, etc.) - Mention both styles in docs (with Style 2 as recommended)
Related
- feat: Add generic type support to Parameter classes #3369 - Add generic type support to Parameter classes (adds
__new__overrides) - feat: add parameter.pyi for type hinting parameters #3370 - Add parameter.pyi for type hinting parameters
- Implement mypy plugin for
luigi.Task#3315 - mypy plugin (uses Style 1)