Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][Macro] Emit error if AccessorMacro is applied to node that is not a variable #2624

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

RayZhao1998
Copy link

@RayZhao1998 RayZhao1998 commented Apr 25, 2024

This PR resolves #2207 .

Copy link
Member

@ahoppen ahoppen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on @RayZhao1998 and welcome to swift-syntax’s development! 🙏🏽

Instead of repeating the logic of which attributes we need to remove, I think we should collect the AttributeSyntax nodes of the attributes that we did indeed expand in MacroApplication and then remove those nodes for the AttributeRemover.

@RayZhao1998
Copy link
Author

I've made some changes. But as mentioned in the issue, it fails testTypeWrapperTransform now. Please help me check whether this test is correct.

@RayZhao1998 RayZhao1998 force-pushed the feature/assertMacroExpansion-notvariable-diagnostics branch from 82da284 to 09252c7 Compare April 26, 2024 18:38
@RayZhao1998 RayZhao1998 marked this pull request as ready for review April 26, 2024 18:39
@ahoppen
Copy link
Member

ahoppen commented Apr 26, 2024

That’s why I mentioned

Instead of repeating the logic of which attributes we need to remove, I think we should collect the AttributeSyntax nodes of the attributes that we did indeed expand in MacroApplication and then remove those nodes for the AttributeRemover.

The problem here is that we are essentially duplicating the logic of whether a macro can be applied to a given declaration here and that logic doesn’t quite match what the rest of MacroApplication does.

@RayZhao1998
Copy link
Author

RayZhao1998 commented Apr 27, 2024

Sorry, I misunderstood before.

I ran into a problem. Is there a way to check equal between two AttributeSyntax nodes? It seems the AttributeSyntax.id will change after expanding. Now I'm using the attributeName to compare but I don't think it's a reasonable way.

@RayZhao1998 RayZhao1998 marked this pull request as draft April 27, 2024 16:57
@RayZhao1998 RayZhao1998 force-pushed the feature/assertMacroExpansion-notvariable-diagnostics branch from 44216a3 to d870f06 Compare April 27, 2024 17:05
@ahoppen
Copy link
Member

ahoppen commented Apr 29, 2024

Ah, I forgot about that. A proper way of doing this might be to productize #2118 but I don’t think that’s happening in the near future.

I think macro expansions always add code after the location of the macro attribute. So, we should be able to use the attribute’s position to identify it.

Comment on lines 731 to 732
if let macroRole = try? inferAttachedMacroRole(definition: spec.type),
isInvalidAttachedMacro(macroRole: macroRole, attachedTo: declSyntax)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even need to check isInvalidAttachedMacro here? I would think that we should diagnose all macro attributes that didn’t actually expand. Or am I missing something?

Copy link
Author

@RayZhao1998 RayZhao1998 May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found out how it diagnose when AccessorMacro is applied to struct in the compiler and imitated it. Also the diagnostics I'v implemented below is just for attached macros(actually only AccessorMacro).

If we should diagnose all macro attributes that didn't expand, what error messages should we emit, a common error message or the same as what compiler emit? Also I don't know all the cases that attribute won't expand.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made some changes. I've abstracted a method called diagnosticForUnexpandedMacro. Currently, it only implements diagnostics for AccessorMacro. The rest can be left for others to implement.

Sources/SwiftSyntaxMacroExpansion/MacroExpansion.swift Outdated Show resolved Hide resolved
func nodeTypeNameForDiagnostics(allowBlockNames: Bool) -> String? {
public func nodeTypeNameForDiagnostics(allowBlockNames: Bool) -> String? {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think we should make this method public. The entire logic with allowBlockNames is very specific to SwiftParserDiagnsotic. Using SyntaxKind.nameForDiagnostics would probably be the better choice.

Copy link
Author

@RayZhao1998 RayZhao1998 May 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using SyntaxKind.nameForDiagnostics would probably be the better choice.

How to make nameForDiagnostics public? I found the file is generated.

By the way, if the files are generated, should we add generated files to .gitignore?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The files are being generated as described here:

https://github.com/apple/swift-syntax/blob/main/CONTRIBUTING.md#generating-source-code

We need to commit the generated files to resolve a circular dependency: The sources are being generated using swift-syntax itself, which means that you need a version of swift-syntax to generate the sources of swift-syntax. If the generated sources weren’t committed anywhere, you couldn’t build swift-syntax.

Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift Outdated Show resolved Hide resolved
if let index = self.expandedAttributes.firstIndex(where: { expandedAttribute in
expandedAttribute.position == attribute.position
}) {
self.expandedAttributes.remove(at: index)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is removing necessary for correctness or do we just do it to keep the number of elements in self.expandedAttributes low (which is a good reason on its own). Would be worth a comment though, I think.

Copy link
Author

@RayZhao1998 RayZhao1998 May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be incorrect if it's possible that two attributes with same position are actually different during the process. For example, the list still contains an attribute expanded in the last visit, and in the same position there is attribute not expanded in this visit, I think it'll go wrong.

I think we should remove it from the list if it has been checked to make sure it's correct

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I thought we were fine because macros only ever introduce code after the the macro attribute itself, so I thought that the position would still uniquely identify a macro within the source file but thinking about it again, I realize that isn’t true because the macro itself gets removed, which shifts positions forward and thus allows another macro to have the same position.

For example, you could have the following

@NoOpMacroAB
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
sturct Foo {
func foo() {}
}

which expands to

sturct Foo {
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
func foo() {}
}

And the two @MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro have the same location despite being two different macro expansions.


And yes, I think with the removal of attributes from self.expandedAttributes, this should be fine because we always clear that list whenever removing attributes.

I hope that I’m making sense here. I think it would be good to have a test case for what I described above and make sure that we do actually diagnose that MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro can’t be attached to func foo(). Also, feel free to pick a shorter name for MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro. I only picked it to convey the meaning of the attribute without writing an implementation.

@RayZhao1998 RayZhao1998 force-pushed the feature/assertMacroExpansion-notvariable-diagnostics branch from 0287d59 to ad9e5ae Compare May 2, 2024 04:02
Copy link
Member

@ahoppen ahoppen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your continued work on this @RayZhao1998. I think together we are iterating towards a good solution here. Reading through it again, I have some more comments. I think the main one is the one I left on the declaration of expandedAttributes. Could you check if the idea I posted there works or if I missed something?

@@ -633,6 +635,7 @@ private enum MacroApplicationError: DiagnosticMessage, Error {
case accessorMacroOnVariableWithMultipleBindings
case peerMacroOnVariableWithMultipleBindings
case malformedAccessor
case macroAttachedToInvalidDecl(String, String)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add labels for these associated values so it’s easier to tell what the two strings do?

@@ -16,6 +16,7 @@ internal import SwiftOperators
@_spi(MacroExpansion) internal import SwiftParser
public import SwiftSyntax
internal import SwiftSyntaxBuilder
internal import SwiftParserDiagnostics
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these imports still necessary if you use nameKindForDiagnostics? Same for the dependency declaration in the package manifest.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it doesn't need any more.

var fixitMessage: FixItMessage? = nil
switch macroType {
case is AccessorMacro.Type:
if (!attachedTo.is(VariableDeclSyntax.self) && !attachedTo.is(SubscriptDeclSyntax.self)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (!attachedTo.is(VariableDeclSyntax.self) && !attachedTo.is(SubscriptDeclSyntax.self)) {
if !attachedTo.is(VariableDeclSyntax.self) && !attachedTo.is(SubscriptDeclSyntax.self) {

)
}
default:
break
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your error message '\(macroType)' macro cannot be attached to \(declType) is sufficiently generic that we could apply it for all cases, not just the accessor macro attached to a non-variable and non-subscript case.

Copy link
Author

@RayZhao1998 RayZhao1998 May 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's the case, I believe there's no need to diagnose based on the type of macroType or the type of DeclSyntax. A unified diagnosis with message '\(macroType)' macro cannot be attached to \(declType) will suffice.

if let index = self.expandedAttributes.firstIndex(where: { expandedAttribute in
expandedAttribute.position == attribute.position
}) {
self.expandedAttributes.remove(at: index)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I thought we were fine because macros only ever introduce code after the the macro attribute itself, so I thought that the position would still uniquely identify a macro within the source file but thinking about it again, I realize that isn’t true because the macro itself gets removed, which shifts positions forward and thus allows another macro to have the same position.

For example, you could have the following

@NoOpMacroAB
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
sturct Foo {
func foo() {}
}

which expands to

sturct Foo {
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
func foo() {}
}

And the two @MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro have the same location despite being two different macro expansions.


And yes, I think with the removal of attributes from self.expandedAttributes, this should be fine because we always clear that list whenever removing attributes.

I hope that I’m making sense here. I think it would be good to have a test case for what I described above and make sure that we do actually diagnose that MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro can’t be attached to func foo(). Also, feel free to pick a shorter name for MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro. I only picked it to convey the meaning of the attribute without writing an implementation.

@@ -666,6 +671,7 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
/// Store expanded extension while visiting member decls. This should be
/// added to top-level 'CodeBlockItemList'.
var extensions: [CodeBlockItemSyntax] = []
var expandedAttributes: [AttributeSyntax] = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be private.


Also, I just tried to suggest a doc comment for it and couldn’t really come up with a great explanation, which is usually a sign that the design isn’t quite right (maybe you can find a good explanation). Thinking about it made me wonder if it would it be easier to

  • Change this variable to
/// Attributes that look like macro attributes and should be expanded. 
///
/// Attributes are added to this list from `visitAny`. Any attributes that weren’t actually expanded (eg. because they are accessor macros but not attached to a variable or subscript), will be diagnosed from `visitAny`.
var attributesToExpand: [AttributeSyntax] = []
  • In visitAny set this variable to self.macroAttributes(attachedTo: declSyntax)
  • Whenever we expand a macro, remove it from the list (I think we might even do this based on the ID now and don’t need to resort to the position).
  • If there are macros left over in attributesToExpand, diagnose them.
  • After visitAny is done, restore attributesToExpand to the value it had before we assigned it to self.macroAttributes(attachedTo: declSyntax).

Do you think that would work?

Copy link
Author

@RayZhao1998 RayZhao1998 May 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might even do this based on the ID now and don’t need to resort to the position

I found that the id is still different. The macroAttributes method depends on the attachedTo node. In visitAny the declSyntax's root is SourceFileSyntax, while in visit(for example visit(_ node: VariableDeclSyntax)) the node's root is VariableDeclSyntax(itself).

It seems that in visitChildren, the node will have a new layout and generate a new syntax without origin root info. Is this by design?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also delved into the implementation of AttributeSyntax.position. It's the position of the start of this node's leading trivia(not the root). For Attribute, it always represents its index in the AttributeList. So I don't think it's a proper way to compare two attributes by position

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I’ll need to play around with the code myself a little to figure out how we want to continue, but am too busy with other things at the moment. I’ll definitely come back to you but it might take a few weeks.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ahoppen , are there any updates on this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was super busy implementing background indexing in SourceKit-LSP over the last few weeks. I’m hoping that this will calm down now and I’ll allow me to get back to this PR this week.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, I found some time now. Looks like we were really close, the only issue was that in visit(_: VariableDeclSyntax) we needed to call macroAttributes(attachedTo:ofType:) on the original node, not the one that we already processed using super.visit.

Diff

diff --git a/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift b/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift
index 23127070..f350c097 100644
--- a/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift
+++ b/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift
@@ -669,7 +669,7 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
   /// Store expanded extension while visiting member decls. This should be
   /// added to top-level 'CodeBlockItemList'.
   var extensions: [CodeBlockItemSyntax] = []
-  var expandedAttributes: [AttributeSyntax] = []
+  /// Attributes that we are expecting to expand from `visitAny`. As macros are expanded, they should be removed from this list. Any attributes that are still left in this array after a `visitAny` call will generate diagnostics.
+  var attributesToExpand: [(attributeNode: AttributeSyntax, spec: MacroSpec)] = []
 
   init(
     macroSystem: MacroSystem,
@@ -707,6 +707,11 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
       let attributedNode = node.asProtocol(WithAttributesSyntax.self),
       !attributedNode.attributes.isEmpty
     {
+      let previousAttributesToExpand = attributesToExpand
+      defer {
+        attributesToExpand = previousAttributesToExpand
+      }
+      attributesToExpand = self.macroAttributes(attachedTo: declSyntax)
       // Apply body and preamble macros.
       if let nodeWithBody = node.asProtocol(WithOptionalCodeBlockSyntax.self),
         let declNodeWithBody = nodeWithBody as? any DeclSyntaxProtocol & WithOptionalCodeBlockSyntax
@@ -720,14 +725,8 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
       skipVisitAnyHandling.remove(Syntax(declSyntax))
 
       let attributesToRemove = self.macroAttributes(attachedTo: visitedNode)
-      for (attribute, spec) in attributesToRemove {
-        if let index = self.expandedAttributes.firstIndex(where: { expandedAttribute in
-          expandedAttribute.position == attribute.position
-        }) {
-          self.expandedAttributes.remove(at: index)
-        } else {
-          self.diagnosticForUnexpandedMacro(attribute: attribute, macroType: spec.type, attachedTo: declSyntax)
-        }
+      for attribute in attributesToExpand {
+        self.diagnosticForUnexpandedMacro(attribute: attribute.attributeNode, macroType: attribute.spec.type, attachedTo: declSyntax)
       }
       return AttributeRemover(removingWhere: { attributesToRemove.map(\.attributeNode).contains($0) }).rewrite(
         visitedNode
@@ -961,10 +960,10 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
     )
   }
 
-  override func visit(_ node: VariableDeclSyntax) -> DeclSyntax {
-    var node = super.visit(node).cast(VariableDeclSyntax.self)
+  override func visit(_ originalNode: VariableDeclSyntax) -> DeclSyntax {
+    var node = super.visit(originalNode).cast(VariableDeclSyntax.self)
 
-    guard !macroAttributes(attachedTo: DeclSyntax(node), ofType: AccessorMacro.Type.self).isEmpty else {
+    guard !macroAttributes(attachedTo: DeclSyntax(originalNode), ofType: AccessorMacro.Type.self).isEmpty else {
       return DeclSyntax(node)
     }
 
@@ -1061,6 +1060,7 @@ extension MacroApplication {
     var result: [ExpandedNode] = []
 
     for macroAttribute in macroAttributes(attachedTo: decl, ofType: ofType) {
+      attributesToExpand = attributesToExpand.filter { $0.attributeNode != macroAttribute.attributeNode }
       do {
         if let expanded = try expandMacro(
           macroAttribute.attributeNode,
@@ -1068,7 +1068,6 @@ extension MacroApplication {
           macroAttribute.conformanceList
         ) {
           result += expanded
-          self.expandedAttributes.append(macroAttribute.attributeNode)
         }
       } catch {
         contextGenerator(Syntax(decl)).addDiagnostics(from: error, node: macroAttribute.attributeNode)
@@ -1216,7 +1215,7 @@ extension MacroApplication {
               from: newAccessors,
               indentationWidth: self.indentationWidth
             )
-            self.expandedAttributes.append(macro.attributeNode)
+            attributesToExpand = attributesToExpand.filter { $0.attributeNode != macro.attributeNode }
           }
         } else if let newAccessors = try expandAccessorMacroWithoutExistingAccessors(
           definition: macro.definition,
@@ -1239,7 +1238,7 @@ extension MacroApplication {
           } else {
             newAccessorsBlock = newAccessors
           }
-          self.expandedAttributes.append(macro.attributeNode)
+          attributesToExpand = attributesToExpand.filter { $0.attributeNode != macro.attributeNode }
         }
       } catch {
         contextGenerator(Syntax(storage)).addDiagnostics(from: error, node: macro.attributeNode)

Copy link
Author

@RayZhao1998 RayZhao1998 Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your help.

@@ -961,10 +960,10 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
     )
   }
 
-  override func visit(_ node: VariableDeclSyntax) -> DeclSyntax {
-    var node = super.visit(node).cast(VariableDeclSyntax.self)
+  override func visit(_ originalNode: VariableDeclSyntax) -> DeclSyntax {
+    var node = super.visit(originalNode).cast(VariableDeclSyntax.self)
 
-    guard !macroAttributes(attachedTo: DeclSyntax(node), ofType: AccessorMacro.Type.self).isEmpty else {
+    guard !macroAttributes(attachedTo: DeclSyntax(originalNode), ofType: AccessorMacro.Type.self).isEmpty else {
       return DeclSyntax(node)
     }

The changes of visit here are weird. When should we use originalNode and when to use the processed node?

Should we use originalNode when expandAccessor? If not, the comparison of attributedNode in expandAccessor will not be equal too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the code again, I don’t understand what I did myself 🤦🏽 Do you have a test at hand that shows the issue? And I’m getting the feeling that instead of working around things ad-hoc here, the right solution is to productize #2118, which will allow us to keep track of the node’s in the original tree.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pushed my code. As we discussed above, unified diagnostic for unexpanded macros is already implemented. Because of the unexpected result of comparing two attributedNode, a lot of tests will fail.

Thinking back to the problem I encountered at the beginning, it was to compare whether two attributeNodes were the same in the origin syntax tree. So I agree with that #2118 should be the right solution.

@RayZhao1998 RayZhao1998 force-pushed the feature/assertMacroExpansion-notvariable-diagnostics branch from 50d3974 to e029a45 Compare May 4, 2024 17:15
@ahoppen
Copy link
Member

ahoppen commented Jun 27, 2024

OK, let me try and push for #2118 to get merged again, everything else seems like a fairly hacky solution. I’m sorry that this PR has already been dragged along for this long but getting #2118 ready might take another couple of weeks as we are currently busy finishing up the 6.0 release. Once that has calmed down, I will try to get #2118 merged and when that’s done, this PR should be trivial to implement and become the first user of the node tracking logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

assertMacroExpansion should emit an error if macro is applied to node that is not a variable
2 participants