From 9398e4c603a529d3c644c2a0458b633fb32992c7 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 25 Sep 2025 12:23:55 -0500 Subject: [PATCH 1/8] Add entrypoint to generate plugin specs Signed-off-by: Ben Sherman --- adrs/0000-00-00-template.md | 76 +++++++++ adrs/2025-09-22-plugin-spec.md | 154 ++++++++++++++++++ .../config/schema/JsonRenderer.groovy | 77 +-------- .../nextflow/plugin/spec/ConfigSpec.groovy | 101 ++++++++++++ .../nextflow/plugin/spec/FunctionSpec.groovy | 69 ++++++++ .../nextflow/plugin/spec/PluginSpec.groovy | 105 ++++++++++++ .../plugin/spec/PluginSpecTest.groovy | 140 ++++++++++++++++ 7 files changed, 648 insertions(+), 74 deletions(-) create mode 100644 adrs/0000-00-00-template.md create mode 100644 adrs/2025-09-22-plugin-spec.md create mode 100644 modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy create mode 100644 modules/nextflow/src/test/groovy/nextflow/plugin/spec/PluginSpecTest.groovy diff --git a/adrs/0000-00-00-template.md b/adrs/0000-00-00-template.md new file mode 100644 index 0000000000..474aff73e5 --- /dev/null +++ b/adrs/0000-00-00-template.md @@ -0,0 +1,76 @@ +# [short title of solved problem and solution] + +- Authors: [who wrote the ADR] +- Status: [draft | proposed | rejected | accepted | deprecated | … | superseded by [xxx](xxx.md)] +- Deciders: [list everyone involved in the decision] +- Date: [YYYY-MM-DD when the decision was last updated] +- Tags: [space and/or comma separated list of tags] + +Technical Story: [description | ticket/issue URL] + +## Summary + +Quick description of the problem and the context. Should not take more than 2-3 lines. + +## Problem Statement + +Description of the technical problem to solve or to decision to make. This should be concise but provide all required details and the context related to the technical decision to be taken. + +## Goals or Decision Drivers + +Depending the context define clearly what are the goals or what are the most important decision drivers. + +- [driver 1, e.g., a force, facing concern, …] +- [driver 2, e.g., a force, facing concern, …] +- … + +## Non-goals + +Define what's out of the scope of this ADR. + +## Considered Options + +- [option 1] +- [option 2] +- [option 3] +- … + + +## Pros and Cons of the Options + +### [option 1] + +[example | description | pointer to more information | …] + +- Good, because [argument a] +- Good, because [argument b] +- Bad, because [argument c] +- … + +### [option 2] + +[example | description | pointer to more information | …] + +- Good, because [argument a] +- Good, because [argument b] +- Bad, because [argument c] +- … + + +## Solution or decision outcome + +Summarize the solution or decision outcome in one-two lines. + +## Rationale & discussion + +Describe the solution or the decision outcome discussing how decision drivers have been applied and how it matches the declared goals. This section is expected to be concise though providing comprehensive description of the technical solution and covering all uncertainty or ambiguous points. + +## Links + +- [Link type](link to adr) +- … + +## More information + +- [What is an ADR and why should you use them](https://github.com/thomvaill/log4brains/tree/master#-what-is-an-adr-and-why-should-you-use-them) +- [ADR GitHub organization](https://adr.github.io/) diff --git a/adrs/2025-09-22-plugin-spec.md b/adrs/2025-09-22-plugin-spec.md new file mode 100644 index 0000000000..1da649de34 --- /dev/null +++ b/adrs/2025-09-22-plugin-spec.md @@ -0,0 +1,154 @@ +# Plugin Spec + +- Authors: Ben Sherman +- Status: accepted +- Deciders: Ben Sherman, Paolo Di Tommaso +- Date: 2025-09-22 +- Tags: plugins + +## Summary + +Provide a way for external systems to understand key information about third-party plugins. + +## Problem Statement + +Nextflow plugins need a way to statically declare extensions to the Nextflow language so that external systems can extract information about a plugin without loading it in the JVM. + +Primary use cases: + +- The Nextflow language server needs to know about any config scopes, custom functions, etc, defined by a plugin, in order to recognize them in Nextflow scripts and config files. + +- The Nextflow plugin registry (or other user interfaces) can use this information to provide API documentation. + +## Goals or Decision Drivers + +- External systems (e.g. language server) need to be able to understand plugins without having to load them in the JVM. + +## Non-goals + +- Defining specs for the core runtime and core plugins: these definitions are handled separately, although they may share some functionality with plugin specs. + +## Considered Options + +### Nextflow plugin system + +Require external systems to use Nextflow's plugin system to load plugins at runtime in order to extract information about them. + +- **Pro:** Allows any information to be extracted since the entire plugin is loaded + +- **Con:** Requires the entire Nextflow plugin system to be reused or reimplemented. Not ideal for Java applications since the plugin system is implemented in Groovy, incompatible with non-JVM applications + +- **Con:** Requires plugins to be downloaded, cached, loaded in the JVM, even though there is no need to use the plugin. + +### Plugin spec + +Define a plugin spec for every plugin release which is stored and served by the plugin registry as JSON. + +- **Pro:** Allows any system to inspect plugin definitions through a standard JSON document, instead of downloading plugins and loading them into a JVM. + +- **Con:** Requires the plugin spec to be generated at build-time and stored in the plugin registry. + +- **Con:** Requires a standard format to ensure interoperability across different versions of Nextflow, the language server, and third-party plugins. + +## Solution + +Define a plugin spec for every plugin release which is stored and served by the plugin registry as JSON. + +- Plugin developers only need to define [extension points](https://nextflow.io/docs/latest/plugins/developing-plugins.html#extension-points) as usual, and the Gradle plugin will extract the plugin spec and store it in the plugin registry as part of each plugin release. + +- The language server can infer which third-party plugins are required from the `plugins` block in a config file. It will retrieve the appropriate plugin specs from the plugin registry. + +A plugin spec consists of a list of *definitions*. Each definition has a *type* and a *spec*. + +Example: + +```json +{ + "$schema": "https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/schema.json", + "definitions": [ + { + "type": "ConfigScope", + "spec": { + // ... + } + }, + { + "type": "Function", + "spec": { + // ... + } + }, + ] +} +``` + +The following types of definitions are allowed: + +**ConfigScope** + +Defines a top-level config scope. The spec consists of a *name*, an optional *description*, and *children*. + +The children should be a list of definitions corresponding to nested config scopes and options. The following definitions are allowed: + +- **ConfigOption**: Defines a config option. The spec consists of a *description* and *type*. + +- **ConfigScope**: Defines a nested config scope, using the same spec as for top-level scopes. + +Example: + +```json +{ + "type": "ConfigScope", + "spec": { + "name": "hello", + "description": "The `hello` scope controls the behavior of the `nf-hello` plugin.", + "children": [ + { + "type": "ConfigOption", + "spec": { + "name": "message", + "description": "Message to print to standard output when the plugin is enabled.", + "type": "String" + } + } + ] + } +} +``` + +**Factory** + +Defines a channel factory that can be included in Nextflow scripts. The spec is the same as for functions. + +**Function** + +Defines a function that can be included in Nextflow scripts. The spec consists of a *name*, an optional *description*, a *return type*, and a list of *parameters*. Each parameter consists of a *name* and a *type*. + +Example: + +```json +{ + "type": "Function", + "spec": { + "name": "sayHello", + "description": "Say hello to the given target", + "returnType": "void", + "parameters": [ + { + "name": "target", + "type": "String" + } + ] + } +} +``` + +**Operator** + +Defines a channel operator that can be included in Nextflow scripts. The spec is the same as for functions. + +## Rationale & discussion + +Now that there is a Gradle plugin for building Nextflow plugins and a registry to publish and retrieve plugins, it is possible to generate, publish, and retrieve plugin specs in a way that is transparent to plugin developers. + +Plugins specs adhere to a pre-defined [schema](https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/schema.json) to ensure consistency across different versions of Nextflow. In the future, new versions of the schema can be defined as needed to support new behaviors or requirements. diff --git a/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy b/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy index 56390388e8..956ca4ea05 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy @@ -18,9 +18,8 @@ package nextflow.config.schema import groovy.json.JsonOutput import groovy.transform.TypeChecked import nextflow.plugin.Plugins +import nextflow.plugin.spec.ConfigSpec import nextflow.script.dsl.Description -import nextflow.script.types.Types -import org.codehaus.groovy.ast.ClassNode @TypeChecked class JsonRenderer { @@ -38,86 +37,16 @@ class JsonRenderer { final description = clazz.getAnnotation(Description)?.value() if( scopeName == '' ) { SchemaNode.Scope.of(clazz, '').children().each { name, node -> - result.put(name, fromNode(node)) + result.put(name, ConfigSpec.of(node)) } continue } if( !scopeName ) continue final node = SchemaNode.Scope.of(clazz, description) - result.put(scopeName, fromNode(node, scopeName)) + result.put(scopeName, ConfigSpec.of(node, scopeName)) } return result } - private static Map fromNode(SchemaNode node, String name=null) { - if( node instanceof SchemaNode.Option ) - return fromOption(node) - if( node instanceof SchemaNode.Placeholder ) - return fromPlaceholder(node) - if( node instanceof SchemaNode.Scope ) - return fromScope(node, name) - throw new IllegalStateException() - } - - private static Map fromOption(SchemaNode.Option node) { - final description = node.description().stripIndent(true).trim() - final type = fromType(new ClassNode(node.type())) - - return [ - type: 'Option', - spec: [ - description: description, - type: type - ] - ] - } - - private static Map fromPlaceholder(SchemaNode.Placeholder node) { - final description = node.description().stripIndent(true).trim() - final placeholderName = node.placeholderName() - final scope = fromScope(node.scope()) - - return [ - type: 'Placeholder', - spec: [ - description: description, - placeholderName: placeholderName, - scope: scope.spec - ] - ] - } - - private static Map fromScope(SchemaNode.Scope node, String scopeName=null) { - final description = node.description().stripIndent(true).trim() - final children = node.children().collectEntries { name, child -> - Map.entry(name, fromNode(child, name)) - } - - return [ - type: 'Scope', - spec: [ - description: withLink(scopeName, description), - children: children - ] - ] - } - - private static String withLink(String scopeName, String description) { - return scopeName - ? "$description\n\n[Read more](https://nextflow.io/docs/latest/reference/config.html#$scopeName)\n" - : description - } - - private static Object fromType(ClassNode cn) { - final name = Types.getName(cn.getTypeClass()) - if( cn.isUsingGenerics() ) { - final typeArguments = cn.getGenericsTypes().collect { gt -> fromType(gt.getType()) } - return [ name: name, typeArguments: typeArguments ] - } - else { - return name - } - } - } diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy new file mode 100644 index 0000000000..e379172258 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy @@ -0,0 +1,101 @@ +/* + * Copyright 2024-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.plugin.spec + +import groovy.transform.CompileStatic +import nextflow.config.schema.SchemaNode +import nextflow.script.types.Types +import org.codehaus.groovy.ast.ClassNode + +/** + * Generate specs for config scopes. + * + * @author Ben Sherman + */ +@CompileStatic +class ConfigSpec { + + static Map of(SchemaNode node, String name=null) { + return fromNode(node, name) + } + + private static Map fromNode(SchemaNode node, String name=null) { + if( node instanceof SchemaNode.Option ) + return fromOption(node, name) + if( node instanceof SchemaNode.Placeholder ) + return fromPlaceholder(node, name) + if( node instanceof SchemaNode.Scope ) + return fromScope(node, name) + throw new IllegalStateException() + } + + private static Map fromOption(SchemaNode.Option node, String name) { + final description = node.description().stripIndent(true).trim() + final type = fromType(new ClassNode(node.type())) + + return [ + type: 'ConfigOption', + spec: [ + name: name, + description: description, + type: type + ] + ] + } + + private static Map fromPlaceholder(SchemaNode.Placeholder node, String name) { + final description = node.description().stripIndent(true).trim() + final placeholderName = node.placeholderName() + final scope = fromScope(node.scope()) + + return [ + type: 'ConfigPlaceholderScope', + spec: [ + name: name, + description: description, + placeholderName: placeholderName, + scope: scope.spec + ] + ] + } + + private static Map fromScope(SchemaNode.Scope node, String scopeName=null) { + final description = node.description().stripIndent(true).trim() + final children = node.children().collect { name, child -> + fromNode(child, name) + } + + return [ + type: 'ConfigScope', + spec: [ + name: scopeName, + description: description, + children: children + ] + ] + } + + private static Object fromType(ClassNode cn) { + final name = Types.getName(cn.getTypeClass()) + if( cn.getGenericsTypes() != null ) { + final typeArguments = cn.getGenericsTypes().collect { gt -> fromType(gt.getType()) } + return [ name: name, typeArguments: typeArguments ] + } + else { + return name + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy new file mode 100644 index 0000000000..c95bdfc86b --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy @@ -0,0 +1,69 @@ +/* + * Copyright 2024-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.plugin.spec + +import java.lang.reflect.Method + +import groovy.transform.CompileStatic +import nextflow.script.dsl.Description +import nextflow.script.types.Types +import org.codehaus.groovy.ast.ClassNode + +/** + * Generate specs for functions, channel factories, and operators. + * + * @author Ben Sherman + */ +@CompileStatic +class FunctionSpec { + + static Map of(Method method, String type) { + final name = method.getName() + final description = method.getAnnotation(Description)?.value() + final returnType = fromType(method.getReturnType()) + final parameters = method.getParameters().collect { param -> + [ + name: param.getName(), + type: fromType(param.getType()) + ] + } + + return [ + type: type, + spec: [ + name: name, + description: description, + returnType: returnType, + parameters: parameters + ] + ] + } + + private static Object fromType(Class c) { + return fromType(new ClassNode(c)) + } + + private static Object fromType(ClassNode cn) { + final name = Types.getName(cn.getTypeClass()) + if( cn.getGenericsTypes() != null ) { + final typeArguments = cn.getGenericsTypes().collect { gt -> fromType(gt.getType()) } + return [ name: name, typeArguments: typeArguments ] + } + else { + return name + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy new file mode 100644 index 0000000000..e4eba7bd5b --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy @@ -0,0 +1,105 @@ +/* + * Copyright 2024-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.plugin.spec + +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import nextflow.config.schema.ConfigScope +import nextflow.config.schema.SchemaNode +import nextflow.config.schema.ScopeName +import nextflow.plugin.extension.Factory +import nextflow.plugin.extension.Function +import nextflow.plugin.extension.Operator +import nextflow.plugin.extension.PluginExtensionPoint +import nextflow.script.dsl.Description + +/** + * Generate a plugin spec. + * + * @author Ben Sherman + */ +@CompileStatic +class PluginSpec { + + private List extensionPoints + + PluginSpec(List extensionPoints) { + this.extensionPoints = extensionPoints + } + + Map build() { + final classLoader = Thread.currentThread().getContextClassLoader() + + // extract schema for each plugin definition + final definitions = [] + + for( final className : extensionPoints ) { + final clazz = classLoader.loadClass(className) + + if( ConfigScope.class.isAssignableFrom(clazz) ) { + final scopeName = clazz.getAnnotation(ScopeName)?.value() + final description = clazz.getAnnotation(Description)?.value() + if( !scopeName ) + continue + final node = SchemaNode.Scope.of(clazz, description) + + definitions.add(ConfigSpec.of(node, scopeName)) + } + + if( PluginExtensionPoint.class.isAssignableFrom(clazz) ) { + final methods = clazz.getDeclaredMethods() + for( final method : methods ) { + if( method.getAnnotation(Factory) ) + definitions.add(FunctionSpec.of(method, 'Factory')) + else if( method.getAnnotation(Function) ) + definitions.add(FunctionSpec.of(method, 'Function')) + else if( method.getAnnotation(Operator) ) + definitions.add(FunctionSpec.of(method, 'Operator')) + } + } + } + + return [ + '$schema': 'https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/schema.json', + 'definitions': definitions + ] + } +} + + +@CompileStatic +class PluginSpecWriter { + + static void main(String[] args) { + if (args.length < 2) { + System.err.println("Usage: PluginSpecWriter [class2] ...") + System.exit(1) + } + + final outputPath = args[0] + final extensionPoints = args[1..-1] + + // build plugin spec + final spec = new PluginSpec(extensionPoints).build() + + // write plugin spec to JSON file + final file = new File(outputPath) + file.parentFile.mkdirs() + file.text = JsonOutput.toJson(spec) + + println "Saved plugin spec to $file" + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/plugin/spec/PluginSpecTest.groovy b/modules/nextflow/src/test/groovy/nextflow/plugin/spec/PluginSpecTest.groovy new file mode 100644 index 0000000000..ab7d54e445 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/plugin/spec/PluginSpecTest.groovy @@ -0,0 +1,140 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package nextflow.plugin.spec + +import groovyx.gpars.dataflow.DataflowReadChannel +import groovyx.gpars.dataflow.DataflowWriteChannel +import nextflow.Session +import nextflow.config.schema.ConfigOption +import nextflow.config.schema.ConfigScope +import nextflow.config.schema.ScopeName +import nextflow.plugin.extension.Factory +import nextflow.plugin.extension.Function +import nextflow.plugin.extension.Operator +import nextflow.plugin.extension.PluginExtensionPoint +import nextflow.script.dsl.Description +import spock.lang.Specification + +/** + * @author Ben Sherman + */ +class PluginSpecTest extends Specification { + + def "should generate a plugin spec" () { + given: + def extensionPoints = [ + 'nextflow.plugin.spec.TestConfig', + 'nextflow.plugin.spec.TestExtension', + ] + def spec = new PluginSpec(extensionPoints).build() + and: + def definitions = spec.definitions.sort { d -> d.spec.name } + + expect: + definitions.size() == 4 + and: + definitions[0] == [ + type: 'ConfigScope', + spec: [ + name: 'hello', + description: 'The `hello` scope controls the behavior of the `nf-hello` plugin.', + children: [ + [ + type: 'ConfigOption', + spec: [ + name: 'message', + description: 'Message to print to standard output when the plugin is enabled.', + type: 'String' + ] + ] + ] + ] + ] + definitions[1] == [ + type: 'Factory', + spec: [ + name: 'helloFactory', + description: null, + returnType: 'DataflowWriteChannel', + parameters: [] + ] + ] + definitions[2] == [ + type: 'Operator', + spec: [ + name: 'helloOperator', + description: null, + returnType: 'DataflowWriteChannel', + parameters: [ + [ + name: 'arg0', + type: 'DataflowReadChannel' + ] + ] + ] + ] + definitions[3] == [ + type: 'Function', + spec: [ + name: 'sayHello', + description: 'Say hello to the given targets', + returnType: 'void', + parameters: [ + [ + name: 'arg0', + type: 'List' + ] + ] + ] + ] + } +} + + +@ScopeName('hello') +@Description("The `hello` scope controls the behavior of the `nf-hello` plugin.") +class TestConfig implements ConfigScope { + + @ConfigOption + @Description(''' + Message to print to standard output when the plugin is enabled. + ''') + String message +} + + +class TestExtension extends PluginExtensionPoint { + + @Override + protected void init(Session session) { + } + + @Factory + DataflowWriteChannel helloFactory() { + } + + @Function + @Description("Say hello to the given targets") + void sayHello(List targets) { + } + + @Operator + DataflowWriteChannel helloOperator(DataflowReadChannel source) { + } + +} From 5a394336a904afe4045adb49d4d67bd6c23fbce1 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 25 Sep 2025 12:24:04 -0500 Subject: [PATCH 2/8] Rename existing PluginSpec to PluginRef Signed-off-by: Ben Sherman --- docs/developer/diagrams/nextflow.plugin.mmd | 4 +- .../nextflow/plugin/DefaultPlugins.groovy | 10 +-- .../plugin/HttpPluginRepository.groovy | 8 +- .../{PluginSpec.groovy => PluginRef.groovy} | 12 +-- .../main/nextflow/plugin/PluginUpdater.groovy | 6 +- .../main/nextflow/plugin/PluginsFacade.groovy | 38 +++++----- .../plugin/PrefetchUpdateRepository.groovy | 2 +- .../nextflow/plugin/DefaultPluginsTest.groovy | 12 +-- .../plugin/HttpPluginRepositoryTest.groovy | 14 ++-- .../test/nextflow/plugin/PluginRefTest.groovy | 42 ++++++++++ .../nextflow/plugin/PluginSpecTest.groovy | 42 ---------- .../nextflow/plugin/PluginUpdaterTest.groovy | 2 +- .../nextflow/plugin/PluginsFacadeTest.groovy | 76 +++++++++---------- 13 files changed, 134 insertions(+), 134 deletions(-) rename modules/nf-commons/src/main/nextflow/plugin/{PluginSpec.groovy => PluginRef.groovy} (84%) create mode 100644 modules/nf-commons/src/test/nextflow/plugin/PluginRefTest.groovy delete mode 100644 modules/nf-commons/src/test/nextflow/plugin/PluginSpecTest.groovy diff --git a/docs/developer/diagrams/nextflow.plugin.mmd b/docs/developer/diagrams/nextflow.plugin.mmd index 83926f39f7..2d576aaad8 100644 --- a/docs/developer/diagrams/nextflow.plugin.mmd +++ b/docs/developer/diagrams/nextflow.plugin.mmd @@ -6,9 +6,9 @@ classDiagram Plugins --> PluginsFacade : init - PluginsFacade "1" --> "*" PluginSpec : load + PluginsFacade "1" --> "*" PluginRef : load - class PluginSpec { + class PluginRef { id : String version : String } diff --git a/modules/nf-commons/src/main/nextflow/plugin/DefaultPlugins.groovy b/modules/nf-commons/src/main/nextflow/plugin/DefaultPlugins.groovy index 2670d03e4a..39ab217465 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/DefaultPlugins.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/DefaultPlugins.groovy @@ -29,26 +29,26 @@ class DefaultPlugins { public static final DefaultPlugins INSTANCE = new DefaultPlugins() - private Map plugins = new HashMap<>(20) + private Map plugins = new HashMap<>(20) protected DefaultPlugins() { final meta = this.class.getResourceAsStream('/META-INF/plugins-info.txt')?.text plugins = parseMeta(meta) } - protected Map parseMeta(String meta) { + protected Map parseMeta(String meta) { if( !meta ) return Collections.emptyMap() final result = new HashMap(20) for( String line : meta.readLines() ) { - final spec = PluginSpec.parse(line) + final spec = PluginRef.parse(line) result[spec.id] = spec } return result } - PluginSpec getPlugin(String pluginId) throws IllegalArgumentException { + PluginRef getPlugin(String pluginId) throws IllegalArgumentException { if( !pluginId ) throw new IllegalArgumentException("Missing pluginId argument") final result = plugins.get(pluginId) @@ -61,7 +61,7 @@ class DefaultPlugins { return plugins.containsKey(pluginId) } - List getPlugins() { + List getPlugins() { return new ArrayList(plugins.values()) } diff --git a/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy b/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy index 26d36f0e73..8d95095355 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy @@ -65,7 +65,7 @@ class HttpPluginRepository implements PrefetchUpdateRepository { // the map. Once the prefetch is complete, this repository will behave // like any other implementation of UpdateRepository. @Override - void prefetch(List plugins) { + void prefetch(List plugins) { if (plugins && !plugins.isEmpty()) { this.plugins = fetchMetadata(plugins) } @@ -114,16 +114,16 @@ class HttpPluginRepository implements PrefetchUpdateRepository { // http handling private Map fetchMetadataByIds(Collection ids) { - def specs = ids.collect(id -> new PluginSpec(id, null)) + def specs = ids.collect(id -> new PluginRef(id, null)) return fetchMetadata(specs) } - private Map fetchMetadata(Collection specs) { + private Map fetchMetadata(Collection specs) { final ordered = specs.sort(false) return fetchMetadata0(ordered) } - private Map fetchMetadata0(List specs) { + private Map fetchMetadata0(List specs) { def pluginsParam = specs.collect { "${it.id}${it.version ? '@' + it.version : ''}" }.join(',') def uri = url.resolve("v1/plugins/dependencies?plugins=${URLEncoder.encode(pluginsParam, 'UTF-8')}&nextflowVersion=${URLEncoder.encode(BuildInfo.version, 'UTF-8')}") def req = HttpRequest.newBuilder() diff --git a/modules/nf-commons/src/main/nextflow/plugin/PluginSpec.groovy b/modules/nf-commons/src/main/nextflow/plugin/PluginRef.groovy similarity index 84% rename from modules/nf-commons/src/main/nextflow/plugin/PluginSpec.groovy rename to modules/nf-commons/src/main/nextflow/plugin/PluginRef.groovy index 2aed3252f2..5780c38aa5 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PluginSpec.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PluginRef.groovy @@ -27,7 +27,7 @@ import nextflow.util.CacheHelper * @author Paolo Di Tommaso */ @Canonical -class PluginSpec implements CacheFunnel, Comparable { +class PluginRef implements CacheFunnel, Comparable { /** * Plugin unique ID @@ -43,17 +43,17 @@ class PluginSpec implements CacheFunnel, Comparable { * Parse a plugin fully-qualified ID eg. nf-amazon@1.2.0 * * @param fqid The fully qualified plugin id - * @return A {@link PluginSpec} representing the plugin + * @return A {@link PluginRef} representing the plugin */ - static PluginSpec parse(String fqid, DefaultPlugins defaultPlugins=null) { + static PluginRef parse(String fqid, DefaultPlugins defaultPlugins=null) { final tokens = fqid.tokenize('@') as List final id = tokens[0] final ver = tokens[1] if( ver || defaultPlugins==null ) - return new PluginSpec(id, ver) + return new PluginRef(id, ver) if( defaultPlugins.hasPlugin(id) ) return defaultPlugins.getPlugin(id) - return new PluginSpec(id) + return new PluginRef(id) } @Override @@ -70,7 +70,7 @@ class PluginSpec implements CacheFunnel, Comparable { } @Override - int compareTo(PluginSpec that) { + int compareTo(PluginRef that) { return this.toString() <=> that.toString() } } diff --git a/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy b/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy index 3ad4ef80ba..95c5cd0c07 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy @@ -146,7 +146,7 @@ class PluginUpdater extends UpdateManager { * Prefetch metadata for plugins. This gives an opportunity for certain * repository types to perform some data-loading optimisations. */ - void prefetchMetadata(List plugins) { + void prefetchMetadata(List plugins) { // use direct field access to avoid the refresh() call in getRepositories() // which could fail anything which hasn't had a chance to prefetch yet for( def repo : this.@repositories ) { @@ -185,9 +185,9 @@ class PluginUpdater extends UpdateManager { void pullPlugins(List plugins) { pullOnly=true try { - final specs = plugins.collect(it -> PluginSpec.parse(it,defaultPlugins)) + final specs = plugins.collect(it -> PluginRef.parse(it,defaultPlugins)) prefetchMetadata(specs) - for( PluginSpec spec : specs ) { + for( PluginRef spec : specs ) { pullPlugin0(spec.id, spec.version) } } diff --git a/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy b/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy index 4646e16583..60a48820a5 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy @@ -373,10 +373,10 @@ class PluginsFacade implements PluginStateListener { } void start(String pluginId) { - start(List.of(PluginSpec.parse(pluginId, defaultPlugins))) + start(List.of(PluginRef.parse(pluginId, defaultPlugins))) } - void start(List specs) { + void start(List specs) { // check if the plugins are allowed to start final disallow = specs.find(it-> !isAllowed(it)) if( disallow ) { @@ -397,7 +397,7 @@ class PluginsFacade implements PluginStateListener { // prefetch the plugins meta updater.prefetchMetadata(startable) // finally start the plugins - for( PluginSpec plugin : startable ) { + for( PluginRef plugin : startable ) { updater.prepareAndStart(plugin.id, plugin.version) } } @@ -415,7 +415,7 @@ class PluginsFacade implements PluginStateListener { return embedded } - protected List pluginsRequirement(Map config) { + protected List pluginsRequirement(Map config) { def specs = parseConf(config) if( isEmbedded() && specs ) { // custom plugins are not allowed for nextflow self-contained package @@ -448,7 +448,7 @@ class PluginsFacade implements PluginStateListener { return specs } - protected List defaultPluginsConf(Map config) { + protected List defaultPluginsConf(Map config) { // retrieve the list from the env var final commaSepList = env.get('NXF_PLUGINS_DEFAULT') if( commaSepList && commaSepList !in ['true','false'] ) { @@ -456,11 +456,11 @@ class PluginsFacade implements PluginStateListener { // specified in the defaults list. Otherwise parse the provider id@version string to the corresponding spec return commaSepList .tokenize(',') - .collect( it-> defaultPlugins.hasPlugin(it) ? defaultPlugins.getPlugin(it) : PluginSpec.parse(it) ) + .collect( it-> defaultPlugins.hasPlugin(it) ? defaultPlugins.getPlugin(it) : PluginRef.parse(it) ) } // infer from app config - final plugins = new ArrayList() + final plugins = new ArrayList() final workDir = config.workDir as String final bucketDir = config.bucketDir as String final executor = Bolts.navigate(config, 'process.executor') @@ -478,7 +478,7 @@ class PluginsFacade implements PluginStateListener { plugins << defaultPlugins.getPlugin('nf-k8s') if( Bolts.navigate(config, 'weblog.enabled')) - plugins << new PluginSpec('nf-weblog') + plugins << new PluginRef('nf-weblog') return plugins } @@ -489,11 +489,11 @@ class PluginsFacade implements PluginStateListener { * @param config The nextflow config as a Map object * @return The list of declared plugins */ - protected List parseConf(Map config) { + protected List parseConf(Map config) { final pluginsConf = config.plugins as List final result = new ArrayList( pluginsConf?.size() ?: 0 ) if(pluginsConf) for( String it : pluginsConf ) { - result.add( PluginSpec.parse(it, defaultPlugins) ) + result.add( PluginRef.parse(it, defaultPlugins) ) } return result } @@ -529,28 +529,28 @@ class PluginsFacade implements PluginStateListener { * @return * The list of plugins resulting from merging the two lists */ - protected List mergePluginSpecs(List configPlugins, List defaultPlugins) { - final map = new LinkedHashMap(10) + protected List mergePluginSpecs(List configPlugins, List defaultPlugins) { + final map = new LinkedHashMap(10) // add all plugins in the 'configPlugins' argument - for( PluginSpec plugin : configPlugins ) { + for( PluginRef plugin : configPlugins ) { map.put(plugin.id, plugin) } // add the plugin in the 'defaultPlugins' argument // if the map already contains the plugin, // override it only if it does not specify a version - for( PluginSpec plugin : defaultPlugins ) { + for( PluginRef plugin : defaultPlugins ) { if( !map[plugin.id] || !map[plugin.id].version ) { map.put(plugin.id, plugin) } } - return new ArrayList(map.values()) + return new ArrayList(map.values()) } - protected List parseAllowedPlugins(Map env) { + protected List parseAllowedPlugins(Map env) { final list = env.get('NXF_PLUGINS_ALLOWED') // note: empty string means no plugins is allowed return list!=null - ? list.tokenize(',').collect(it-> PluginSpec.parse(it)) + ? list.tokenize(',').collect(it-> PluginRef.parse(it)) : null } @@ -561,7 +561,7 @@ class PluginsFacade implements PluginStateListener { * The list of allowed plugins. {@code null} mean all. Empty list means none */ @Memoized - protected List getAllowedPlugins() { + protected List getAllowedPlugins() { return parseAllowedPlugins(env) } @@ -572,7 +572,7 @@ class PluginsFacade implements PluginStateListener { : true } - protected isAllowed(PluginSpec plugin) { + protected isAllowed(PluginRef plugin) { return isAllowed(plugin.id) } diff --git a/modules/nf-commons/src/main/nextflow/plugin/PrefetchUpdateRepository.groovy b/modules/nf-commons/src/main/nextflow/plugin/PrefetchUpdateRepository.groovy index 8aaa5d6514..d4bee66128 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PrefetchUpdateRepository.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PrefetchUpdateRepository.groovy @@ -16,5 +16,5 @@ interface PrefetchUpdateRepository extends UpdateRepository { * This will be called when Nextflow starts, before * initialising the plugins. */ - void prefetch(List plugins) + void prefetch(List plugins) } diff --git a/modules/nf-commons/src/test/nextflow/plugin/DefaultPluginsTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/DefaultPluginsTest.groovy index 97076b2c28..41ee0883af 100644 --- a/modules/nf-commons/src/test/nextflow/plugin/DefaultPluginsTest.groovy +++ b/modules/nf-commons/src/test/nextflow/plugin/DefaultPluginsTest.groovy @@ -17,9 +17,9 @@ class DefaultPluginsTest extends Specification { def 'should validate get plugins' () { given: - def AMAZON = new PluginSpec('nf-amazon', '0.1.0') - def GOOGLE = new PluginSpec('nf-google', '0.2.0') - def AZURE = new PluginSpec('nf-azure', '0.3.0') + def AMAZON = new PluginRef('nf-amazon', '0.1.0') + def GOOGLE = new PluginRef('nf-google', '0.2.0') + def AZURE = new PluginRef('nf-azure', '0.3.0') and: def defaults = new DefaultPlugins(plugins: [ 'nf-amazon': AMAZON, @@ -41,9 +41,9 @@ class DefaultPluginsTest extends Specification { def 'should get sorted string' () { given: - def DELTA = new PluginSpec('nf-delta', '1.0.0') - def OMEGA = new PluginSpec('nf-omega', '2.0.0') - def ALPHA = new PluginSpec('nf-alpha', '3.0.0') + def DELTA = new PluginRef('nf-delta', '1.0.0') + def OMEGA = new PluginRef('nf-omega', '2.0.0') + def ALPHA = new PluginRef('nf-alpha', '3.0.0') and: def defaults = new DefaultPlugins(plugins: [ 'nf-delta': DELTA, diff --git a/modules/nf-commons/src/test/nextflow/plugin/HttpPluginRepositoryTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/HttpPluginRepositoryTest.groovy index f27b1cea20..8a1bae248f 100644 --- a/modules/nf-commons/src/test/nextflow/plugin/HttpPluginRepositoryTest.groovy +++ b/modules/nf-commons/src/test/nextflow/plugin/HttpPluginRepositoryTest.groovy @@ -46,7 +46,7 @@ class HttpPluginRepositoryTest extends Specification { } when: - unit.prefetch([new PluginSpec("nf-fake")]) + unit.prefetch([new PluginRef("nf-fake")]) then: def plugins = unit.getPlugins() @@ -85,7 +85,7 @@ class HttpPluginRepositoryTest extends Specification { } when: - unit.prefetch([new PluginSpec("nf-fake")]) + unit.prefetch([new PluginRef("nf-fake")]) then: def plugins = unit.getPlugins() @@ -108,7 +108,7 @@ class HttpPluginRepositoryTest extends Specification { wiremock.stop() when: - unit.prefetch([new PluginSpec("nf-fake")]) + unit.prefetch([new PluginRef("nf-fake")]) then: def err = thrown PluginRuntimeException @@ -131,7 +131,7 @@ class HttpPluginRepositoryTest extends Specification { } when: - unit.prefetch([new PluginSpec("nf-fake")]) + unit.prefetch([new PluginRef("nf-fake")]) then: def err = thrown PluginRuntimeException @@ -164,7 +164,7 @@ class HttpPluginRepositoryTest extends Specification { } when: - unit.prefetch([new PluginSpec("nf-fake")]) + unit.prefetch([new PluginRef("nf-fake")]) then: def err = thrown PluginRuntimeException @@ -190,7 +190,7 @@ class HttpPluginRepositoryTest extends Specification { } when: - unit.prefetch([new PluginSpec("nf-fake")]) + unit.prefetch([new PluginRef("nf-fake")]) then: def err = thrown PluginRuntimeException @@ -256,7 +256,7 @@ class HttpPluginRepositoryTest extends Specification { } when: - unit.prefetch([new PluginSpec("date-test-plugin")]) + unit.prefetch([new PluginRef("date-test-plugin")]) then: def plugins = unit.getPlugins() diff --git a/modules/nf-commons/src/test/nextflow/plugin/PluginRefTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/PluginRefTest.groovy new file mode 100644 index 0000000000..1a0eb679b0 --- /dev/null +++ b/modules/nf-commons/src/test/nextflow/plugin/PluginRefTest.groovy @@ -0,0 +1,42 @@ +package nextflow.plugin + +import spock.lang.Specification + +/** + * + * @author Paolo Di Tommaso + */ +class PluginRefTest extends Specification { + + def 'should parse a plugin ref' () { + + when: + def ref = PluginRef.parse(FQID) + then: + ref.id == ID + ref.version == VER + + + where: + FQID | ID | VER + 'foo@1.0' | 'foo' | '1.0' + } + + def 'should compare refs' () { + given: + def ref1 = new PluginRef('nf-alpha','1.0.0') + def ref2 = new PluginRef('nf-alpha','1.0.0') + def ref3 = new PluginRef('nf-delta','2.0.0') + + expect: + ref1 == ref2 + ref1 != ref3 + and: + ref1 < ref3 + and: + ref1.hashCode() == ref2.hashCode() + ref1.hashCode() != ref3.hashCode() + + } + +} diff --git a/modules/nf-commons/src/test/nextflow/plugin/PluginSpecTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/PluginSpecTest.groovy deleted file mode 100644 index 8aeb7cadc5..0000000000 --- a/modules/nf-commons/src/test/nextflow/plugin/PluginSpecTest.groovy +++ /dev/null @@ -1,42 +0,0 @@ -package nextflow.plugin - -import spock.lang.Specification - -/** - * - * @author Paolo Di Tommaso - */ -class PluginSpecTest extends Specification { - - def 'should parse plugins spec' () { - - when: - def spec = PluginSpec.parse(FQID) - then: - spec.id == ID - spec.version == VER - - - where: - FQID | ID | VER - 'foo@1.0' | 'foo' | '1.0' - } - - def 'should compare specs' () { - given: - def spec1 = new PluginSpec('nf-alpha','1.0.0') - def spec2 = new PluginSpec('nf-alpha','1.0.0') - def spec3 = new PluginSpec('nf-delta','2.0.0') - - expect: - spec1 == spec2 - spec1 != spec3 - and: - spec1 < spec3 - and: - spec1.hashCode() == spec2.hashCode() - spec1.hashCode() != spec3.hashCode() - - } - -} diff --git a/modules/nf-commons/src/test/nextflow/plugin/PluginUpdaterTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/PluginUpdaterTest.groovy index e552405ab9..5d96d17240 100644 --- a/modules/nf-commons/src/test/nextflow/plugin/PluginUpdaterTest.groovy +++ b/modules/nf-commons/src/test/nextflow/plugin/PluginUpdaterTest.groovy @@ -549,7 +549,7 @@ class PluginUpdaterTest extends Specification { then: // Verify prefetch is called with the correct plugin specs - 1 * mockRepo.prefetch({ List specs -> + 1 * mockRepo.prefetch({ List specs -> specs.size() == 2 && specs[0].id == 'my-plugin' && specs[0].version == '1.0.0' && specs[1].id == 'another-plugin' && specs[1].version == '2.0.0' diff --git a/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy index 351d448085..014f13e21b 100644 --- a/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy +++ b/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy @@ -59,7 +59,7 @@ class PluginsFacadeTest extends Specification { def 'should parse plugins config' () { given: def defaults = new DefaultPlugins(plugins: [ - 'delta': new PluginSpec('delta', '0.1.0'), + 'delta': new PluginRef('delta', '0.1.0'), ]) def handler = new PluginsFacade(defaultPlugins: defaults) @@ -87,11 +87,11 @@ class PluginsFacadeTest extends Specification { def 'should return plugin requirements' () { given: def defaults = new DefaultPlugins(plugins: [ - 'nf-amazon': new PluginSpec('nf-amazon', '0.1.0'), - 'nf-cloudcache': new PluginSpec('nf-cloudcache', '0.1.0'), - 'nf-google': new PluginSpec('nf-google', '0.1.0'), - 'nf-tower': new PluginSpec('nf-tower', '0.1.0'), - 'nf-wave': new PluginSpec('nf-wave', '0.1.0') + 'nf-amazon': new PluginRef('nf-amazon', '0.1.0'), + 'nf-cloudcache': new PluginRef('nf-cloudcache', '0.1.0'), + 'nf-google': new PluginRef('nf-google', '0.1.0'), + 'nf-tower': new PluginRef('nf-tower', '0.1.0'), + 'nf-wave': new PluginRef('nf-wave', '0.1.0') ]) and: def handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) @@ -117,62 +117,62 @@ class PluginsFacadeTest extends Specification { handler = new PluginsFacade(defaultPlugins: defaults, env: [NXF_PLUGINS_DEFAULT:'true']) result = handler.pluginsRequirement([tower:[enabled:true]]) then: - result == [ new PluginSpec('nf-tower', '0.1.0') ] + result == [ new PluginRef('nf-tower', '0.1.0') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [TOWER_ACCESS_TOKEN:'xyz']) result = handler.pluginsRequirement([:]) then: - result == [ new PluginSpec('nf-tower', '0.1.0') ] + result == [ new PluginRef('nf-tower', '0.1.0') ] // fusion requires both nf-tower and nf-wave when: handler = new PluginsFacade(defaultPlugins: defaults, env: [NXF_PLUGINS_DEFAULT:'true']) result = handler.pluginsRequirement([fusion:[enabled:true]]) then: - result == [ new PluginSpec('nf-tower', '0.1.0'), new PluginSpec('nf-wave', '0.1.0') ] + result == [ new PluginRef('nf-tower', '0.1.0'), new PluginRef('nf-wave', '0.1.0') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) result = handler.pluginsRequirement([wave:[enabled:true]]) then: - result == [ new PluginSpec('nf-wave', '0.1.0') ] + result == [ new PluginRef('nf-wave', '0.1.0') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) result = handler.pluginsRequirement([plugins: [ 'foo@1.2.3']]) then: - result == [ new PluginSpec('foo', '1.2.3') ] + result == [ new PluginRef('foo', '1.2.3') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) result = handler.pluginsRequirement([plugins: [ 'nf-amazon@1.2.3']]) then: - result == [ new PluginSpec('nf-amazon', '1.2.3') ] + result == [ new PluginRef('nf-amazon', '1.2.3') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) result = handler.pluginsRequirement([plugins: [ 'nf-amazon']]) then: - result == [ new PluginSpec('nf-amazon', '0.1.0') ] // <-- config is taken from the default config + result == [ new PluginRef('nf-amazon', '0.1.0') ] // <-- config is taken from the default config when: handler = new PluginsFacade(defaultPlugins: defaults, env: [NXF_PLUGINS_DEFAULT:'nf-google@2.0.0']) result = handler.pluginsRequirement([plugins: [ 'nf-amazon@1.2.3']]) then: - result == [ new PluginSpec('nf-amazon', '1.2.3'), new PluginSpec('nf-google','2.0.0') ] + result == [ new PluginRef('nf-amazon', '1.2.3'), new PluginRef('nf-google','2.0.0') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [NXF_PLUGINS_DEFAULT:'nf-google@2.0.0']) result = handler.pluginsRequirement([:]) then: - result == [ new PluginSpec('nf-google','2.0.0') ] + result == [ new PluginRef('nf-google','2.0.0') ] when: handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) result = handler.pluginsRequirement([cloudcache:[enabled:true]]) then: - result == [ new PluginSpec('nf-cloudcache', '0.1.0') ] + result == [ new PluginRef('nf-cloudcache', '0.1.0') ] } @@ -181,11 +181,11 @@ class PluginsFacadeTest extends Specification { SysEnv.push([:]) and: def defaults = new DefaultPlugins(plugins: [ - 'nf-amazon': new PluginSpec('nf-amazon', '0.1.0'), - 'nf-google': new PluginSpec('nf-google', '0.1.0'), - 'nf-azure': new PluginSpec('nf-azure', '0.1.0'), - 'nf-tower': new PluginSpec('nf-tower', '0.1.0'), - 'nf-k8s': new PluginSpec('nf-k8s', '0.1.0') + 'nf-amazon': new PluginRef('nf-amazon', '0.1.0'), + 'nf-google': new PluginRef('nf-google', '0.1.0'), + 'nf-azure': new PluginRef('nf-azure', '0.1.0'), + 'nf-tower': new PluginRef('nf-tower', '0.1.0'), + 'nf-k8s': new PluginRef('nf-k8s', '0.1.0') ]) and: def handler = new PluginsFacade(defaultPlugins: defaults) @@ -234,10 +234,10 @@ class PluginsFacadeTest extends Specification { SysEnv.push([:]) and: def defaults = new DefaultPlugins(plugins: [ - 'nf-amazon': new PluginSpec('nf-amazon', '0.1.0'), - 'nf-google': new PluginSpec('nf-google', '0.1.0'), - 'nf-azure': new PluginSpec('nf-azure', '0.1.0'), - 'nf-tower': new PluginSpec('nf-tower', '0.1.0') + 'nf-amazon': new PluginRef('nf-amazon', '0.1.0'), + 'nf-google': new PluginRef('nf-google', '0.1.0'), + 'nf-azure': new PluginRef('nf-azure', '0.1.0'), + 'nf-tower': new PluginRef('nf-tower', '0.1.0') ]) and: def handler = new PluginsFacade(defaultPlugins: defaults) @@ -279,10 +279,10 @@ class PluginsFacadeTest extends Specification { SysEnv.push([:]) and: def defaults = new DefaultPlugins(plugins: [ - 'nf-amazon': new PluginSpec('nf-amazon', '0.1.0'), - 'nf-google': new PluginSpec('nf-google', '0.1.0'), - 'nf-azure': new PluginSpec('nf-azure', '0.1.0'), - 'nf-tower': new PluginSpec('nf-tower', '0.1.0') + 'nf-amazon': new PluginRef('nf-amazon', '0.1.0'), + 'nf-google': new PluginRef('nf-google', '0.1.0'), + 'nf-azure': new PluginRef('nf-azure', '0.1.0'), + 'nf-tower': new PluginRef('nf-tower', '0.1.0') ]) and: def handler = new PluginsFacade(defaultPlugins: defaults) @@ -323,9 +323,9 @@ class PluginsFacadeTest extends Specification { given: def defaults = new DefaultPlugins(plugins: [ - 'nf-amazon': new PluginSpec('nf-amazon', '0.1.0'), - 'nf-google': new PluginSpec('nf-google', '0.1.0'), - 'nf-tower': new PluginSpec('nf-tower', '0.1.0') + 'nf-amazon': new PluginRef('nf-amazon', '0.1.0'), + 'nf-google': new PluginRef('nf-google', '0.1.0'), + 'nf-tower': new PluginRef('nf-tower', '0.1.0') ]) and: def handler = new PluginsFacade(defaultPlugins: defaults, env: [NXF_PLUGINS_DEFAULT: 'nf-amazon,nf-tower@1.0.1,nf-foo@2.2.0,nf-bar']) @@ -385,9 +385,9 @@ class PluginsFacadeTest extends Specification { def 'should merge plugins' () { given: def facade = new PluginsFacade() - def configPlugins = CONFIG.tokenize(',').collect { PluginSpec.parse(it) } - def defaultPlugins = DEFAULT.tokenize(',').collect { PluginSpec.parse(it) } - def expectedPlugins = EXPECTED.tokenize(',').collect { PluginSpec.parse(it) } + def configPlugins = CONFIG.tokenize(',').collect { PluginRef.parse(it) } + def defaultPlugins = DEFAULT.tokenize(',').collect { PluginRef.parse(it) } + def expectedPlugins = EXPECTED.tokenize(',').collect { PluginRef.parse(it) } expect: facade.mergePluginSpecs(configPlugins, defaultPlugins) == expectedPlugins @@ -471,7 +471,7 @@ class PluginsFacadeTest extends Specification { } def 'should prefetch plugin metadata when starting plugins'() { - def specs = [new PluginSpec("nf-one"), new PluginSpec("nf-two", "~1.2.0")] + def specs = [new PluginRef("nf-one"), new PluginRef("nf-two", "~1.2.0")] given: def updater = Mock(PluginUpdater) @@ -516,7 +516,7 @@ class PluginsFacadeTest extends Specification { ENV | EXPECTED [:] | null [NXF_PLUGINS_ALLOWED:''] | [] - [NXF_PLUGINS_ALLOWED:'nf-amazon,nf-google'] | [PluginSpec.parse('nf-amazon'), PluginSpec.parse('nf-google')] + [NXF_PLUGINS_ALLOWED:'nf-amazon,nf-google'] | [PluginRef.parse('nf-amazon'), PluginRef.parse('nf-google')] } @Unroll @@ -526,7 +526,7 @@ class PluginsFacadeTest extends Specification { expect: facade.isAllowed(REQUEST) == EXPECTED - facade.isAllowed(PluginSpec.parse(REQUEST)) == EXPECTED + facade.isAllowed(PluginRef.parse(REQUEST)) == EXPECTED where: ENV | REQUEST | EXPECTED From 9b8669d504cb2f69419773f9c95960e9c908e658 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 25 Sep 2025 13:36:34 -0500 Subject: [PATCH 3/8] Move JsonRenderer to language server Signed-off-by: Ben Sherman --- .../config/schema/JsonRenderer.groovy | 52 ------------------- .../nextflow/plugin/spec/ConfigSpec.groovy | 4 +- 2 files changed, 2 insertions(+), 54 deletions(-) delete mode 100644 modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy diff --git a/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy b/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy deleted file mode 100644 index 956ca4ea05..0000000000 --- a/modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy +++ /dev/null @@ -1,52 +0,0 @@ -/* - * Copyright 2024-2025, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package nextflow.config.schema - -import groovy.json.JsonOutput -import groovy.transform.TypeChecked -import nextflow.plugin.Plugins -import nextflow.plugin.spec.ConfigSpec -import nextflow.script.dsl.Description - -@TypeChecked -class JsonRenderer { - - String render() { - final schema = getSchema() - return JsonOutput.toJson(schema) - } - - private static Map getSchema() { - final result = new HashMap() - for( final scope : Plugins.getExtensions(ConfigScope) ) { - final clazz = scope.getClass() - final scopeName = clazz.getAnnotation(ScopeName)?.value() - final description = clazz.getAnnotation(Description)?.value() - if( scopeName == '' ) { - SchemaNode.Scope.of(clazz, '').children().each { name, node -> - result.put(name, ConfigSpec.of(node)) - } - continue - } - if( !scopeName ) - continue - final node = SchemaNode.Scope.of(clazz, description) - result.put(scopeName, ConfigSpec.of(node, scopeName)) - } - return result - } - -} diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy index e379172258..985438dfac 100644 --- a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy @@ -28,11 +28,11 @@ import org.codehaus.groovy.ast.ClassNode @CompileStatic class ConfigSpec { - static Map of(SchemaNode node, String name=null) { + static Map of(SchemaNode node, String name) { return fromNode(node, name) } - private static Map fromNode(SchemaNode node, String name=null) { + private static Map fromNode(SchemaNode node, String name) { if( node instanceof SchemaNode.Option ) return fromOption(node, name) if( node instanceof SchemaNode.Placeholder ) From 3e68ef40254103f6c029f94ec40b6b48f3840745 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 25 Sep 2025 16:16:52 -0500 Subject: [PATCH 4/8] Save config schema for each config node Signed-off-by: Ben Sherman --- .../src/main/java/nextflow/config/ast/ConfigNode.java | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/modules/nf-lang/src/main/java/nextflow/config/ast/ConfigNode.java b/modules/nf-lang/src/main/java/nextflow/config/ast/ConfigNode.java index 70713444a7..124fdcc9e5 100644 --- a/modules/nf-lang/src/main/java/nextflow/config/ast/ConfigNode.java +++ b/modules/nf-lang/src/main/java/nextflow/config/ast/ConfigNode.java @@ -18,6 +18,7 @@ import java.util.ArrayList; import java.util.List; +import nextflow.config.schema.SchemaNode; import org.codehaus.groovy.ast.ModuleNode; import org.codehaus.groovy.control.SourceUnit; @@ -30,6 +31,8 @@ public class ConfigNode extends ModuleNode { private List configStatements = new ArrayList<>(); + private SchemaNode.Scope schema; + public ConfigNode(SourceUnit sourceUnit) { super(sourceUnit); } @@ -41,4 +44,12 @@ public List getConfigStatements() { public void addConfigStatement(ConfigStatement statement) { configStatements.add(statement); } + + public SchemaNode.Scope getSchema() { + return schema; + } + + public void setSchema(SchemaNode.Scope schema) { + this.schema = schema; + } } From 20a8c92d66d8b17025dff7b161a7f381c9566424 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Mon, 29 Sep 2025 11:31:08 -0500 Subject: [PATCH 5/8] Fix generic type detection Signed-off-by: Ben Sherman --- .../src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy index 985438dfac..e8a743f6ad 100644 --- a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy @@ -90,7 +90,7 @@ class ConfigSpec { private static Object fromType(ClassNode cn) { final name = Types.getName(cn.getTypeClass()) - if( cn.getGenericsTypes() != null ) { + if( !cn.isGenericsPlaceHolder() && cn.getGenericsTypes() != null ) { final typeArguments = cn.getGenericsTypes().collect { gt -> fromType(gt.getType()) } return [ name: name, typeArguments: typeArguments ] } From 74744824f0f5ef9d0da8e9b6103af6cd4a66e80d Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Mon, 29 Sep 2025 11:48:13 -0500 Subject: [PATCH 6/8] Apply type detection fix to FunctionSpec Signed-off-by: Ben Sherman --- .../src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy index c95bdfc86b..10abc32559 100644 --- a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/FunctionSpec.groovy @@ -58,7 +58,7 @@ class FunctionSpec { private static Object fromType(ClassNode cn) { final name = Types.getName(cn.getTypeClass()) - if( cn.getGenericsTypes() != null ) { + if( !cn.isGenericsPlaceHolder() && cn.getGenericsTypes() != null ) { final typeArguments = cn.getGenericsTypes().collect { gt -> fromType(gt.getType()) } return [ name: name, typeArguments: typeArguments ] } From 8b772f46a574d9c5061b4a17fca849e4a968edc8 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 30 Sep 2025 13:21:21 -0500 Subject: [PATCH 7/8] Apply suggestions from review Signed-off-by: Ben Sherman --- adrs/2025-09-22-plugin-spec.md | 4 +- .../nextflow/plugin/spec/PluginSpec.groovy | 28 +--------- .../plugin/spec/PluginSpecWriter.groovy | 54 +++++++++++++++++++ 3 files changed, 57 insertions(+), 29 deletions(-) create mode 100644 modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpecWriter.groovy diff --git a/adrs/2025-09-22-plugin-spec.md b/adrs/2025-09-22-plugin-spec.md index 1da649de34..8860199b35 100644 --- a/adrs/2025-09-22-plugin-spec.md +++ b/adrs/2025-09-22-plugin-spec.md @@ -64,7 +64,7 @@ Example: ```json { - "$schema": "https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/schema.json", + "$schema": "https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json", "definitions": [ { "type": "ConfigScope", @@ -151,4 +151,4 @@ Defines a channel operator that can be included in Nextflow scripts. The spec is Now that there is a Gradle plugin for building Nextflow plugins and a registry to publish and retrieve plugins, it is possible to generate, publish, and retrieve plugin specs in a way that is transparent to plugin developers. -Plugins specs adhere to a pre-defined [schema](https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/schema.json) to ensure consistency across different versions of Nextflow. In the future, new versions of the schema can be defined as needed to support new behaviors or requirements. +Plugins specs adhere to a pre-defined [schema](https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json) to ensure consistency across different versions of Nextflow. In the future, new versions of the schema can be defined as needed to support new behaviors or requirements. diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy index e4eba7bd5b..26019e175e 100644 --- a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpec.groovy @@ -15,7 +15,6 @@ */ package nextflow.plugin.spec -import groovy.json.JsonOutput import groovy.transform.CompileStatic import nextflow.config.schema.ConfigScope import nextflow.config.schema.SchemaNode @@ -73,33 +72,8 @@ class PluginSpec { } return [ - '$schema': 'https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/schema.json', + '$schema': 'https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json', 'definitions': definitions ] } } - - -@CompileStatic -class PluginSpecWriter { - - static void main(String[] args) { - if (args.length < 2) { - System.err.println("Usage: PluginSpecWriter [class2] ...") - System.exit(1) - } - - final outputPath = args[0] - final extensionPoints = args[1..-1] - - // build plugin spec - final spec = new PluginSpec(extensionPoints).build() - - // write plugin spec to JSON file - final file = new File(outputPath) - file.parentFile.mkdirs() - file.text = JsonOutput.toJson(spec) - - println "Saved plugin spec to $file" - } -} diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpecWriter.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpecWriter.groovy new file mode 100644 index 0000000000..f727579e7b --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/spec/PluginSpecWriter.groovy @@ -0,0 +1,54 @@ +/* + * Copyright 2024-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.plugin.spec + +import groovy.json.JsonOutput +import groovy.transform.CompileStatic + +/** + * This entrypoint is used by the Nextflow Gradle plugin + * to generate plugin specs when packaging a plugin. + * + * It must be defined in the Nextflow runtime so that the + * Gradle plugin can use it without depending on Nextflow at + * compile-time (i.e. the plugin should use the version of + * Nextflow specified by the user). + * + * @author Ben Sherman + */ +@CompileStatic +class PluginSpecWriter { + + static void main(String[] args) { + if (args.length < 2) { + System.err.println("Usage: PluginSpecWriter [class2] ...") + System.exit(1) + } + + final outputPath = args[0] + final extensionPoints = args[1..-1] + + // build plugin spec + final spec = new PluginSpec(extensionPoints).build() + + // write plugin spec to JSON file + final file = new File(outputPath) + file.parentFile.mkdirs() + file.text = JsonOutput.toJson(spec) + + println "Saved plugin spec to $file" + } +} From 2cb1b54721c09b3b0388bad3b286ee5fe130e355 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 7 Oct 2025 16:43:19 +0200 Subject: [PATCH 8/8] Aling ADR naming to other projects [ci skip] Signed-off-by: Paolo Di Tommaso --- CLAUDE.md | 1 + adrs/2025-09-22-plugin-spec.md => adr/20250922-plugin-spec.md | 0 adrs/0000-00-00-template.md => adr/YYYYMMDD-template-name.md | 0 3 files changed, 1 insertion(+) rename adrs/2025-09-22-plugin-spec.md => adr/20250922-plugin-spec.md (100%) rename adrs/0000-00-00-template.md => adr/YYYYMMDD-template-name.md (100%) diff --git a/CLAUDE.md b/CLAUDE.md index 2fce58972f..c88f86a2a4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -104,6 +104,7 @@ The project follows a modular architecture with a plugin-based system for cloud - `build.gradle`: Root build configuration with multi-module setup - `settings.gradle`: Gradle project structure definition - `plugins/*/VERSION`: Define the version of the corresponding plugin sub-project. +- `adr/`: Architecture Decision Records (ADRs) documenting significant structural and technical decisions in the project ## Release process diff --git a/adrs/2025-09-22-plugin-spec.md b/adr/20250922-plugin-spec.md similarity index 100% rename from adrs/2025-09-22-plugin-spec.md rename to adr/20250922-plugin-spec.md diff --git a/adrs/0000-00-00-template.md b/adr/YYYYMMDD-template-name.md similarity index 100% rename from adrs/0000-00-00-template.md rename to adr/YYYYMMDD-template-name.md