Blank Project Template / NLP Project Migration Option

[pmbaumgartner]

I had an existing NLP project that didn't make heavy use of a spaCy pipeline, but I wanted to start transferring over to v3 and start integrating. None of the templates for projects really fit my use case, so I found myself removing most of what was in an existing template on the project repo. Because of this, I was wondering:

1. Are there plans to include a "blank" template in the projects repo? Something that doesn't include any assumptions about what type of pipeline or model exists, but creates all the key/value combinations for the project.yml file?
2. Additionally, what about a project "migration" toolkit that helps people port existing projects, even if not yet a spaCy based project?

I also did a bit of my own problem solving and built my own tool for this use case because I wanted to try out questionary. It's fairly simple, but should be easy enough for to adapt to whatever is necessary if it's helpful.

🔗 https://github.com/pmbaumgartner/spacy-v3-project-startup

[ines]

Oh wow, this wizard is super cool 😍 Thanks for sharing!

I think we initially considered adding a project init command but decided against that in favour of more specific clone-able project templates. We could definitely provide a pipelines/empty template or similar, though, and the advantage of doing it in the projects repo would be that we can edit it independently of the library and update it without requiring a new version of spaCy.

Additionally, what about a project "migration" toolkit that helps people port existing projects, even if not yet a spaCy based project?

I really like that idea and I'm very keen to explore this further 🤓 What did your not-yet-spaCy project look like, was it mostly standalone scripts? If so, maybe the wizard could ask the user to provide the path to each script and whether the script requires any inputs/produces any outputs and then generate a command for each script. It could also ask about any input data files (local paths or URLs) and add them to the assets block. It'll be difficult to cover all use cases and we don't want to try and be too clever either – but even if we're able to just wrap whatever is already there as a project, it could feel very magical and immediately give the user an idea of how the projects system works and the benefits it provides.

If the migration toolkit/wizard lives in a separate package, it's also much easier to keep it up to date and make it use other libraries and tools. For example, we wouldn't want to include questionary as a dependency of the core library, but it can totally be a dependency of the migration toolkit.

[pmbaumgartner]

I love the idea of an 'empty' template. There's already a lot of great guiding comments in the examples in the projects repo, I think it would just be pulling all of those together and giving some simple keys/values in the YAML file.

The existing project was mostly standalone scripts. We had multiple, static flat files as data sources that each needed to go through some individual processing steps then be recombined. Each input text also had to go through some domain-specific preprocessing (regexes that expanded abbreviations, mostly) that I also wanted to document in some way -- so some of the steps had supplementary terminal outputs that were information about how the data was processed. Once we had the texts preprocessed, the remaining task was making a model, which I've been using embeddings from sentence transformers + logistic regression to do recently as a strong baseline. Because of all of these moving parts and dependencies, I'd love to wrap it all up in a spaCy project... the limitations of my markdown files and somewhat-organized scripts were demonstrated to me as I tried to explain how everything worked to hand this project off before paternity leave 😬

I also agree that that templating the assets combined with the potential scripts/commands is challenging. Maybe there's some sort of happy path there that covers most use cases, but I think the possibilities for inputs and outputs depending on the project is probably too large to find some sort of general way to template that out. The wizard just currently puts the paths and md5s of a folder of assets in the project.yml and that's it. I also don't touch remotes in the wizard because I didn't need them, but that could be another simple thing to set up if necessary.

Wrapping existing projects is a good way to introduce the spaCy "way of thinking" about projects by anchoring it in something that the user is already familiar with, so I'm in total agreement that could be a magical way to learn things.

Finally, I'm not sure where inspiration for spaCy projects idea came from, but it reminded me of Kedro, which is similar but used more generically for data science projects. Their system is more heavyweight, but I thought their tutorials were a great way of getting up to speed on their framework and understanding all the moving parts.