WIP: Plugins v3 #2419
WIP: Plugins v3 #2419
Conversation
| allow plugins to provide translations for user facing strings. | ||
|
|
||
| Plugins could provide gettext `.mo` files that will be loaded under a plugin | ||
| specific translation domain. |
There was a problem hiding this comment.
How would that work?
- Plugins have their own domain, but cannot access picard domains, this may be a problem for constants/attributes from database
- Plugins have their own domain, a set of methods from plugin API to use it ((), N() are those from Picard, P_() and PN_() for plugins)
- We append plugin domain to the main one (chaining fallbacks https://docs.python.org/3/library/gettext.html#gettext.NullTranslations.add_fallback), basically Picard main domain is used, but plugin "extends" it, standard picard/i18n are used
- ?
We need to test each approach to find the best.
Loading plugin translations can be either automatic (we expect .mo files to be somewhere identical for all plugins), or by a specific call from the plugin to the API to indicate where to find translations (decided by plugin author). I tend to prefer the first approach, as it is simple, safer, and "normalize" plugins.
There was a problem hiding this comment.
My preference would also be 1. I think plugins should have their own translation generally. For thing like attribute translations we have separate domains anyway. If this becomes a requirement we could consider exposing this via the API as well.
Opiton 3. sounds interesting though, we should probably try how well this works.
There was a problem hiding this comment.
Actually using fallbacks isn't a good idea, as it may leads to conflicts between strings (if we search main domain first and plugin has a different translation for the same string, it will not appear, and the reverse searching plugin domain first and falling back on main, would let plugin translations to override main translations).
Also .mo aren't supposed to be portable between systems.
Though GNU gettext is able to take care of endianess at runtime it seems, it is usually preferable to compile .po to .mo on target system.
An interesting thread about that: https://stackoverflow.com/questions/53285634/is-there-a-portable-way-to-provide-localization-of-a-package-distributed-on-pypi
|
@phw, I rebased onto master and added fixes for imports related to extension_points changes |
|
In I am surprised that you guys have decided to adopt a one-plugin-per-repository model in order to support external 3rd-party (i.e. dubious security) git repos. I know that this can potentially be controlled by limiting the repos to metabrainz owned repos, where you guys can set the security permissions etc., but in practice this will become very unwieldy and become very insecure very quickly. It is also likely that 3rd party repos will have default security (i.e. insecure) which allow merges by any Github user. I urge you to reconsider this approach, because I consider it extremely insecure and therefore likely to eventually be used for malicious purposes and become a brand-image issue. (I really really really do NOT want to be saying "Told you so" in a few years (or months) time.) I see nothing in the pygit2 calling code to verify (by a hash or other means) that the local clone matches the git main source version (and has not been maliciously modified locally). I see nothing in the code which asks the user explicitly to authorise manually installed and locally modified plugins - the hashes for these should be stored locally and reauthorisation requested if the local source code changes. I see no changes to the options page for plugins to distinguish between valid (i.e. has matching hashes) centrally downloaded plugins, manually installed plugins, and plugins that were downloaded from a central repo but have since been locally modified. IMO this should be done by having several sections to the plugin list (manually installed and locally modified sections only being shown when needed - so for most users they will continue to see only a single section). When using git cloning, I am unclear how you tell it to download a release rather than the current commit (which may be a WIP). I am also unclear how the plugins website / API will know when a new release is published. Future plugin upgrades e.g. from v3 to v4: By allowing external plugin repos you are going to make it MUCH more difficult to migrate to a newer incompatible api. At present you can create a v3 branch in the musicbrainz/picard-plugins repo and edit each plugin to meet the new standard. How will you do this when there are hundreds of separate external repos? @zas / @phw Zas / Philipp - I urge you to reconsider this approach. |
|
@Sophist-UK We'll continue to support "official" plugins, and there will be a list of them from an official website. Nothing changes here. User will get a warning about installing "third-party" plugins.
Yes, that's why that's a Draft Pull Request, with WIP in the title... we hardly started to implement this.
And what do you suggest exactly? |
|
@Sophist-UK This PR is all WIP right now. The primary goal of the existing code was to validate and prototype the concepts we had in mind for the new plugin. Also the primary focus so far has been on the plugin structure and API, and ideas about distribution are just coming together. But having said this, having the ability to allow third-party plugin repositories is definitely one of the goals. We have seen that the single repo controlled by Picard developers doesn't scale. This leads to situations like the classical extras. We can (and will) still have official repositories. And we can war users about using third party plugins. But in the end the users can already now add plugins from other places. I don't see how the repository system is much different. But it definitely makes it easier for both users and developers. |
|
Sorry I haven't had much chance to go over this yet. So each (third-party) plugin will have to be in a separate repo, right? I can work with that, but it means that I'll need to create a bunch of new repos 😀 and it won't be quite as easy to get a full list of all of my plugins. Not a big deal, and I'm sure I can work around that somehow. Am I understanding correctly that the third-party plugins will still be installed the same way (by downloading and installing a zip file), and that active plugins will be checked for updates (by querying the plugin repo) during Picard startup? Sort of like subscribing to a plugin? I'm wondering if a potential enhancement might be to allow the user to enter (paste) a url to the plugin repo and Picard could do the download and install? Another possible enhancement might be to allow the user to subscribe to a plugin provider so they can see all plugins available from that provider (similar to the way that they can see all the plugins available from picard.musicbrainz.org/plugins). |
There will be 2 main sources of plugins:
There will be 2 ways to add plugins:
There will be an "official" plugin selector:
The plugin UI will provide means to:
A way for the user to change the ref used by a git-based plugin: it will let a dev to run its beta version, or a user to select a certain version of the plugin. If ref doesn't match "official" list then it's considered a user-installed plugin, and it will not be upgraded automatically. Using git will provide:
Every plugin has 3 mandatory things:
So we drop support for single file plugins (because we want to separate metadata from plugin code, hence MANIFEST file). That's more or less our thoughts about this at this point. But things can evolved along the way, that's work in progress. |
|
@phw Philipp - so not so much WIP as an experiment? Nothing wrong with that, but it does mean different things than WIP. I don't have any issues for e.g. a well supported plugin by a reputable known developer having a separate source (and Classical Extras almost certainly fits this profile) - and we can provide guidelines to ensure that the repo is secure. But that is wildly different from A) switching to Git without ensuring that it will be secure; and B) insisting that every plugin has its own repo (which IMO might be unmanageable in practice - there isn't a huge influx of new plugins, so more of a big one-off change than an ongoing overhead, but still more likely to have human errors in security settings, and any churn on the change approval team will be a big overhead too). Also, git does NOT have a concept of releases (and no concept of beta / full releases either) - these are Github enhancements. Git does have a concept of Tags which you could use (if you insist on being aligned to semantic-versioning). I would suggest that we have 5 tiers of plugins:
All the above would be in the Picard plugin catalogue.
I have no real issue with using a git clone as a means of downloading so long as there is good central and local security control for any plugins in the catalogue and good local security controls and warnings for any plugins not in the catalogue. |
|
Supporting one repo with multiple plugins isn't excluded. But it shouldn't be mandatory either. So we could support 2 structures: For plugin "release" version, we can rely on version from MANIFEST: we update the plugin code only when this version is increased. Then we need to separate code repository from code of the installed plugin. Also what happens when code was changed but version not updated, or the reverse. It would be much more reliable to use git tags as "releases" imho. Repos are resynced on regular basis (automatically or manually). But installing a new version only happens if the version from MANIFEST changed. Actually it would be convenient to manage lists of plugins in a git repo too, so we can easily update it without redeploying Picard website. Each listed plugin has simple metadata: a list of plugin UUIDs, a git URL with a ref (let's called this "Official Plugin List") Each plugin has a MANIFEST listing a version, an UUID (matching the list above), and the rest of metadata (name, description, etc) On first sync, Picard clones official repo containing "Official Plugin List", loads it, and for each of entries, clone matching repositories locally (let's call that "Installable plugin repos"). Then it looks for all MANIFEST files in "Installable plugin repos/repo_name/" or "Installable plugin repos/repo_name/subdir". Now the user can choose to install one plugin:
Now let's explore the evolution over time:
Installing a plugin is basically: cloning to a temp dir (in the destination), with a certain revision, removing .git directory, renaming temp dir to final dir, unloading/reloading the plugin if it was loaded. I really think we need an UUID for each plugin to handle changes of sources (author moves the git repo elsewhere, but the plugin is still the same). We could use an "unique name" (org.metabrainz.picard.plugins.pluginA) instead of an UUID perhaps, not sure about this. Still many questions. |
|
Document zipped plugins as an open discussion, extended the description of items open for discussion.
- documented plugin folder location - separated id and display name in manifest
- they'll be used to install or update plugins - the idea is to support both raw directories and git repos from local media - and of course support for git remote repositories - PluginSources are expected to be configured by UI
|
Few notes after discussion we had with @outsidecontext First steps:
List of plugins available from a known (team-managed) repository:
After, UI, from scratch:
Picard will not install a blacklisted plugin, match domain / path If a plugin changes its url, it is considered as a new plugin (so the plugin list needs to be updated, we need to handle that somehow, perhaps listing previous urls in metadata file?) |
|
Just so I understand correctly, the toml file will be retrieved from the Picard website each time Picard is started, and it will also contain the list of blacklisted sites? If each plugin is going to be contained in a separate repo, does that mean MetaBrainz will be adding a bunch of new repos for the "team supported" plugins? |
Yes, in order to have the blocked plugins effectively disabled in case of security issues, especially those by plugins done in bad faith, the list needs to be loaded early. However, I think we need to inform the user about this. But generally we should have some consent for performing update checks as well.
The idea is to allow multiple Plugins in one repository as wellence the "path" in zas' description above. Authors can use this to have multiple smaller Plugins in a single repo. We would use that for the official plugins as well. |
Summary
Problem
This is the work in progress of a proposed rework of the plugin system.
The main part here is an initial specification of the plugin API in PLUGINS.md. The document builds upon the extended discussion of requirements for a new plugin system on the wiki. But currently the focus is on the plugin API and loading inside Picard, not so much on the distributions side. But @zas had some good ideas here of using git, and parts of this are documented in the spec
Beside the document there is only very rough prototyping to evaluate the ideas a bit. The very last commit does do some initial plugin loading, but this was committed as current state very much in the middle of the work.
This is IMHO not yet ready for review, but as I had little time the recent weeks let's have this up here to build upon. So far it was mostly @zas and myself discussing some ideas. I think the general direction is solid, but we should put this up for a wider discussion.
Solution
Action
Additional actions required: