Support for auto-completion

[dbuenzli]

It seems a lot of people want cmdliner to help with auto-completion which seems a rather natural request. There are already quite a few things that could help inside (e.g. optional arguments can be specified by a prefix).

1. How do we invoke the tool in completion mode ? environment variable ?
2. For open-ended arguments (e.g. opam package name) how do we specify a user-defined function to be called ? Do we extend 'a converter or do we add (OCaml) optional arguments to command line argument combinators ? The problem is, is there already sufficient context at that point for the thing to be usefull ?
3. How can we make it shell independent ?

• bash completion tutorial
• opam completion
• genet completion

[bobot]

I made in the past some experiment for automatic generation of completion function for zsh. That can give ideas or be a base of code: https://gist.github.com/bobot/9989037

[bobot]

For restarting the conversation:

How do we invoke the tool in completion mode ? environment variable ?

Since one should take into account the --help command and not do any initialization, a specific hidden command line argument is possible --cmdliner--completion that must be the first argument. The last provided option is the one to complete. The option after the point of completion are removed (it is usually what I experimented with the completion of other tools)

The problem is, is there already sufficient context at that point for the thing to be usefull ?

I think two mechanisms must be provided for keeping the simple case simple and yet provide way for complicated case:

1. For context independent completion: The Cmdliner converter is extended for handling this case. For dir and file one optional wildcard pattern can be given for guiding the completion. Moreover they can be put statically in the completion file for avoiding calls to the program but in this case the completion script must be regenerated and installed more often during development.
2. For context dependent completion: Another function is provided for the completion case where required options become optional and the argument to complete is given specially.
   [Given of 'a | Absent | ToComplete of string] for the required case and [Given of 'a | ToComplete of string] for the optional case

How can we make it shell independent ?

Is it really needed? The completion code must be put in a different directory for each shell. So they can be different.

[dbuenzli]

Thanks for these comments. They are not being ignored. It's just that I have too much other things in my head and todo at the moment.

[dra27]

A couple of quick thoughts triggered by another discussion. I agree that hidden (though of course documented!) command line switch seems the best bet, but how about making it one that would be mostly invalid for normal command line syntax? So foo ++cmdliner++auto or something?
I also agree that as shell-independent as possible would be good, inasmuch as the library user should not have to write any shell code at all, ideally. Given that any non-trivial completion will need to invoke the command at some point, it should be OK that the syntax completion is mostly implemented in the program itself.
Bash-style completion is (probably) the most common, which would guide the kind of input (either via environment or command line) and output which Cmdliner would expect with this command switch, and it should be relatively easy to wrap a bash-style completion interface with a shell-specific stub script for converting to/from other shells' requirements (see, for example, https://github.com/dra27/opam/blob/windows/shell/opam_completion.lua#L106-L111 and https://github.com/dra27/opam/blob/windows/shell/opam_completion.lua#L267-L292 converting a Lua interface into Bash-style to allow line-by-line translation of a bash completion script).
Perhaps the shell-script stubs is also something Cmdliner might generate automatically (as with manpages) - e.g. foo ++cmdliner++auto --shell=bash which generates text to put into foo-complete.sh for installing?
It's then "just" a matter of supplying appropriate hooks in the command line itself 😄

[dbuenzli]

A couple of quick thoughts triggered by another discussion. I agree that hidden (though of course documented!) command line switch seems the best bet, but how about making it one that would be mostly invalid for normal command line syntax? So foo ++cmdliner++auto or something?

No, whatever you'll find it could be a legitimate positional argument if appropriately quoted and end up confusing and limiting the end-user. If you use a regular command line option you never limit the end-user's input capabilities, you only constrain the design space of the developer who can't use it in its program.

[AltGr]

1. A new argument to --help would seem the best fit to me
2. I don't think the converter can be a good fit there indeed, it lacks too much context. Having the developers define their own functions doesn't seem to be a problem to me — either in the program itself or in the completion script.
3. Letting the developers write the scripts sounds fine to me, although templates could be nice.

I would already be happy with any --help output that I don't need ugly sed scripts to parse :)
See for example https://github.com/OCamlPro/opam/blob/05b057fc864d2edeb38271b2c675461eb30a657a/shell/opam_completion.sh#L37-L50 , which gets the DIR label after opam --root.

[talex5]

0install does completion with multiple shells (bash, fish, zsh). There are some scripts that get installed in the shell's config: https://github.com/0install/0install/tree/master/share

These invoke 0install _complete, which is a hidden sub-command. I guess doing something with --help would be more general.

There are some normalise methods that try to correct for the differences between the way various shells report the current state: https://github.com/0install/0install/blob/master/ocaml/completion.ml

Options names and subcommands get completed automatically, but there's lots of special-case code for option and argument values.

This was about the first OCaml code I ever wrote, so probably a bit messy ;-) But it might flag up some nasty edge-cases, e.g. https://github.com/0install/0install/blob/master/ocaml/completion.ml#L197

[chetmurthy]

Coming late to this conversation, but I thought I'd add that cmdliner's support for multiple subcommands is really useful, and if shell-completion is added, it would be good if it interacted well with multiple subcommands. Just 2c.

[dbuenzli]

This issue annoys me very much because after all these years I still don't really know what to do about it. The only step that was taken is to reserve the --cmdliner option in 784e829.

Today while washing the dishes I thought maybe the best way to try to move on this is that when cmdliner is invoked with --cmdliner it should simply dump on stdout all the static information it knows about command lines in some form of structured data format like json or sexp.

Maybe then people can try to go on to experiment developing generic tools taking that data and further annotation to instruct how to complete given arguments (e.g. on opam install complete positional arguments with the result of opam list -s).

This means to side step the problem completely and not try to give any support for it in cmdliner, or at least before we have a pretty good idea of what it entails (e.g. what to invoke for completion by completion scripts could eventually get attached to arg values and directly spit out in the data on --cmdliner).

[dra27]

In terms of the output formats, it might be useful to have something which is "trivially" parsed in a shell-script - just thinking for example of src/state/complete.sh in opam, which at present greps the manpages.

[bobot]

My two cents, would be to version this output format. I hesitate if the version should be selected in the code or on the command line. The first version 1 could provide the simplest one which would only list the options name. So that we can start with something like:

_foo() 
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts="$(foo --cmdliner 1)"

    if [[ ${cur} == -* ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
        return 0
    fi
}
complete -F _foo foo

Should cmdliner provide a command for generating those files for the common cases and shells?

[dbuenzli]

Given the reactions I think my proposal was misunderstood.

The idea is not to help you write these terrible shell runes directly by allowing you to invoke --cmdliner and work on that.

My proposal is to dump all the static information (including doc strings) cmdliner has in a clean and structured format (json or sexp). Using this information (and likely additional annotation for defining context sensitive argument name completion), tools can then certainly be developed to actually meta-program these horrible completions shell scripts in a generic manner.

If something good stabilizes I'm not against integrating these tools in the cmdliner project itself, but for now I prefer if people develop and get them right outside.

[bobot]

Ah I see. If things are static, why do you prefer a structured format instead of an OCaml API? I though the --cmdliner argument would be kept for dynamic queries, such as ultimately resolving the existing package for opam install completion.

[dbuenzli]

Ah I see. If things are static, why do you prefer a structured format instead of an OCaml API?

I don't know, that could be an idea. But I have the impression that an API

1. Would somehow be paramount to expose the internal datastructures of cmdliner
2. Might be less convenient to work with. You'd need to expose your terms which may not be that natural for simple programs.

But maybe indeed simply providing introspection on Term.t values may do it. I will have a look once I eventually get to #123

such as ultimately resolving the existing package for opam install completion.

I think opam install is a good exemple on why these things are hard to solve and likely not best done in the program itself but via invocations of the program itself (i.e. to complete opam install PKG you invoke opam list -s).

Before being able to opam list to complete you somehow already need to have determined the configuration which is in itself determined by the command line. So you start having to design your program to support partial command line specifications or at least different working mode, it feels like the UI is now creeping into your model.

[dbuenzli]

Your comment actually made me think of something else. To simplify let's just start with option/command names. No option values and no positional argument values.

What about using the --cmdliner option to denote the position of the user insertion point ‸. So for example:

opam list --b‸ --short             # Given prompt line and a request for completion
opam list --b --cmdliner --short   # Invoke cmdliner that way and you get on stdout
--base
--best-effort

I think this would be pretty easy to do and it wouldn't require anything at all from current cmdliner users. Would that be useful for writing these completion script ? Would that enable to write them in a generic manner ?

We might actually need two options, one for denoting the insertion point right after a printable char and one for after whitespace for later trying to complete option arguments and positional argument:

opam list --short‸
opam list --short --cmdliner-finish
--short

opam list --short ‸
opam list --short --cmdliner-start
angstrom
ask
astring
b0

(Of course --cmdliner-* are context sensitive optional abominations but it's not meant for end users).

[bobot]

We might actually need two options, one for denoting the insertion point right after a printable char and one for after whitespace for later trying to complete option arguments and positional argument

It could be simpler if the empty string is used when completing after a whitespace. --short --cmdliner-finish would be --cmdliner=--short and --short --cmdliner-start would be --short --cmdliner=. It seems that in all the frameworks getting the current option completed is simple, having to handle two different calling methods would be more complicated.

Would that be useful for writing these completion script ?

It would be clearly helpful, it is similar but simpler than 0install method pointed by @talex5 .

Would that enable to write them in a generic manner ?

It seems that in 0install they have to handle in OCaml some differences between the shell https://github.com/0install/0install/blob/master/src/cli/completion.ml#L185 . But it could be only in more complicated cases.

[dbuenzli]

Thanks for your input @bobot.

It could be simpler if the empty string is used when completing after a whitespace. --short --cmdliner-finish would be --cmdliner=--short and --short --cmdliner-start would be --short --cmdliner=. It seems that in all the frameworks getting the current option completed is simple, having to handle two different calling methods would be more complicated.

If that works for the completion scripts, seems even better. (I certainly need to force myself to read and understand at least one of these shell completion systems at some point).

So basically the --cmdliner option takes an argument and the argument is the user input to complete. Maybe we could even use --cmdliner-complete, I bet no tool out there should be using that one. It would be clearer an we could still reserve --cmdliner for other more general introspection stuff if it ever happens.

It seems that in 0install they have to handle in OCaml some differences between the shell

Yes, I don't expect shells to behave in a consistent way otherwise this issue would likely be resolved. When I say in a generic manner, I mean can we write one completion script per shell for all cmdliner tools (leaving aside for now tool specific completions like e.g. package names in opam).

[dbuenzli]

So for the past two hours I tried to understand how zsh completion works (because that's the shell by default on my system). I didn't manage, I feel very dumb.

I understand I can simply register my own function _fun for completing tool tool by doing:

compdef _fun tool

So I thought as a first step I wanted to try to provide a _cmdliner_generic function that would use the mecanism we devised above with @bobot. By just finding out about the context and invoking the tool with the cli and the --cmdliner option at the completion point.

But mind you I couldn't find out which arguments are actually given to the function and how you are supposed to return your result – except by delegating to other obscure utility functions with insane syntaxes. These shell people really have a problem with the notion of function.

Another thing I found out is that, the OCaml Arg module provides excellent completion capabilities :–) just do for example:

compdef _gnu_generic ocamlc

(This relies on parsing the output of --help)

This leaves me a bit wondering whether the more complex mecanism planned above is worth doing w.r.t. to just dump the options of a command and their doc string on an invocation --cmdliner.

[dbuenzli]

So after a few more hours in this insanity I think this was the document I was looking for. Trying to understand this document leaves you wondering whether this complexity is actually needed or whether that is just bad design.

In any case the rough idea is that the generic completion script will look like this:

function _cmdliner_generic {
    local comps
    words[$CURRENT]="--cmdliner-complete=${words[$CURRENT]}"
    comps=("${(@f)$(eval ${words})}")
    compadd -a comps
}

Basically this gets the current cli inserts --cmdliner-complete in front of the string to complete and invokes that cli. Cmdliner should be able to output what it knows could be expected at that point. (I believe something equally simple can be written for bash with COMP_WORDS COMPREPLY).

I'll try that as a first step but it is likely too primitive. We want more meaningful exit codes and/or structured output. In particular for:

1. Whether the cli is already in error at that point.
2. Whether an option argument value is expected whose domain we can't determine (but that an extension of _cmdliner_generic could pick up).
3. Whether a positional argument is expected whose domain we can't determine (but that an extension of _cmdliner_generic could pick up).

Basically 2+3 are for "dynamic" completions (e.g opam package names). Here again moving slowly I think it's preferable if we try to solve that in completion scripts themselves.

An idea is that in these cases we output the position argument or option value ~docv meta variable prefixed by CMDLINER:. The astute shell programmer can then easily search and replace these tokens in comps above with dynamic completions.

For example trying to complete:

 opam list --depends-on ‸

would lead the generic script to invoke

opam list --depends-on --cmdliner-complete=

which would output (as per current opam list --help docs):

CMDLINER:PACKAGES

which could be replaced in an extension of _cmdliner_generic by the results of a call to opam list -s -all.

Maybe more context is actually needed, e.g. which sub(sub)command, so it would rather be CMDLINER[:CMD]*:METAVAR (CMDLINER:list:PACKAGES in the example above).

[aryx]

Seems like the Command library from Jane street support this auto-completion feature:
https://dev.realworldocaml.org/command-line-parsing.html#scrollNav-6
maybe you can just copy their code to implement the same for Cmdliner?

[aryx]

Any update on this issue @dbuenzli ?

[aryx]

Can we get at least a v0 doing some basic autocompletion.

[Alizter]

@aryx We have a WIP completion PR for Dune: ocaml/dune#6377

Cmdliner is vendored in vendor/ and there are some modifications to allow for completion that could be useful.