Add copilot server example

[chenhunghan]

This PR adds an example HTTP server wrapping exllamav2, which can be used as the server replacing Github Copilot backend.

[19h]

Wow, this is pretty cool 👍

[KaruroChori]

I am having issues with cache = ExLlamaV2Cache(model) failing because of

File "/app/exllamav2/cache.py", line 25, in __init__
    p_key_states = torch.zeros(self.batch_size, self.max_seq_len, num_key_value_heads, head_dim, dtype = torch.float16, device = self.model.cache_map[i])
KeyError: 0

I made some minor modifications to the code so that the model is not downloaded from huggingface, and changed the path to match the one I used for the docker config I added. Still, I don't think this was the cause.
The model used is a 13B llama v2 which is working fine with chat.py.

Was anyone else successful with the original sourcecode in this PR?

[KaruroChori]

I found the issue. You code was missing

model.load()

before cache is reserved.
I need to test if the rest is working, but at least now it does not halt.

[chenhunghan]

I found the issue. You code was missing

model.load()

before cache is reserved. I need to test if the rest is working, but at least now it does not halt.

Thank you :)

[chenhunghan]

I have fixed few bugs, it's more or less in working status. Has been tested with cloud GPUs.

[SinanAkkoyun]

That is very cool. How do insertions work here? Copilot is trained to insert code "in the middle", is that also possible with this endpoint or is the only thing it receives the previous code?

[chenhunghan]

That is very cool. How do insertions work here? Copilot is trained to insert code "in the middle", is that also possible with this endpoint or is the only thing it receives the previous code?

Only the previous code.

[Skoolin]

This is great! Is there a plan to include code insertion / infilling? Codellama has been trained with infilling, but would need the special tokens <PRE>, <MID> and <SUF>. Does the llama tokenizer implementation of exllamav2 support those tokens? Otherwise I might try to implement those, as that would be really useful to me.

[chenhunghan]

I don't have plan as this was just for fun. You can definitely add yours, enjoy the hack 😀