Share your AI ideas! (NOT AI provider suggestions)

[end-4]

My dotfiles are not only for visual pleasure. The experience I created surely looks good to me and to an average person, which is wonderful. But what really excites me is how it's different: a centered bar layout that's still mouse-friendly, an overview widget, and most recently, an AI widget that does not use a webview.

---

The real question

• Since Google Gemini is free, I'm wondering if anyone has any idea about how else we can utilize AI in the desktop, other than general chatting

Some ideas and problems with them

Colorscheme generator using gemini-pro-vision

• While it can't be worse than pywal, the results so far look uninteresting either due to [1 hue only], [colors look too similar], or [look too much like material colors -> meaningless]
• It doesn't work at all times.

Smart window rearrangement

• Only the window title and class don't seem to be enough for the model to give a good arrangement.

Chat based on window content

• Privacy concerns in case a user accidentally activates it
• We can only use a screenshot but not select text -> how well will that work?...

Keep in mind

• The application should be somewhat complex enough. So far assistants in major OSes are impractical because they only work for basic tasks
• We can tell the model to give a formalized response like a JSON, so don't just think of it as a text generator.

Let me know your ideas!

[clsty]

Talking about json, AI is very useful to generate things in specified format.

e.g. You can give it some markdown formatted text to translate and still formatted as markdown,
while the non-AI translators will often mess up with the part that is formatting-related and should not be translated.

P.S. Google gemini sure is free, but still it's not available in some regions where ChatGPT is.

[midn8hustlr]

Regarding colorscheme generator, I feel that the 8 terminal colors are not much different to provide much meaningful information. IMO, it works great for ags top bar, side bar or overview as it utilizes material-like layout, but for terminal it's kinda useless. I realized this when I was writing a material-like tmux bar config, and picking colors from terminal itself (as its easier to update them dynamically with escape sequences), but the 8-colors look very similar.

Btw, how did you decide how to map the material colors to 8 terminal colors, did you follow any guide or template, or u are the first one to do it? Maybe I will try to look into this to make this more engaging.

[end-4]

Btw, how did you decide how to map the material colors to 8 terminal colors, did you follow any guide or template, or u are the first one to do it? Maybe I will try to look into this to make this more engaging.

No basis. All random.

[midn8hustlr]

Can you elaborate more on smart window rearrangement. Like, what kind of rearrangement are you trying to achieve?

[end-4]

Just grouping related stuff together... Here's the prompt I tried:

---

Init prompt

You are given a JSON array of windows of a desktop environment. Each window is an object containing the following keys: address (which serves as an unique identifier for each window), class (which is basically the window's app name), title (which is basically a summary of the window's current task), workspace (the number of the workspace in which the window is, can only be from 1 to 10. if a window has a negative workspace number, ignore it)- It is your job to reorganize windows across workspaces. That is, sorting related apps together into workspaces for each category.- Steps to perform this task: 1. Evaluating: Take a look at window titles and their classes to see how different windows relate. Put them in categories. Also decide if each category is a foreground or background one (for example, updates or setting apps are background tasks). 2. Ordering: Put foreground categories (that is, every window in a group) in workspaces at start (1, 2, 3, etc.). Put background categories in workspaces at the end (10, 9, 8, etc.) 3. Returning final result: Output your suggestion as a JSON object, with key/value pairs in this format: ADDRESS: WORKSPACE

---

I also gave it some examples

2 cases

User

[{ "address": "0x56459a8dd620", "class": "foot", "title": "sudo pacman -Syu","workspace": 1  },{ "address": "0x56459a8e89b0", "class": "org.gnome.Nautilus", "title": "ags","workspace": 5  },{ "address": "0x56459a8dd670", "class": "Code", "title": "~/.config/ags/config.js - Visual Studio Code","workspace": 4},{ "address": "0x56459a8adc10", "class": "thorium-browser","title": "Untitled prompt | Google AI Studio - Thorium","workspace": 9},{ "address": "0x56459a9460ba", "class": "foot","title": "~","workspace": 10},{ "address": "0x56459a9360d1", "class": "foot","title": "~","workspace": 1},{ "address": "0x56459a9460b6", "class": "foot","title": "~","workspace": 2},{ "address": "0x56459a9420c4", "class": "foot","title": "~","workspace": 9},{ "address": "0x56459a9460b2", "class": "foot","title": "~","workspace": 8},{ "address": "0x56459a9460b1", "class": "foot","title": "~","workspace": 8}]

Model

{"0x56459a8dd620": 10,"0x56459a8e89b0": 1,"0x56459a8dd670": 1,"0x56459a8adc10": 2,"0x56459a9460ba": 9,"0x56459a9360d1": 9,"0x56459a9460b6": 9,"0x56459a9420c4": 9,"0x56459a9460b2": 9,"0x56459a9460b1": 9}

User

[{ "address": "0x56459a98607b", "class": "gnome-tweaks","title": "GNOME Tweaks","workspace": 8,},{ "address": "0x56419a94607f", "class": "org.gnome.Settings","title": "Settings","workspace": 1},{ "address": "0x56459a9460d6", "class": "chromium","title": "GitHub - Chromium","workspace": 7},{ "address": "0x56459a9460a7", "class": "GitHub Desktop","title": "GitHub Desktop","workspace": 6},{ "address": "0x56459aa56769", "class": "foot","title": "~","workspace": 2},{ "address": "0x56459a9193ec", "class": "foot","title": "~","workspace": 3}]

Model

{"0x56459a98607b": 10,"0x56419a94607f": 10,"0x56459a9460d6": 1,"0x56459a9460a7": 1,"0x56459aa56769": 9,"0x56459a9193ec": 9}

[H0mire]

A "Chat with Hyprland" Chatbot. A LLM with added knowledge of Hyprland, dots, ags and more related projects (Wikis, Forums) . Advanced: capable of fixing hyprland. For example add common fixes: ("reinstall_project", "reload ags", etc...) or even suggesting solutions after reading log files (privacy concerns? Use local LLM?).

[end-4]

I guess we can feed the models the hyprland/other relevant docs?
(I just asked Gemini how 2 disable damage tracking and it told me to use gsettings lol)

[H0mire]

Yes this should be possible. As I have no access to Gemini in my region I can only speak about openai. With the OpenAI library for python you can add files as knowledge. Should be the same for other Languages.

[Boforg]

The AI assistant seems really cool! Is there a way for me to just install that widget as I already the rest of my Hyprland set-up. I'm not a programmer(I'm trying to learn) and making my own widget with waybar/eww/whatever else is above my abilities but I'd like to see how yours works. I was considering trying to implement dolphin-2.7-mixtral-8x7b and run it locally through ollama. It's supposed to be able to code pretty well so I was hoping to use it to help troubleshoot as I learn.

[end-4]

you can go into .config/ags/config.js and comment out everything other than stuff related to "Sideleft" in the Windows array. then only the left sidebar will be loaded when you run ags. to toggle it, run ags -t sideleft
idk about dependencies

[Boforg]

Thanks! I'll give it a go later.

[bpawnzZ]

Please add the ability to use custom endpoints! For instance I use https://openrouter.ai/ for interacting with AI. Would you please consider the ability to set Custom API endpoints and model. This would be amazing!. Not a fan of Gemini and OpenAI is overpriced. Openrouter is an amazing service and it would be amazing if you would implement it. Just a suggestion that would make this setup even better. In general props on what you have accomplished.

[end-4]

Actually for now you can edit the PROVIDERS object in ~/.config/ags/services/gpt.js
It's just not very update-friendly in case you actually do. Though I'd be happy to add that api if you provide the needed fields for each provider in that list (name, logo, description, base url, url to get link, model name (if necessary))

[bpawnzZ]

Actually for now you can edit the PROVIDERS object in ~/.config/ags/services/gpt.js It's just not very update-friendly in case you actually do. Though I'd be happy to add that api if you provide the needed fields for each provider in that list (name, logo, description, base url, url to get link, model name (if necessary))

That was actually extremely easy to implement once I knew where the file location was. Thanks for that. I attached the svg for icon i used and I also created a file called openrouter_key in its appropriate location. Works as you would expect. Good shit man!

I added the following to the top of PROVIDERS object in ~/.config/ags/services/gpt.js

'openrouter': {
        'name': 'OpenRouter',
        'logo_name': 'openrouter-symbolic',
        'description': 'A unified interface for LLMs',
        'base_url': 'https://openrouter.ai/api/v1/chat/completions',
        'key_get_url': 'https://openrouter.ai/keys',
        'key_file': 'openrouter_key.txt',
        'model': 'meta-llama/llama-3-70b-instruct',
    },

[bpawnzZ]

[end-4]

cool
added that in 2a11489

[x-leehe]

The Gemini API needs an upgrade, for the gemini-pro API was deprecated a few days ago. See this:

I've already made a modification on my computer myself.

The process:

• Replaced "const CHAT_MODELS = ["gemini-pro"]" to "const CHAT_MODELS = ["gemini-1.5-flash"]" in ~/.config/ags/services/gemini.js Line 47

• Replaced "label: 'Assistant (Gemini Pro)'," to "Assistant (Gemini 1.5 Flash)'," in ~/.config/ags/modules/sideleft/apis/gemini.js Line 37 (So does the tipText in Line 52)

Anyway though, this function is stunning!

[end-4]

i noticed that, but it's for 1.0 pro vision
but it doesn't hurt upgrading, and 1.5-flash seems to be faster, cool 👀

[x-leehe]

Yeah, but the thing is that maybe the gemini-pro stopped working too. Last time (about 4 days ago) it gave me an empty response when I was trying to use it, and then Google emailed me. Thx😉

[bpawnzZ]

I have a suggestion. You should allow the connection to a searxng instance to give the ability for websearch. Searx is implemented in crewai, chat-ui, langchain connects to it. its the 'no api' local way(runs in docker) way to do web search, atleast that i know of. That would really turn up the badass of AI in this project.

I believe this could be very easily implemented with Langchain. Just wondering what u thought about it.

I been running end-4 for a couple months now and like it alot. Nice job by the way

[AndhikaFW]

Add gpt vision and chat with docs capabilities. I got mine patched with those.