How to build the chatbot of graphic programming language?

[SunriseB]

Hello! I am a student working on a graphical programming chatbot and currently facing challenges in implementing a chatbot specifically for graphical programming languages（Blockly）. I would like to ask how you managed to implement it?
I currently have two ideas:

1. By fine-tuning an open-source large model to learn the programming syntax of graphical programming (based on XML), but this method requires high quality and quantity of fine-tuning data, which I currently do not have enough of (if you have any good data processing suggestions, please share them with me! Thanks.)

2. Using ChatGPT's GPT store feature to create my own GPT specifically for answering graphical programming questions, but how to teach GPT the syntax of graphical programming is also a challenge.

Based on this, I would like to ask how feasible it is to implement a chatbot for graphical programming? If possible, could you share your successful methods? I would be very grateful!

Thank you! I look forward to further communication.

[yutakobayashidev]

Hello @SunriseB! I have a few ideas.

There's a project called CreatiCode Scratch that provided inspiration for creating this project. It seems that this project tunes the prompts so that ChatGPT generates pseudocode for Scratch, as Scratch does not have a standard syntax.

https://community.openai.com/t/creaticode-scratch-plugin-enable-chatgpt-for-block-based-coding/193493/8

However, since I wanted to convert the blocks generated on the web directly into code and try them out in the browser immediately, I couldn't bear the effort of converting the pseudocode into XML. Instead, I took an approach to directly generate XML, and so far, simple blocks are being generated.

While developing this system, I also considered fine-tuning for XML generation, but I haven't tried it because I'm not sure if it would yield results worth the cost.

Perhaps we can collaborate in some way. Do you have any contact details?

Let me know if you need any adjustments!

[SunriseB]

Thx for reply!
I have used the CreatiCode Scratch and i noted a very important function which i don't know how to achieve it: generate a block program screenshot.

In my mind, I think it can be achieved by Putteteer, a tool to execute headless actions(such as screenshot). I could write a js script which gets an input with xml codes, and gets an output with the block program screenshot after rendering completed. But it is too slow for the user, and so far I have not completed this function.

This function is very important to me, and I will try to find the person in charge of CreatiCode Scratch project to find a way to implement it. If you have a good method and are willing to tell me, that would be even better!

[yutakobayashidev]

@SunriseB

I'm honored that you have reached out to me!

By the way, in this blockly-bot project, there is a feature where AI can generate blocks and preview those blocks as images, and copy them to the workspace.

https://github.com/yutakobayashidev/blockly-bot/blob/3cf1bc3ec80c6cc82682831ba675f267a6e2b2c3/apps/frontend/src/lib/helper.ts#L5C1-L81

blockly-bot/apps/frontend/src/Blockly/BlocklyComponent.tsx

Lines 149 to 167 in 3cf1bc3

	if (!hiddenWorkspaceRef.current) {
	hiddenWorkspaceRef.current = Blockly.inject(
	document.createElement("div"),
	{
	readOnly: true,
	}
	);
	}
	const workspaceSvg = hiddenWorkspaceRef.current;
	Blockly.Xml.domToWorkspace(
	Blockly.utils.xml.textToDom(JSON.parse(res).xml),
	workspaceSvg
	);
	const block = hiddenWorkspaceRef.current.getTopBlocks()[0];
	workspaceSvg.clear();
	const base64Image = await blockToPngBase64(block);

Apologies if you have already confirmed this!

If it's not a headless screen like GPTs, I think the easiest approach to generate screenshots of block programs is to prepare a transparent Blockly workspace instance and convert the blocks into images.

The idea of using Puppeteer is also good, but with other approaches, it might be possible to build a URL parameter based on a pseudo-syntax and create an API to dynamically generate images.