[raylib 6.0 discussion] Keyboard Input Discussion

[JeffM2501]

Problem

The way that raylib currently handles input polling only allows users to poll keys using names from the US ASCII QUERTY layout, This can be confusing for people who have different keyboard layouts.

Terms

Before I start, I want to define some terms, because part of this issue is confusion in terms between different platforms.

Physical Key

This is the actual keyboard switch on the keyboard hardware. It is provided by the actual hardware device and does not know anything about locale or keyboard mapping. It represents a physical location on the keyboard, regardless of what is printed on the keycap.

Each of these keys has an ID. In the case of the 4 red keys, the IDs are 87,65,83, and 68.
Because of history, these codes happen to align with the ASCII characters for W A S, and D, but are considered OEM codes.
A different keyboard with a different layout like dvorak, will still return the same physical key codes for the same red keys.

This is because keyboard mapping is done at the OS level not the hardware level, you can take a QUERTY Keyboard and swap the keycaps around to DVORAK, but the physical switches in the hardware do not change what signals they send. These keys will still send 87,65,83, and 68 even though they are labeled as < A O and E.

Physical keys are useful for game inputs. In a FPS game the developer would want to default the controls to the red keys, regardless of keyboard layout, since that’s where the player’s left hand will sit. The use of physical key codes allows the developer to check the actual device and not have to worry about actual layout for these kinds of inputs.

Raylib’s KeyboardKey is the physical key code. The enums are named based on the US QUERTY layout.
Different platforms call this code different things.

• Windows = ScanCode as part of the wParam on the key event
• GLFW = 'key code'
• SDL = 'scan code'

Virtual Key Code

This is a code that unlike the physical key code, is mapped to the current OS keyboard layout. It returns a value based on what is printed on the key and what the OS uses when the keyboard is pressed. In the examples above, the red keys would return different values based on the keyboard layout.

A user may want to check for specific named keys, not just physical locations for specific types of input in the game.

Raylib does not expose the Virtual Key Code in any way.

GLFW has something called scancode that looks like it would be the mapped item, but it is not.
GLFW does not expose the virtual key in any way.

SDL exposes this as a virtual keycode.

Windows provides this as the 'VK' Code as part of the lParam in the keyboard event.

Key Character

The key character is the printable unicode character that is generated when a key/key combination is pressed by the user. It takes the keyboard mapping, locale, and key modifier states (shift and caps-lock) into account to generate a printable character.

This character is only available for keys that have printable characters, Key such as F1, escape, insert, and numlock, do not generate characters.

These characters are sent as part of input messages to the platform code, separate from the physical and virtual key codes.

Raylib exposes these to the game with the GetCharPressed event functions, and is the basis for all text input.

Keyboard Information Available

There are 4 pieces of info potentially available for each key.

• Physical Key Code (has always been provided by raylib)
• Virtual (Mapped) Key Code (is not available from raylib)
• Key Character (has always been provided by raylib)
• Key Name. (was just added to raylib in 5.5)

Providing access to virtual mapped key codes

The big thing confusing people is that all the poll able codes in raylib are purely physical codes, there is no way to poll a virtual or mapped key. This makes people feel that raylib ‘only works with US keyboards’

I think part of this can be solved with education, making sure that people know that the KeyboardKey enum is physical keys, not virtual ones, since this is what most game input wants.

The recent addition of GetKeyName in 5.5 also allows people to use physical key codes and still provide a localized keymapping GUI, by simply asking what the display name of the key should be.

What we don’t provide is a way for people to truly poll the virtual key code. This would be a use case where someone wants to know when A is pressed, regardless of where A is on the keyboard. I’m not 100% sure this is a common use case. I think the major complaints were due to the lack of GetKeyName, but perhaps someone has a legit use case and can let us know.

Action Items

I think at the very least we need to make it clear in the header that KeyboardKey is a physical key. We should also maybe make an example that uses GetKeyName to show people how to get the mapped name for setup GUIs, perhaps something like the gamepad example but with keys.

If anyone has any use cases that require polling a virtual key, please respond here so we can discuss

[ColleagueRiley]

This issue might be related: glfw/glfw#1502

[orcmid]

I think the recognition of the issue is great there, although I'm not sanguine about the suggested solution.

[orcmid]

@JeffM2501 That's an awesome breakdown of the considerations.

Is there any straightforward way to have access to the platform's knowledge of the assumed keyboard layout, or does GetKeyName solve the problem?

You introduce VirtualKeyCode. Is this like mappedkey is used at #4481 ? Are you associating KeyboardKey with the a derivative of the scan code, the identification that is standardized for layouts of physical keyboards?

(Sorry, the terms being thrown around hurt my brain.)

[JeffM2501]

Is there any straightforward way to have access to the platform's knowledge of the assumed keyboard layout, or does GetKeyName solve the problem?

I do think that GetKeyName does solve the core issue with creating a GUI that lets users map keys to actions. I can not think of any real world use cases that require the use of the virtual key in gaming. The OS already handles text entry and that'd be the biggest one.

I really think the next step is making an example that shows actions bounds to keys with local names in a physical layout.

[orcmid]

@JeffM2501

I do think that GetKeyName does solve the core issue with creating a GUI that lets users map keys to actions. I can not think of any real world use cases that require the use of the virtual key in gaming. The OS already handles text entry and that'd be the biggest one.

I really think the next step is making an example that shows actions bounds to keys with local names in a physical layout.

I'm thinking a great addition to /examples would be one that draws a keyboard layout and names the keytop defaults on the currrent keyboard. Then we could explain it all in an unambiguous manner. I'm uncertain about platforms and setups like X11.

PS: I think your use of "keyboard layout" is ambiguous in the comments about the Dvorak keyboard. I think I would use "key assignment" even though that's a matter for printed keytops and alignment with the operating system.

PPS: I was acquainted with August Dvorak at one time. Watching him debug at an IBM 650's front panel was a riot. He'd get impatient over a hang/crash and start hammering random switches.

[orcmid]

@JeffM2501

I do think that GetKeyName does solve the core issue with creating a GUI that lets users map keys to actions. I can not think of any real world use cases that require the use of the virtual key in gaming. The OS already handles text entry and that'd be the biggest one.

The AAA Games I play all have settings and descriptions of key mappings. One game I have displays a keyboard with keytops identified with their game functions. Most show a KeyCharacter list and default values, plus ways to re-assign them. In the common case of keyboard-mouse "mappings" (the term in the game), it is all KeyCharacter, plus Shift, Ctrl (sometimes L- or R-), and Alt.

Even in a game like Star Wars Outlaws, which can be played via Amazon Luma in my browser (and fortunate because my CPU does not have the features required for the installed game), I believe I can work the settings in the same way.

PS: I just confirmed that play over Luna offers the same settings as if I was playing a desktop install. My laptop does qualify but I find Luna more convenient on my PC. I also see that Amazon is promoting play of more games that I might have. The wonderful thing is that saved games and game progress work across the different ways of playing.

[ColleagueRiley]

@JeffM2501 wparam is used by GLFW for character callbacks.

https://github.com/glfw/glfw/blob/21fea01161e0d6b70c0c5c1f52dc8e7a7df14a50/src/win32_window.c#L684

Raylib already uses the character callback here:

raylib/src/platforms/rcore_desktop_glfw.c

Line 1815 in 1f704be

static void CharCallback(GLFWwindow *window, unsigned int codepoint)

Since Raylib's keys map to the ASCII chars (I think), it should be possible to use the character callback in order to get the keycode press.

[JeffM2501]

Since Raylib's keys map to the ASCII chars (I think), it should be possible to use the character callback in order to get the keycode press.

My fear is that callback only works for printable characters, not modifiers like Control, Alt, Command and Option (on apple keyboards).

I really think we need to get a real world use case that is not serviced by GetKeyName before we think about adding any features to raylib.

[orcmid]

Yes, the prospect of scope creep here is worrisome. It might be necessary to have a clear line in the sand so long as there is a way for folks wanting to handle locality to do it themselves or with supplementary libraries.

[jamesl-github]

what does the on screen keybboard use ? https://support.microsoft.com/en-us/windows/use-the-on-screen-keyboard-osk-to-type-ecbb5e08-5b4e-d8c8-f794-81dbf896267a https://youtu.be/ORhuppDoC1g?si=ZaKIuc33cGbPWkJN&t=25

…

On Thu, Dec 12, 2024 at 2:33 PM Dennis E. Hamilton ***@***.***> wrote: Yes, the prospect of scope creep here is worrisome. It might be necessary to have a clear line in the sand so long as there is a way for folks wanting to handle locality to do it themselves or with supplementary libraries. — Reply to this email directly, view it on GitHub <#4595 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3KGRZY6M6SVCXIDT4IOO32FHQHHAVCNFSM6AAAAABTQLWS4CVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJUHA3DSNI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[orcmid]

@jamesl-github
Windows has a built-in app for the on-screen keyboard. It is configurable by the user. As far as I can tell, its usage is invisible to a conventional Windows app, and I haven't checked to see if there is a way for an app to detect that case. I just typed a sentence in Notepad using my regular keyboard and the on-screen one, in alternation. The only wrinkle is that using my standard keyboard would cause the on-screen keyboard to be closed.

MORE DETAILS:

Windows also has two provisions, that show in the task bar considering keyboards. One is the language choice (e.g., EN) and the other is the keyboard choice.

Sometimes these impact the language of the GUI interface and I am not clear about that.

One characteristic of the on-screen keyboard is there is no function key row by defualt, and it will not look exactly like the layout of the attached physical keyboard. There is also no numeric island, with arrow keys at the bottom under the "?/" and right-side Shift key. There is also an ESC key on the left of the top row.

By the way, on the bottom left, I see an emoticon button and there is a different order of some of those control keys to the left of the space bar.

PS: I did manage to get a screen capture. Here it is for EN, US, on Windows 10.

PPS: And here it is with the FN button depressed.

I don't think there is much connection to my physical keyboard, a wireless Logi[tech] Wave Keys model
YR0096. Using that keyboard is independent of the on-screen one.

[orcmid]

By the way, a trickier case has to do with IME where non-Roman-similar calligraphy is entered. I have no idea how that integrates with an app although games probably avoid it. I suspect that SDL might support it and I have no expertise whatsoever in this aspect.

[JeffM2501]

I have written up a small quick and dirty demo of showing the keyboard map with names and doing a keybinding type GUI. I think it shows that GetKeyName is sufficient for the most basic needs of a raylib game, but I have not tested it on other keyboard layouts/mappings.

keyboard_settings_example.mp4

#include "raylib.h"
#include "raymath.h"

#include <vector>
#include <string>

struct KeyboardKeyItem
{
	float Width = 1;
	KeyboardKey KeyEnum = KEY_NULL;

	std::string Name;
};

std::vector<std::vector<KeyboardKeyItem>> Keyboard;

std::vector<KeyboardKeyItem>* CurrentRow = nullptr;

void AddKey(KeyboardKey key, float width = 1, const char* defindKeyName = nullptr)
{

	const char* name = GetKeyName(key);
	if (defindKeyName != nullptr)
		name = defindKeyName;

	std::string keyName = "??";
	if (name != nullptr)
		keyName = name;

	CurrentRow->emplace_back(KeyboardKeyItem{ width, key, keyName });
}

struct Action
{
    std::string Name;
    KeyboardKey key = KEY_NULL;
};

std::vector<Action> Actions;

bool WaitForKey = false;

Action* WaitingAction = nullptr;

void GameInit()
{
    Actions.emplace_back(Action{ "Forward", KEY_W });
    Actions.emplace_back(Action{ "Back", KEY_S });
    Actions.emplace_back(Action{ "Left", KEY_A });
    Actions.emplace_back(Action{ "Right", KEY_D });
    Actions.emplace_back(Action{ "Jump", KEY_SPACE });

    Keyboard.emplace_back();
    CurrentRow = &Keyboard.back();

    AddKey(KEY_F1, 1, "F1");
    AddKey(KEY_F2, 1, "F2");
    AddKey(KEY_F3, 1, "F3");
    AddKey(KEY_F4, 1, "F4");
    AddKey(KEY_F5, 1, "F5");
    AddKey(KEY_F6, 1, "F6");
    AddKey(KEY_F7, 1, "F7");
    AddKey(KEY_F8, 1, "F8");
    AddKey(KEY_F9, 1, "F9");
    AddKey(KEY_F10, 1, "F10");
    AddKey(KEY_F11, 1, "F11");
    AddKey(KEY_F12, 1, "F12");
    AddKey(KEY_INSERT, 1, "IN");
    AddKey(KEY_PRINT_SCREEN, 1, "PS");
    AddKey(KEY_DELETE, 2, "DEL");

	Keyboard.emplace_back();
	CurrentRow = &Keyboard.back();

    AddKey(KEY_GRAVE);
    AddKey(KEY_ONE);
    AddKey(KEY_TWO);
    AddKey(KEY_THREE);
    AddKey(KEY_FOUR);
    AddKey(KEY_FIVE);
    AddKey(KEY_SIX);
    AddKey(KEY_SEVEN);
    AddKey(KEY_EIGHT);
    AddKey(KEY_NINE);
    AddKey(KEY_ZERO);
    AddKey(KEY_MINUS);
    AddKey(KEY_EQUAL);
    AddKey(KEY_BACKSPACE, 1.5f, "BS");


    Keyboard.emplace_back();
    CurrentRow = &Keyboard.back();
	AddKey(KEY_TAB, 1.5f, "Tab");
	AddKey(KEY_Q);
	AddKey(KEY_W);
	AddKey(KEY_E);
	AddKey(KEY_R);
	AddKey(KEY_T);
	AddKey(KEY_Y);
	AddKey(KEY_U);
	AddKey(KEY_I);
	AddKey(KEY_O);
	AddKey(KEY_P);
	AddKey(KEY_LEFT_BRACKET);
	AddKey(KEY_RIGHT_BRACKET);
	AddKey(KEY_BACKSLASH);

    Keyboard.emplace_back();
    CurrentRow = &Keyboard.back();

    AddKey(KEY_CAPS_LOCK, 2.0f, "CL");
    AddKey(KEY_A);
    AddKey(KEY_S);
    AddKey(KEY_D);
    AddKey(KEY_F);
    AddKey(KEY_G);
    AddKey(KEY_H);
    AddKey(KEY_J);
    AddKey(KEY_K);
    AddKey(KEY_L);
    AddKey(KEY_SEMICOLON);
    AddKey(KEY_APOSTROPHE);
    AddKey(KEY_ENTER, 1.6f, "ENT");

    Keyboard.emplace_back();
    CurrentRow = &Keyboard.back();

    AddKey(KEY_LEFT_SHIFT, 3.0f, "L Shift");
    AddKey(KEY_Z);
    AddKey(KEY_X);
    AddKey(KEY_C);
    AddKey(KEY_V);
    AddKey(KEY_B);
    AddKey(KEY_N);
    AddKey(KEY_M);
    AddKey(KEY_COMMA);
    AddKey(KEY_PERIOD);
    AddKey(KEY_SLASH);
    AddKey(KEY_RIGHT_SHIFT, 3.0f, "R Shift");

    Keyboard.emplace_back();
    CurrentRow = &Keyboard.back();

    AddKey(KEY_LEFT_CONTROL, 3.0f, "LCTRL");
    AddKey(KEY_MENU, 1, "MN");
    AddKey(KEY_LEFT_ALT, 2, "LALT");
    AddKey(KEY_SPACE, 5, "Space");
    AddKey(KEY_RIGHT_ALT, 2, "RALT");
    AddKey(KEY_RIGHT_CONTROL, 3, "RCTRL");
}

bool GameUpdate()
{
	return true;
}

void GameDraw()
{
	BeginDrawing();
	ClearBackground(DARKGRAY);

    Vector2 keySize = { 30,30 };
    Vector2 padding = { 4,4 };
	Vector2 pos = { 10,60 };

    DrawText("Keyboard State", 10, 10, 40, WHITE);
    bool firstRow = true;
	for (auto& row : Keyboard)
	{
		pos.x = 10;
		for (auto& key : row)
		{
			Rectangle bounds = { pos.x, pos.y, keySize.x * key.Width, keySize.y };

			Color bgColor = LIGHTGRAY;
			if (IsKeyDown(key.KeyEnum))
				bgColor = GREEN;

			DrawRectangleRounded(bounds, 0.5f, 12, bgColor);
			DrawText(key.Name.c_str(), pos.x + 5, pos.y + 2, 20, BLACK);

            for (auto& action : Actions)
            {
                if (action.key == key.KeyEnum)
                {
                    DrawRectangleRoundedLinesEx(bounds, 0.5f, 12, 3, YELLOW);
                }
            }

			pos.x += keySize.x * key.Width;
			pos.x += padding.x;
		}
		pos.y += keySize.y + padding.y;

        if (firstRow)
            pos.y += padding.y;

        firstRow = false;
	}


    pos.x = 20;
    pos.y += 20;
    DrawText("Actions", pos.x, pos.y, 40, BLUE);
    pos.y += 50;

    for (auto& action : Actions)
    {
        Rectangle rect = { pos.x, pos.y, 300, 25 };

        if (CheckCollisionPointRec(GetMousePosition(), rect) && IsMouseButtonPressed(MOUSE_LEFT_BUTTON) && !WaitForKey)
        {
            WaitForKey = true;
            WaitingAction = &action;
        }
        DrawRectangleRec(rect, GRAY);
        DrawText(action.Name.c_str(), pos.x + 10, pos.y + 2, 20, DARKBLUE);

        bool found = false;
        for (auto rows : Keyboard)
        {
            for (auto key : rows)
            {
                if (key.KeyEnum == action.key)
                {
                    DrawText(key.Name.c_str(), pos.x + 100, pos.y + 2, 20, BLACK);
                    found = true;
                    break;
                }
            }

            if (found)
                break;
        }

        if (!found)
            DrawText("Unknown", pos.x + 100, pos.y + 2, 20, BLACK);

        if (!WaitForKey && IsKeyDown(action.key))
        {
            DrawRectangle(pos.x + 180, pos.y + 2, 120, 22, YELLOW);
            DrawText("Active", pos.x + 200, pos.y + 2, 20, BLACK);
        }
       
        pos.y += 28;
    }

    DrawText("Click to change action binding", pos.x, pos.y, 20, PURPLE);

    if (WaitForKey)
    {
        DrawRectangle(400, 100, 400, 400, GRAY);
        DrawText(TextFormat("Press a key for %s", WaitingAction->Name.c_str()), 420, 120, 20, BLACK);

        int key = GetKeyPressed();
        if (key != KEY_NULL)
        {
            WaitingAction->key = (KeyboardKey)key;
            WaitForKey = false;
            WaitingAction = nullptr;
        }
    }

	EndDrawing();
}

int main()
{
	SetConfigFlags(FLAG_VSYNC_HINT);
	InitWindow(1000, 600, "Example");
	SetTargetFPS(144);

	GameInit();

	while (!WindowShouldClose())
	{
		if (!GameUpdate())
			break;
		
		GameDraw();
	}

	CloseWindow();
	return 0;
}