MNIST NN with (Q)Keras on Cyclone V always wrong numbers, data type conversion

[JD92-G]

Hello,
I'm new to hls4ml. Thank you to all for your effort in this great work.
I try using it for creating a normal dense neural network for MNIST and migrate it to a Cyclone V FPGA. I do this for my bachelor thesis and I'm new to every part in it.

I worked out the work flow, including the Intel HLS Compiler. I get the NN flashed properly on the FPGA via Quartus, but in the end the NN always shows me wrong numbers for the mnist data set, for example the first mnist test data should be a 7, but fpga says 2 and so on.
I start getting desperate on what I'm doing wrong for weeks now. I tried it with Keras and with QKeras, both resulting in the same problem. I tried feeding the neural network with uint8 and I tried it with fixed point, but it seems to be a long chain and a little mistake or wrong conversion somewhere leads to wrong results.

But I think I haven't fully understand about when which conversion takes place and how it differs in (Q)Keras and hls4ml.

I haven't understand how to do testing with Intel HLS and the Quartus backend, so all I do is creating a ROM within the FPGA which contains the raw MNIST data and which I can change with switches on my Cyclone V board to feed it different pictures. The NN definitly works, as it shows Busy, Done and the output, which I coded onto the LEDs.

Can anybody here give me a hint about how to start the NN with QKeras, so it is build properly with hls4ml?
I would prefer to just feed the NN 784 (28x28 pixels) uint8 words. And then the NN should normalize and convert it into fixed point itself. Is Qkeras even capable doing this? What would be the best way?
And why does hls4ml change the data type different from what my QKeras model is giving him? For example even though I try everything in QKeras to start with uint8, hls4ml takes the standard 16 bit fixed point anyways.
inp = Input(shape=(784,), dtype="uint8", name="input_uint8")

`x = QActivation(
    activation=quantized_bits(bits=8, integer=8, keep_negative=False),
    name="input_quant"
)(inp)`

I know I'm really bad in understanding all these packages and workflows in such a small time. Can anybody please help me a little bit and give me some understandable clues, about where to begin? Thank you very much in advance.