[BUILD][MISC] Rewritten all Makefiles for efficiency and correctness

[benliao1]

In the past, our builds / Makefiles had two problems:

1. They were inefficient (i.e. compiling all the tests took a really long time; or compiling all of Runtime)
2. (more importantly) they were "incorrect", in the sense that you could change the dependency of a file, run ./runtime build again, and not have all dependent files be rebuilt again.

This PR fixes these issues by:

1. Compiling all source files into object files first, then linking them together into executables
2. Using the -MMD and -MP compiler flags to autogenerate dependency files for each object file

This does make our Makefiles somewhat more complex, however, I tried my best to keep the number of syntax features to a minimum while still getting the job done in a more or less general way.

In addition to the Makefile changes, I fixed / cleaned up / added a few things:

• Reintroduced the PR code format checking job that was in the old Travis CI job but was left out of the original Github Workflows PR
• Changed all #include of Runtime headers to use angle brackets instead of quotes with relative path names (better readability!) since our Makefiles are able to support that now
• Fixed a few compiler type mismatch warnings in dev_handler.c and dev_handler_message.c, as well as incorrect use of printf in test.c
• Cleaned up .gitignore so as to only ignore the necessary folders / files; we used to have a copypasta .gitignore from somewhere that had way too many entries and might lead to confusion for new members
• Removed a couple of the clang files in the root directory to reduce confusion for new members; I also removed the run_clang_format.py script and replaced it with a much simpler, in-house script that I wrote in bash that does basically the same thing. Run a check with ./runtime format check and fix code to comply with the standard with ./runtime format fix

[benliao1]

A lot of small extra cleanup / QOL items were added in this PR. The tests/Makefile in particular got a lot of extra functionality, i.e. it is now possible to run all of the following:

• make net_handler_cli to make just the net_handler_cli
• make cli to make all the CLIs, but nothing else
• make GeneralTestDevice to make just the GeneralTestDevice
• make devices to make all virtual devices, but nothing else
• make tc_150_1 to make just tc_150_1
• make integration/tc_150_1 (same as above)
• make tests to make all test programs
• make all to make all devices, CLIs, and tests
• make help to list all of the above options

I also managed to silence some formatting warnings in the compilation that was cluttering our CI output, and write our own formatting bash script specific to Runtime that is much much simpler than the Python script we had before.

The test script now can take ./runtime test tc_150_1. You used to have to specify either the integration/ or performance/ folders, i.e. with ./runtime test integration/tc_150_1. Both of these now work.

I also deleted the scripts/install_raspi.sh and scripts/provision_raspi.sh scripts because they aren't being referred to and don't really belong in the repo. We should create more wiki pages with how to provision / install Runtime on Raspberry Pi's (or put it in the root README.md)!

[levincent06]

What's this do?

[benliao1]

This is a little complex. So basically the -MMD -MP compiler flags that I added to CFLAGS at the top of Makefile_defs tells the compiler that, when compiling a source file into an object file, to also create a "dependency file" for that object file. The dependency file is in the format of a Makefile and basically lists out all of the sources and header files that the compiler needed to look at in order to generate the object file. The dependency file has the same name as the generated object file, except with a .d extension. So we include all of the .d dependency files associated with the object files needed to make this executable with this line. Thus, when we try to compile a specific object file, make will know which sources and headers that object file is dependent on, and if any of those are newer than the object file we are trying to make, then the object file will be recompiled.

Here are a couple of good articles explaining it:

https://nathandumont.com/blog/automatically-detect-changes-in-header-files-in-a
https://stackoverflow.com/questions/8025766/makefile-auto-dependency-generation
http://www.microhowto.info/howto/automatically_generate_makefile_dependencies.html

[levincent06]

small nit: suggest putting the in-line comments above the line of code for easier readability

[benliao1]

agreed, changed!

[jisungk2]

Why are we adding another parameter, include_dirs here?

[benliao1]

Ah, yes. So when Cython generates ("cythonizes") the studentapi extension (which is how the StudentAPI is currently implemented--perhaps this will change when @BrandonWong14 takes a crack at it later this summer), it needs to know where to find the header files in its C code. Previously, we used quotations to include header files in the student API (i.e. #include "some_header.h") but since i took all of those away and changed them to angle brackets (i.e. #include <some_header.h>) for better readability, I had to specify which directories to look for these headers in this line to create the Cython package.

The official Cython documentation has this page describing how it works.

[jisungk2]

Looks good!
All the Makefiles are in similar formats and they are much more readable than before.
It wouldn't be too difficult to create another Makefile following your format if we were to start working on a new directory.
Thanks for cleaning up the codebase Ben :D

[jisungk2]

Ben: "basically it tells make that the thing listed after .PHONY is not a real file; in this case we don’t actually make a file called all, clean, or dev_handler (we make a file called ../bin/dev_handler)
basically it prevents make from redo-ing commands, like say we want to have make dev_handler build ../bin/dev_handler. if we don’t declare dev_handler as .PHONY then when you run make dev_handler it will always try to look for a file called dev_handler, which will never exist (cuz really the file is ../bin/dev_handler ) so it will run the compilation command again when it doesn’t need to"

Adding it here for future reference.

[benliao1]

Just to clean up this message a bit since that was something I hastily wrote in Slack:

If you're completely confused, read our wiki page on make here.

But in short, this Makefile contains the targets all, clean, and dev_handler. However, none of these targets are real files (there is no file at runtime/dev_handler/all, runtime/dev_handler/clean, or runtime/dev_handler/dev_handler). That last one might be a bit confusing--the dev_handler executable is located in runtime/bin/dev_handler, so from the perspective of the directory that this Makefile is in, the path is ../bin/dev_handler.

Thus, if we just left the Makefile with these three targets, each time we ran make all or make clean or make dev_handler, it would try to find a file called all, clean, or dev_handler, respectively, fail to find the file (since we never make a file with that name), and always run the recipe (which is not what we want to do!). We only want to run the recipe if there is something to be made.

Thus, we tell make that these three special targets are not real files by labeling them as .PHONY. Thus, when we do make dev_handler, it will it won't automatically build everything just because it can't find the dev_handler file (which will never exist).