Git Notes Line Endings

It is remarkable that in the year 2019 we are still dealing with an issue introduced by the workings of a manual typewriter! But alas, here we are. Windows text editors will often add CRLF (Carriage Return + Line Feed) to end of a line in a text file. Depending on your system, this may cause errors when running FDS. When viewed by a the vim text editor with the -b option, you will see a ^M at the end of the line.

We try to automatically scrub the repo of these endings using the .gitattributes file as discussed here. However, inevitably files slip through the cracks, or we have no control over how a Submodule repo is managed. So, these notes are a reminder for how to deal with this problem and make periodic corrections.

For a local directory on a Linux machine, making changes to several files at once can be done with perl. CD into the directory and do the following to edit the files "in place" (danger: this will overwrite your original files, so only use the "i" option if you are sure your command is correct).

perl -pi -e 's/^M//g' ./*.fds

The easiest way I have found to find and correct this problem in bulk for the whole repository is the following. First, at the top level (directory) of the repo run the following bash shell command, which will give you a list of files with CRLF or ^M line endings. (Note: you may want to do a git clean -dxf at the top level of your repo first.)

for i in $(grep -IURl --color --exclude="*.pdf" --exclude-dir=".git" "^M" .); do echo $i; done

Refer to the grep documentation (man grep) for a description of the options. Note that the ^M must be entered via ctrl-v ctrl-m, otherwise the command will not work as intended. At this point you have not made any changes to the files; you have only listed them. Depending on the specific repo and file types you need to consider, you may need to amend the --exclude commands. You can add patterns like this, for example, --exclude="{*.pdf,*.exe}". It may be easier to use --include. For example, to only check line endings of .csv files do --include="*.csv".

Once you have a list of files, double check that the command worked correctly. Open one of the files with vim and make sure you see ^M at the end of the lines.

$ vi -b filename

If so, you may manually remove the ^M with the following command to make sure it works on your system (on macOS do $ brew install dos2unix).

$ dos2unix -o filename

Double check your file to make sure the ^M are now gone. If so, you can move on to a bulk removal as follows. Basically, you are just going to add your sed command into the bash do loop, like so:

for i in $(grep -IURl --color --exclude="*.pdf" --exclude-dir=".git" "^M" .); do echo $i; dos2unix -o $i; done

I emphasize again, make sure ^M is input via ctrl-v ctrl-m.