line-ending detection doesn't work on Windows and --help explain how to use --line-ending

[idbrii]

When passing a file with Unix-style LF line endings to isort, it always converts it to CRLF on Windows.

C:\> isort --version

                 _                 _
                (_) ___  ___  _ __| |_
                | |/ _/ / _ \/ '__  _/
                | |\__ \/\_\/| |  | |_
                |_|\___/\___/\_/   \_/

      isort your imports, so you don't have to.

                    VERSION 7.0.0


C:\> cat C:\scratch\aletest.py | file -
/dev/stdin: Python script, ASCII text executable, with very long lines (965)

C:\> isort - < C:\scratch\aletest.py | file -                   
/dev/stdin: Python script, ASCII text executable, with CRLF line terminators

C:\> file --version
file-5.45
magic file from /usr/share/misc/magic

Same results with a minimal file created from scratch:

C:\> echo import re > testfile.py
C:\> echo import os >> testfile.py
C:\> dos2unix testfile.py
C:\> file testfile.py
testfile.py: ASCII text
C:\> isort - < C:\scratch\testfile.py | file -
/dev/stdin: ASCII text, with CRLF line terminators

On Linux (WSL Ubuntu using bash) it works:

$ isort - < /mnt/c/scratch/aletest.py | file -
/dev/stdin: Python script, ASCII text executable
$ isort --version | grep VERSION
                    VERSION 5.13.2

(I originally encountered the issue on v5 under Windows.)

So auto line ending detection is not working. So I tried to force it to use \n line endings. The docs say:

Line Ending

Forces line endings to the specified value. If not set, values will be guessed per-file.

Type: String
Default:
Config default:
Python & Config File Name: line_ending
CLI Flags:

--le
--line-ending

However, what values are valid? CRLF, \n, unix? There are many ways line endings are described, so it's not entirely clear which ones isort expects.

I think the correct value is the literal character at the end of the line: \n. But how do you specify that? I tried what seemed obvious but it didn't work and always uses CRLF:

C:\> isort --line-ending \n - < C:\scratch\testfile.py 
import os\nimport re\n
C:\> isort --line-ending \\n - < C:\scratch\testfile.py
import os\\nimport re\\n
C:\> isort --line-ending "\n" - < C:\scratch\testfile.py
import os\nimport re\n
C:\> isort --line-ending "\\n" - < C:\scratch\testfile.py
import os\\nimport re\\n

So all of those options seem wrong.

On Win11 in cmd.exe. I'm using file from my git install (via scoop). I got isort via pip on Python 3.11.3.

[DanielNoord]

I believe the detection is defined here:

isort/isort/parse.py

Lines 31 to 36 in 024aa37

	def _infer_line_separator(contents: str) -> str:
	if "\r\n" in contents:
	return "\r\n"
	if "\r" in contents:
	return "\r"
	return "\n"

Could you run this code on your own code?

The config is parsed with argparse. With --show-config you should be able to see the parsed values.

[idbrii]

It looks like the autodetection issue may be due to how isort handles reading from stdin:

C:\> dos2unix testfile.py & isort - < testfile.py | file -
dos2unix: converting file testfile.py to Unix format...
/dev/stdin: ASCII text, with CRLF line terminators

C:\> dos2unix testfile.py & isort - < testfile.py > outfile.py &  file outfile.py
dos2unix: converting file testfile.py to Unix format...
outfile.py: ASCII text, with CRLF line terminators

C:\> dos2unix testfile.py & isort testfile.py & file testfile.py   
dos2unix: converting file testfile.py to Unix format...
testfile.py: ASCII text

(I'm using stdin because that's how vim-ale uses isort.)

---

Using _infer_line_separator in a variety of ways always returns \n for a LF-formatted file. On a CRLF file, it returns \r\n except for when reading from stdin:

# isorttest.py
# Usage: python isorttest.py < c:/scratch/testfile.py

import sys


def print_line_endings(data):
    """Print all line endings we find in the input."""
    print("print_line_endings:")
    prev = None

    for cur in data:
        if cur == ord("\n"):
            if prev == ord("\r"):
                print("CRLF")
            else:
                print("LF")
        elif cur == ord("\r"):
            # Don't print yet; need to see next byte
            pass
        else:
            if prev == ord("\r"):
                print("CR")

        prev = cur

    # Handle file ending with a lone '\r'
    if prev == ord("\r"):
        print("CR")


def _infer_line_separator(contents: str) -> str:
    if "\r\n" in contents:
        return "\r\n"
    if "\r" in contents:
        return "\r"
    return "\n"


line_endings = {
    "\r\n": "CRLF",
    "\r": "CR",
    "\n": "LF",
}


def report_ending(label, file):
    ending = _infer_line_separator(file)
    print(label, line_endings[ending])
    print_line_endings(file.encode("utf-8"))


report_ending("stdin:", sys.stdin.read())

with open("testfile.py", "r", newline="") as f:
    report_ending("open as text:", f.read())

with open("testfile.py", "rb") as f:
    report_ending("open as binary:", f.read().decode("utf-8"))

print("read as binary")
with open("testfile.py", "rb") as f:
    print_line_endings(f.read())

Output:

C:\> dos2unix testfile.py & python isorttest.py < c:/scratch/testfile.py
dos2unix: converting file testfile.py to Unix format...
stdin: LF
print_line_endings:
LF
LF
open as text: LF
print_line_endings:
LF
LF
open as binary: LF
print_line_endings:
LF
LF
read as binary
print_line_endings:
LF
LF

C:\> unix2dos testfile.py & python isorttest.py < c:/scratch/testfile.py
unix2dos: converting file testfile.py to DOS format...
stdin: LF
print_line_endings:
LF
LF
open as text: CRLF
print_line_endings:
CRLF
CRLF
open as binary: CRLF
print_line_endings:
CRLF
CRLF
read as binary
print_line_endings:
CRLF
CRLF

Perhaps there's the universal newline feature is kicking in for stdin. (I used newline="" to disable it.) Regardless, that's the opposite problem I'm having: for isort, LF is seen as CRLF on stdin.

But I don't see sys.stdin in isort, so I'm not sure how it handles stdin.

[DanielNoord]

isort/isort/main.py

Line 1131 in 024aa37

input_stream = sys.stdin if stdin is None else stdin

is what you are looking for I think

[dayananda-ks]

Hi! I've submitted a PR for this issue — #2543
Please take a look when you get a chance 😊