Use `\r\n` in data/whirlwind.warc

[sebastian-nagel]

The WARC file data/whirlwind.warc uses '\n' instead of '\r\n' to separate WARC and HTTP headers. This can make parsing the WARC file fail:

$> java -jar jwarc.jar ls data/whirlwind.warc
Exception in thread "main" java.io.UncheckedIOException: org.netpreserve.jwarc.ParsingException: invalid WARC record at position 8: WARC/1.0<-- HERE -->\nWARC-Type: warcinfo\nWARC-Date: 2024-05-... (offset 0 in whirlwind.warc)
        at org.netpreserve.jwarc.WarcReader$1.hasNext(WarcReader.java:357)
        at org.netpreserve.jwarc.tools.ListTool.main(ListTool.java:12)
        at org.netpreserve.jwarc.tools.WarcTool.main(WarcTool.java:40)
Caused by: org.netpreserve.jwarc.ParsingException: invalid WARC record at position 8: WARC/1.0<-- HERE -->\nWARC-Type: warcinfo\nWARC-Date: 2024-05-... (offset 0 in whirlwind.warc)
        at org.netpreserve.jwarc.WarcParser.parse(WarcParser.java:356)
        at org.netpreserve.jwarc.WarcReader.next(WarcReader.java:181)
        at org.netpreserve.jwarc.WarcReader$1.hasNext(WarcReader.java:355)
        ... 2 more

Same for:

• data/whirlwind.warc.wat
• data/whirlwind.warc.wet

[lfoppiano]

@sebastian-nagel I've been trying to fix this without much success.
I've fixed the .editorconfig but any attempt in touching those files by the IDE or github will result in reverting back to \n.

I've edited the files manually with VIM, but still at the end I have a problem floating around the request record:

(base) ➜  whirlwind-java git:(bugfix/correct-line-ending) ✗ java -jar jwarc.jar ls data/whirlwind.warc
         0 warcinfo   -    -
Exception in thread "main" java.io.UncheckedIOException: org.netpreserve.jwarc.ParsingException: invalid WARC record at position 4: ...ns/warc-format/warc-1.1/\n\nWARC/1.0\r\nWARC<-- HERE -->-Type: request\r\nWARC-Date: 2024-05-18T01... (offset 747 in whirlwind.warc)
        at org.netpreserve.jwarc.WarcReader$1.hasNext(WarcReader.java:357)
        at org.netpreserve.jwarc.tools.ListTool.main(ListTool.java:12)
        at org.netpreserve.jwarc.tools.WarcTool.main(WarcTool.java:40)
Caused by: org.netpreserve.jwarc.ParsingException: invalid WARC record at position 4: ...ns/warc-format/warc-1.1/\n\nWARC/1.0\r\nWARC<-- HERE -->-Type: request\r\nWARC-Date: 2024-05-18T01... (offset 747 in whirlwind.warc)
        at org.netpreserve.jwarc.WarcParser.parse(WarcParser.java:356)
        at org.netpreserve.jwarc.WarcReader.next(WarcReader.java:181)
        at org.netpreserve.jwarc.WarcReader$1.hasNext(WarcReader.java:355)
        ... 2 more
(base) ➜  whirlwind-java git:(bugfix/correct-line-ending) ✗ 

The file is here and when you see it there the \r\n have disappeared: https://github.com/commoncrawl/whirlwind-java/blob/bugfix/correct-line-ending/data/whirlwind.warc

Any suggestion?

[sebastian-nagel]

The line endings can be adjusted by change the git attributes, see

• https://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/
• https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes

Not an easy task once it's wrong.

For verifying the attributes:

$whirlwind-java> git ls-files --eol | grep -E '\.warc(\.w[ae]t)?$'
i/lf    w/lf    attr/                   data/whirlwind.warc
i/lf    w/lf    attr/                   data/whirlwind.warc.wat
i/lf    w/lf    attr/                   data/whirlwind.warc.wet

$whirlwind-python> git ls-files --eol | grep -E '\.warc(\.w[ae]t)?$'
i/mixed w/mixed attr/                   whirlwind.warc
i/crlf  w/crlf  attr/                   whirlwind.warc.wat
i/mixed w/mixed attr/                   whirlwind.warc.wet

Instead of using an editor - why not copying the files from whirlwind-python?

[lfoppiano]

@sebastian-nagel thanks for the help today, this should be fixed in #15.