Bug: Many serious TOML parsing/loading bugs caused by underlying toml library, consider switching?

[pirate]

Thanks so much for building python-benedict, it's really awesome!

So unfortunatley the underlying TOML library https://github.com/uiri/toml used by benedict has a bunch of long-time outstanding bugs that break parsing/loading and generally make it unsafe to load->dump->load the same string.

For example, you cannot dump any dict containing escape sequences without it throwing an exception:

>>> benedict({
	"reset": "\033[00;00m",
	"lightblue": "\033[01;30m",
}).to_toml()

...
File ~/test/.venv/lib/python3.11/site-packages/toml/encoder.py:113, in _dump_str(v)
    111     else:
    112         joiner = "u00"
--> 113     v = [v[0] + joiner + v[1]] + v[2:]
    114 return unicode('"' + v[0] + '"')

IndexError: list index out of range

But thats not it, there are many other fairly major string escaping, quoting, and parse/dump cycle inconsistency bugs that have bitten other projects using uiri/toml:

• more and more backslashes when repeatedly dump and load uiri/toml#441
• String with escaping char dump error uiri/toml#437
• Default TomlEncoder does not escape backslashes correctly uiri/toml#415
• Multiple-line strings don't get escaped when a TOML format included uiri/toml#387
• bug: comma in list's value is not parsed properly uiri/toml#322
• Backslash roundtrip problem uiri/toml#404
• Nested quotes inside of the multi-line strings bugs uiri/toml#230
• Incorrect handling of backslashes in dict keys uiri/toml#280
• Incorrect string escaping in TOML encoder  uiri/toml#201
• mixed types in array don't work uiri/toml#270
• Bug: Comma in string in list raises IndexError: list index out of range  uiri/toml#170
• https://github.com/uiri/toml/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen+escape

Almost all of these are 2yr+ old, indicating they're probably not going to all get fixed anytime soon without significant increase in velocity. No harm no foul, it's open source we can't demand they go faster and they don't owe us anything, but maybe benedict could consider a different library with fewer major outstanding parser consistency issues?

Is benedict open to switching to a library without these issues? Maybe one of these:

• https://github.com/hukkin/tomli
• https://github.com/python-poetry/tomlkit
• https://github.com/hukkin/tomli-w

I'll chip in $20 towards the work to make the switch if it's an option ⬇️

[pirate]

After looking more into the other options unfortunately they are worse in other ways, for example they can barely handle a fraction of the types that uiri/toml supports :/

I guess we are stuck writing a custom TomlEncoder class to work around the bugs for now.

import re
from pathlib import Path

import toml

def better_toml_dump_str(val):
    try:
        return toml.encoder._dump_str(val)
    except Exception:
        # if we hit any of toml's numerous encoding bugs,
        # fall back to using json representation of string
        return json.dumps(str(val), default=repr)

class CustomTOMLEncoder(toml.encoder.TomlEncoder):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        
        # override the dumper funcs for any types you need to support here:
        self.dump_funcs[str] 			= better_toml_dump_str
		self.dump_funcs[Path] 			= better_toml_dump_str
        self.dump_funcs[re.RegexFlag] 	= better_toml_dump_str
        ...

# this works:
>>> test_dict = {
	"string_with_escape_sequences": "\033[00;00m",
	"some_Path": "/some/path/example.txt",
	"example_weird_type": re.IGNORECASE | re.UNICODE | re.MULTILINE,
	# etc. try more weird types here
}
>>> print(benedict(test_dict).to_toml(encoder=CustomTOMLEncoder()))

string_with_escape_sequences = "\u001b[00;00m"
example_weird_type = "re.IGNORECASE|re.UNICODE|re.MULTILINE"

You're welcome to close this issue 🤷, up to you.

[fabiocaccamo]

@pirate thank you for reporting this problem.

Actually, for decoding toml on Python >= 3.11, the tomlib standard lib is used:
https://github.com/fabiocaccamo/python-benedict/blob/main/benedict/serializers/toml.py#L10

I think it would be cool to improve TOML support, would you like to submit a PR with a list of failing test cases (that should succeed with the perfect library)?
So it would be much easier to test other TOML libraries or doing fixes in a different way.

[fabiocaccamo]

@pirate is it in your plans to submit a PR for this?

[pirate]

I think given there are only two real python toml library options and you're already using the best one, I think it's not worth trying to patch the toml bugs on top within benedict.

You can leave it up to end users to figure out a workaround if they really need one (e.g. my CustomTOMLEncoder above). I just wanted to document this issue so people could find it via Google if they have the same problem and wondered the cause.

Here is a good test suite for TOML parsing though if you want one: https://github.com/toml-lang/toml-test

[fabiocaccamo]

@pirate totally agree, it would make more sense to fix these issue at toml lib level.