update sqlite float nan support

[elcritch]

No description provided.

[Araq]

It's a bad idea all around: NaN is a particularly terrible value as it defies basic logic (NaN != NaN, no other Null/None/nil type in history of mankind does that bullshit).

In general Ormin needs to get more precise at NULL Handling, the users should be able to opt-into Option[T] for these Nullable entries. This will avoid the special case for float and better all around.

[elcritch]

It's a bad idea all around: NaN is a particularly terrible value as it defies basic logic (NaN != NaN, no other Null/None/nil type in history of mankind does that bullshit).

What?! In SQL null != null. From Postgres manual:

Do not write expression = NULL because NULL is not "equal to" NULL. (The null value represents an unknown value, and it is not known whether two unknown values are equal.) This behavior conforms to the SQL standard.

Really, NaN != NaN is just basic logic. E.g. implying that if Creature A is not human and Creature B is not human and then saying "well Creature A and Creature B are the same or different" are both logical fallacies. You just don't know either way.

In terms of a null in CS and as a sentinel empty value, then sure it's bullshit. That's not how SQL or (most) Floats treat it for better or worse.

In general Ormin needs to get more precise at NULL Handling, the users should be able to opt-into Option[T] for these Nullable entries. This will avoid the special case for float and better all around.

Sure, however SQLite is weird and doesn't handle float NaN's it seems.

If you write a float NaN using Ormin to SQLite you currently then get 0.0 back. As a user I care less about philosophy but that I can store my value and get the same value back.

That's why this is an opt in compile flag. SQLite can't store NaN it seems so the user should be able to decide the policy.

Otherwise I have to write different code for Postgres and SQLite even with better Option handling.

[elcritch]

An even more general solution could be allowing users to provide custom to/from hooks for SQL similar to json utils. Not sure how to do that with Ormin. Maybe there's a clean syntax for that.

[Araq]

Really, NaN != NaN is just basic logic.

What? No. Do you also want x != nil not to work? It's stupid. Also Inf == Inf, so it's entirely inconsistent with its own logic.

[elcritch]

What? No. Do you also want x != nil not to work? It's stupid.

I mean sure its nice if x != nil works, but we even need to use isNil(x) for some types in Nim, so using isNull(x) is fine.

And there's corner cases where NaN != NaN can be useful when doing maths or SQL stuff.

Also Inf == Inf, so it's entirely inconsistent with its own logic.

Well agreed. Still doesn't help me serialize NaN from floats in either Postgres and SQLite.

If a temperature sensor didn't record anything it should be NaN, not 0.0. That's huge PITA to filter from data when calculating averages, anomalies, etc.

[Araq]

If a temperature sensor didn't record anything it should be NaN, not 0.0.

Nö, NaN is so broken that it should report Option[float] instead where there is a sane None value. It's like float's NaN and SQL's NULL was invented by people who knew nothing about type theory... Or programming for that matter.

[elcritch]

If a temperature sensor didn't record anything it should be NaN, not 0.0.

Nö, NaN is so broken that it should report Option[float] instead where there is a sane None value. It's like float's NaN and SQL's NULL was invented by people who knew nothing about type theory... Or programming for that matter.

Haha, nah mate. The people who invented IEEE 754 NaN and SQL's NULL knew type theory quite well I'd wager. You just don't have to normally deal with the problems NaN != NaN is useful and even required. Programming a game engine is different than programming a physics algorithm used to simulate say nukes.

Looks like William Kahan was the lead for IEEE 754 standard and deep in the innards of numerical analysis while either of us were in diapers. :P Seems he knows his type/set/number theories well enough:

The Davis–Kahan–Weinberger dilation theorem is one of the landmark results in the dilation theory of Hilbert space operators and has found applications in many different areas.[5]

It wasn't an accident they chose what they did. NaN behavior isn't broken, it's just a best of the worst ways to represent some mathematical / numerical properties in floats that are hard to express. Floats are inherently mathematically incorrect, but they're practical. if-its-not-a-number-what-is-it-demystifying-nan-for-the-working-programmer expresses it well:

Even experienced programmers might think its existence a mistake or its behavior buggy. The truth is more complicated. Within contexts where floating-point numbers are the "best tool for the job", NaN is a terrible hack, but it is also the best option we have found. In those contexts, we are not likely to get rid of NaN any time soon.

The skeptics do have a point though, because NaN is a hack. The compromises that created modern floating-point math are very real and often still relevant, but don't always apply. Despite that, many modern programming languages default to treating all non-integer arithmetic as floating-point arithmetic. This can introduce many strange bugs and surprising behaviors to unprepared programmers. NaN is hardly the only example of this (the signed infinities are also oddly behaved), but it is certainly the most spectacular one.

Especially this bit:

It is tempting to suggest that the non-numeric values 1/0 and √-1 are the same non-number because they are both "empty": there are no real numbers that answer the implied question. This does not work because there are extensions to the real numbers where those questions have answers. For 1/0, you can use the extended real line to define it as positive infinity. The question "What is the square root of -1?" has two answers in the complex plane (i and -i). If you find it useful to use one or both of these extensions, the two different non-numbers are not equal to each other. In any case, neither of these values are equal to 0/0, which, again, is technically the set of all real numbers.

[Araq]

Claiming that the false value for == implies a "don't know" is broken behavior and broken type theory, no matter who came up with it. And you even said it yourself, paraphrasing "just use isNaN", so yeah now that == is broken we need a new predicate that is not broken.

The citations argue for +/-Inf and NaN, they don't argue for "comparisons with NaN should always be special snowflakes".

Here is how broken it really is: You cannot even sort a sequence of floats reliably, the NaNs can end up anywhere...

[elcritch]

Claiming that the false value for == implies a "don't know" is broken behavior and broken type theory, no matter who came up with it.

Why is deciding that false value for == implies a “don’t know” broken? You have to choose either true or false if you want to define an == predicate that doesn't raise a fault.

Choosing false isn’t any more broken than choosing true for it, and has some properties that are practical/useful in certain areas. Just because you don’t find those cases useful, doesn’t make their choice illogical just not optimal for your preferences. Hence most DBs let you change the behavior.

I think it’s more that programming languages suck and don’t provide a complete set of infix predicates for all the logic cases for floats.

IEEE 754 also defined predicates for ==, =>, etc that do raise faults right? Why doesn’t Nim use those and throw a defect like it does for overflows. That’s much less broken logically.

The citations argue for +/-Inf and NaN, they don't argue for "comparisons with NaN should always be special snowflakes".

Yes they do. That last paragraph I quoted talks about NaN equals and makes one case for it. It also mentions that signed Infs also have weird behavior for other predicates.

In SQL there’s queries where Null <> Null being false is helpful as well and would be difficult or impossible to do otherwise.

Here is how broken it really is: You cannot even sort a sequence of floats reliably, the NaNs can end up anywhere...

And? There’s no inherent mathematical way to sort NaNs. That’s asking “is not-a-number greater than 3?” — there’s literally no proper answer.

But for practical purposes folks define some cmp behavior that moves NaNs to the top or bottom because people still want to sort floats with NaNs anyway.