Wrong indented lines cause bugs for a long time

[KohakuBlueleaf]

This issue have been mentioned before but since no one have replied it so I open a new one.

https://github.com/TimDettmers/bitsandbytes/blob/a06a0f6a08cb23754b110359a109e069fa97ce9e/bitsandbytes/functional.py#L337-L342

#227
#262

The if-block around functional.py:218 in 2f2063b may have been accidentally indented. tests.test_functional.test_few_bit_quant doesn't cover the inside of the if block, so maybe this was missed (or it was intentional and it'd be great to have confirmation).

This bug(or, feature) left over 6 versions (0.36~0.41) and cause lot of problems:
ShivamShrirao/diffusers#178
#152
kohya-ss/sd-scripts#523

really need fixes or explanation for this

Sorry for @
@TimDettmers

[sdbds]

Too many report about it:
diff with bitsandbytes over 0.35

bmaltais/kohya_ss#1291
bmaltais/kohya_ss#1252
bmaltais/kohya_ss#775

@TimDettmers
plz look here

[2kpr]

I did a few hours of comparison tests today with 0.35.0 and 0.41.0, full DreamBooth, batch size of 4, 400, 800, and 1200 steps.

I altered 0.41.0 in the tests by removing the indents in the code in question above.

The results were that removing the indents in 0.41.0 did fix the 'blotchyness' and that hard overtrained look basically, BUT, when testing with the exact same prompts vs 0.35.0 it's obvious that there is something else wrong, other than the 'indented code' above.

For instance in the 800/1200 steps cases the 0.41.0 is producing images very similar to what is shown here: ShivamShrirao/diffusers#230
Basically the people are heavily 'elongated' and the inferences/generations are NOT following the prompts properly at all, almost like it's been heavily overtrained but also heavily distorted.

On the other hand 0.35.0 at 800, 1200 steps, with the exact same prompts, the inferences/generations are perfectly fine, and they are completely following the prompts with all the added detail added, etc.

So, I just wanted to mention the results, I realize it might be extremely hard to isolate the problem, but basically at this point, and pretty much since the end of 2022, most all the repos for training Stable Diffusion (fine-tuning, dreambooth, lora) have been locking their bitsandbytes down to version 0.35.0 for these very reasons.

It would be great to be able to play with the newer optimizers in the versions above 0.35.0, but doing so just happens at the cost of constant bad inferences/generations unfortunately.

[KohakuBlueleaf]

I did a few hours of comparison tests today with 0.35.0 and 0.41.0, full DreamBooth, batch size of 4, 400, 800, and 1200 steps.

I altered 0.41.0 in the tests by removing the indents in the code in question above.

The results were that removing the indents in 0.41.0 did fix the 'blotchyness' and that hard overtrained look basically completely, BUT, when testing with the exact same prompts vs 0.35.0 it's obvious that there is something else wrong.

For instance in the 800/1200 steps cases the 0.41.0 is producing images very similar to what is shown here: ShivamShrirao/diffusers#230 Basically the people are heavily 'elongated' and the inferences/generations are NOT following the prompts properly at all, almost like it's been heavily overtrained but also heavily distorted.

On the other hand 0.35.0 at 800, 1200 steps, with the exact same prompts, the inferences/generations are perfectly fine, and they are completely following the prompts with all the added detail added, etc.

So, I just wanted to mention the results, I realize it might be extremely hard to isolate the problem, but basically at this point, and pretty much since the end of 2022, most all the repos for training have been locking their bitsandbytes down to version 0.35.0 for these very reasons.

It would be great to be able to play with the newer optimizers in the versions above 0.35.0, but doing so just happens at the cost of constant bad inferences/generations unfortunately.

I think 0.35~0.41 may change some calculation things and it may change some property of the optimizer

But the most important thing is the indented block definitely should not in the loop(

For other difference, we may need more investigation on what bnb have changed

Just like what I said, we don't know if it is "features"
We need explanation

[2kpr]

I think 0.35~0.41 may change some calculation things and it may change some property of the optimizer

But the most important thing is the indented block definitely should not in the loop(

For other difference, we may need more investigation on what bnb have changed

Just like what I said, we don't know if it is "features" We need explanation

Completely agreed! :)

[rationalism]

@KohakuBlueleaf @2kpr This should be fixed by 0.41.1

[wkpark]

indentation fixed by 3c9aca9

commit 3c9aca9124ab8bcd160a8c90bba0d6ca361c141f
Author: Tim Dettmers <tim.dettmers@gmail.com>
Date:   Thu Aug 3 19:47:15 2023 -0700

    Fixed two bugs in dynamic data type creation.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.