Regression: All formatting options in COPY commands require `format.` prefix, but did not in DataFusion 36.0.0

[alamb]

Describe the bug

#9604 from @metesynnada rationalized the COPY syntax to match CREATE EXTERNAL TABLE syntax which is great. However, now all options require a replicated format. string:

COPY source_table
to 'test_files/scratch/copy/table_csv_with_options'
-(format csv,
-'csv.has_header' false,
-'csv.compression' uncompressed,
-'csv.datetime_format' '%FT%H:%M:%S.%9f',
-'csv.delimiter' ';',
-'csv.null_value' 'NULLVAL');
+'format.has_header' false,
+'format.compression' uncompressed,
+'format.datetime_format' '%FT%H:%M:%S.%9f',
+'format.delimiter' ';',
+'format.null_value' 'NULLVAL');

In datafusion 36.0.0 the prefix was not needed

DataFusion CLI v36.0.0
❯ COPY (values (1)) TO 'test_files/scratch/copy/table/' (format parquet, 'compression' 'zstd(10)');

It would be nice to avoid requiring the repetition of format. string

To Reproduce

❯ COPY (select * from (values (1))) to '/tmp/foo.parquet' STORED AS PARQUET OPTIONS ('format.compression' snappy);
+-------+
| count |
+-------+
| 1     |
+-------+
1 row in set. Query took 0.037 seconds.

❯ COPY (select * from (values (1))) to '/tmp/foo.parquet' STORED AS PARQUET OPTIONS ('compression' snappy);
Invalid or Unsupported Configuration: could not find config namespace for key "compression"

Expected behavior

Both queries should do the same thing

Additional context

See discussion synnada-ai#10 (comment)

[devinjdangelo]

@tinfoil-knight I believe the PR you had open but was closed without merging to fix a similar issue should be able to fix this (perhaps with slight modification).

[tinfoil-knight]

take