Add nfc nfd comparison scripts for opus100#863
Open
klei22 wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds tooling under data/opus-100/ to generate NFC/NFD dataset splits, convert Hangul NFC→NFD (Hangul-only), run tokenization comparisons, and inspect resulting vocabularies; also updates the KoCommercial dataset downloader to use a single Parquet shard URL.
Changes:
- Add Hangul-only NFC→NFD conversion script and dataset split helper for Opus100.
- Add scripts for running NFC vs NFD tokenization/vocab comparisons (including a Textual-based TUI for
meta.pklinspection). - Simplify KoCommercial Parquet shard list from 8 URLs to 1 URL.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| data/opus-100/prepare_nfc_and_nfd_dataset_splits.sh | Creates 90/10 splits and generates NFD versions for comparison. |
| data/opus-100/hangul_nfc_to_nfd.py | Implements Hangul-only NFC→NFD conversion with file/stdin modes. |
| data/opus-100/compare_nfc_nfd.sh | Automates running prepare.py across NFC/NFD and vocab sizes. |
| data/opus-100/compare_meta_vocab_tui.py | Adds a Textual TUI to compare vocab entries from two meta.pkl files. |
| data/ko_commercial/get_dataset.py | Updates KoCommercial download list to a single Parquet shard. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+18
| #!/bin/bash | ||
|
|
||
| FILE="$1" | ||
|
|
||
| NFC_90="p90_nfc.txt" | ||
| NFC_10="p10_nfc.txt" | ||
|
|
||
| NFD_90="p90_nfd.txt" | ||
| NFD_10="p10_nfd.txt" | ||
|
|
||
| TOTAL=$(wc -l < "$FILE") | ||
| PCT=$((TOTAL * 10 / 100)) | ||
| head -n -$PCT "$FILE" > "$NFC_90" | ||
| tail -n $PCT "$FILE" > "$NFC_10" | ||
|
|
||
| python3 hangul_nfc_to_nfd.py "$NFC_90" "$NFD_90" | ||
| python3 hangul_nfc_to_nfd.py "$NFC_10" "$NFD_10" | ||
|
|
Comment on lines
+1
to
+12
| #!/bin/bash | ||
|
|
||
| # bash prepare_nfc_and_nfd_dataset_splits.sh input.txt | ||
|
|
||
| for tokenization in "char_bpe"; do | ||
| for type in "nfc" "nfd"; do | ||
| for vocab_size in "2000" "3000" "4000" "5000"; do | ||
| python3 prepare.py -t p90_"$type".txt -v p10_"$type".txt --method "$tokenization" --vocab_size "$vocab_size" -s -S "$type"_"$vocab_size" -T | ||
| mv char_bpe_vocab.json "${tokenization}_${type}_${vocab_size}/" | ||
| done | ||
| done | ||
| done |
Comment on lines
+71
to
+74
| try: | ||
| with os.fdopen(fd, "wb") as tmp: | ||
| tmp.write(converted) | ||
| os.replace(tmp_name, path) |
Comment on lines
+2
to
+6
| """compare_meta_vocab_tui.py | ||
|
|
||
| Textual TUI to compare vocabularies stored in two meta.pkl files. | ||
| Supports sorting both vocab lists by byte length or frequency. | ||
| """ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.