Reimplement DB2ToSqlite in Go#1510
Draft
hillerstorm wants to merge 10 commits into
Draft
Conversation
… writer) First stage of docs/db2tool-migration-plan.md: everything downstream of file extraction, fed by pre-extracted .db2 files. - dbd/: BSD-3 port of DBDefsLib's DBDReader + the SQLite helpers' exact-build version selection (last block whose builds contain the trailing build number; fail-loud when WoWDBDefs lacks the build). - wdc/: MIT port of DBCD.IO's WDC5Reader/WDC4Row/BitReader — multi-section, all 6 compression modes, sparse offset-map, copy tables, trailing relations, negative-base string offsets, and the keyless encrypted-section skip (TactKeyLookup != 0 + zero-filled data). - sqlite/: port of this repo's SQLiteDbCreator/SqliteDataInserter via modernc.org/sqlite (delete-first lifecycle, generated [Col_i] VIRTUAL columns, table-less idx_ names in settings order, relation 0 stays 0, float32 marshaling, C# Array-declared JSON serialization semantics). - Temporary driver (main.go) decodes tools/DB2ToSqlite/dbfilesclient against DBDCache for an explicit --build; a follow-up commit replaces this with local CASC extraction. - internal/golden + tests: schema DDL gate, per-table row parity, the float-notation assertion from plan §5.5, modernc marshaling smoke test. Validation vs the Jul 11 dotnet reference (build 5.5.4.68571, with hotfixes): sqlite_master schema byte-identical (236 objects); row dumps byte-identical for 67/72 tables; the 9 differing lines are exactly the DBCache hotfix overlay (add item 272920/spell 1291315 'Spring Panda', delete item 277947, modify spell 1298412) plus the documented CurvePoint float-notation case (slack table). gen_db against the Go-built DB reproduces the committed assets/database/db.json and leftover_db.json byte-for-byte (.bin files are nondeterministic even from the reference DB).
The new tact/ package reads the full local install — a deliberate behavior change from the CDN-fed .NET tool, proven byte-equivalent (see docs/db2tool-migration-plan.md): - .build.info parse + entry-by-Product + trailing build number - build/CDN config from Data/config (vfs-* lines parsed, never consulted) - local .idx v7 buckets (XOR bucket select, packed archive/offset bits, 30-byte storage frame skip), data.NNN via os.ReadAt - encoding EN table (paged BE binary search, 40-bit sizes) - TSFM root (post-10.1.7 dfVersion 1/2, delta-encoded FDIDs, enUS) - BLTE N/Z; keyless 'E' chunks stay zero-filled (plan §7 C1/M5) - FDID resolution: static path map (80 configured paths) + listfile fallback - listfile.csv + .dbd HTTP fetch/cache (Last-Modified / 24h-mtime rules), with offline fallback to existing copies - main.go: local mode derives the build from .build.info; --build keeps the offline pre-extracted mode; settings-relative paths resolve against tools/db2tool (M4, incl. the raw-TargetDirectory FDID-key carve-out) - makefile: make db / make ptrdb now run the Go tool; the three gen_db listfile.csv literals repoint to tools/db2tool/listfile.csv (§9.2) Pre-flight + gates (live install, wow_classic 5.5.4.68571): - one-line BaseDir patch on the .NET tool with its 1.3GB CDN cache moved aside re-extracted everything byte-identically from local CASC - Go tool, local mode: all 72 .db2 and 8 basestats .txt byte-identical to the CDN-fed originals; wowsims.db vs the fresh no-hotfix dotnet reference: schema identical, 2 residual diff lines (known CurvePoint float-notation case, slack table) - full make db pipeline with no dotnet reproduces the committed assets/database/db.json and leftover_db.json byte-for-byte Reference captures (gitignored tools/db2tool/refs/): wowsims.nohotfix.db, wowsims.hotfix.db, DBCache.68571.bin. Current live hotfix overlay is 7 rows (Spring Panda add, Item 277947 delete, Spell 1298412 description) — the input for the hotfix-overlay commit.
Ports DBCD.IO's HotfixReader/HTFXReader (XFTH v9) plus wow.tools.local's HotfixManager cache scanning and SStrHash to wdc/hotfix.go. Local-CASC runs now scan tools/db2tool/caches/*.bin and <BaseDir>/**/DBCache.bin and apply the exact-build cache's records before insertion: stable ascending-PushId order, add/replace whole rows decoded from the byte-aligned blobs (DBD field metadata, non-inline id from RecordId, non-inline relations read at their DBD-declared MetaDataFieldType), deletes remove rows, Combine dedup on the full record identity including data bytes. --dbcache <file> pins specific caches for deterministic runs; --no-hotfixes disables the overlay; the offline --build mode stays hotfix-free by default. Gates (build 68571, refs/DBCache.68571.bin snapshot): - with hotfixes vs refs/wowsims.hotfix.db: 2 diff lines total across all 72 tables — the documented CurvePoint Id=236585 float-notation divergence - --no-hotfixes vs refs/wowsims.nohotfix.db: same 2 lines only - overlay delta: +Item/ItemSparse 272920, +Spell/SpellName 1291315, -Item 277947, Spell 1298412 AuraDescription_lang modified - go build/vet/gofmt clean; TestHotfixGoldenParity added (skips without refs/), existing golden/wdc/dbd/sqlite tests pass; SStrHash(upper(name)) verified == WDC5 header TableHash for all 72 tables
make db / make ptrdb have run the pure-Go extractor since the previous commits; the dotnet tool, its vendored DLLs (TACTSharp, DBCD, DBCD.IO, DBDefsLib), the CDN cache, and the solution file are no longer needed. - delete tools/DB2ToSqlite/ (csproj, Program.cs, Helpers/, references/*.dll, appsettings, launch configs) and wowsims-mop.sln (it contained only this project) - the gitignored artifact dirs (dbfilesclient/, DBDCache/, listfile.csv) already live under tools/db2tool/; add tools/db2tool/caches/ (optional hotfix-cache scan dir) to .gitignore - docs/commands.md: make db now needs a local WoW install (Settings.BaseDir) instead of dotnet 9
The effect generator wrote raw template output, so the gofmt'ed committed files picked up whitespace churn on every regeneration. Render through go/format.Source instead. Entry sorts also only compared the first variant's item ID, which ties for items with multiple effects (e.g. Blessing of Zuldazar 138925/138967) and left the order to map iteration — tie-break on SpellID so the tooltip comments stop swapping between runs. enchants_auto_gen.go picks up a one-time formatting diff (it was committed unformatted); all three generated files now reproduce byte-identically.
The binary protobuf encoding is nondeterministic (map ordering), so a no-op regeneration previously churned the .bin files on every run. The JSON output is deterministic and covers all UIDatabase fields, so compare the freshly built JSON against the file on disk and skip writing both files when identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Also ports any c# dependency to go, including proper license text in
tools/db2tool/NOTICES.mdand/or relevant files.