Skip to content
dahliaPublic

About

A Rust/JavaScript library and CLI tool that converts mixed-script Korean text into hangul-only text

Topics

Resources

Contributing

Stars

28 stars

Watchers

1 watching

Forks

Latest commit

 

History

504 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Gukhanmun

crates.io JSR npm PyPI GitHub Actions License: GPL-3.0-only GitHub Sponsors Open Collective Funded by FOSS for All Microgrants 2026

Also available: 韓國語 (Korean).

Gukhanmun converts Korean text written in mixed script (國漢文混用體, hanja characters interleaved with hangul) into hangul-only text. It is the successor to Seonbi, narrowed to the hanja-to-hangul conversion pipeline and extended along several axes: streaming I/O, pluggable dictionaries, lattice-based segmentation, and a wider range of output formats. The library is implemented in Rust and exposed as a Rust library, a command-line tool, a WebAssembly package, a native Node-API addon, and a Python package.

Features

  • Lattice segmentation finds the best split rather than greedily taking the longest match. 行事場所 segments as 行事 + 場所, not 行事場 + 所.
  • Pluggable dictionaries: in-memory map, mmap-friendly FST files, or CDB files, composable via ChainDictionary.
  • The Standard Korean Language Dictionary (標準國語大辭典) and Open Korean Dictionary (우리말샘) ship as compiled FST/CDB data. Rust bundles them; JavaScript and Python provide separate, opt-in data packages.
  • Format adapters for plain text, HTML fragments, and Markdown. The engine is format-neutral; adapters handle parsing and serialization.
  • Six rendering modes: hangul-only, hangul(hanja) parentheses, hanja(hangul) parentheses, ruby on hangul, ruby on hanja, and original mixed script with selective glossing.
  • The engine supports incremental conversion. Context-dependent annotation may buffer more input; the JavaScript and Python chunked APIs buffer the whole document until finalization.
  • Initial sound law (頭音法則) for South Korean orthography, applied to fallback readings. Dictionary entries encode the correct reading already.
  • The core crate (gukhanmun-core) is no_std + alloc, suitable for embedded targets.
  • JavaScript and TypeScript bindings ship in two flavours: a WebAssembly package that runs in browsers, Deno, Node.js, Bun, and edge runtimes, and a native Node-API addon for higher server-side throughput.
  • Python bindings provide typed, immutable configuration and reusable converters. Native conversion releases the global interpreter lock (GIL), and a configured converter can be shared across threads.

Installation

Command-line tool

Via mise

If you use mise, install a prebuilt binary with a single command:

mise use -g aqua:dahlia/gukhanmun

The -g flag installs it globally. Omit it to activate the tool only in the current project directory.

Via Windows Package Manager (winget)

On Windows, install via winget:

winget install HongMinhee.Gukhanmun

From crates.io

If you have a Rust toolchain installed, install from crates.io:

cargo install gukhanmun-cli gukhanmun-mkdict

This compiles the binaries and places them in ~/.cargo/bin/. Make sure that directory is on your PATH.

Via npm or Python tooling

With Node.js 22 or later, install both commands from npm:

npm install -g @gukhanmun/cli@latest

With Python 3.11 or later, use one isolated tool installer:

pipx install gukhanmun-cli
# Or:
uv tool install gukhanmun-cli

These packages include both gukhanmun and gukhanmun-mkdict, with the converter's dictionaries built in. They are separate from the JavaScript and Python libraries; neither requires Rust. See the CLI installation guide for platform support, development builds and shell completions.

Prebuilt binaries

Prebuilt binaries for Linux (x86_64, aarch64), macOS (x86_64, aarch64), and Windows (x86_64, ARM64) are attached to each release on GitHub:

https://github.com/dahlia/gukhanmun/releases

Download the archive for your platform, extract it, and place the gukhanmun and gukhanmun-mkdict executables somewhere on your PATH.

Rust library

Add to Cargo.toml:

cargo add gukhanmun-core

Optionally add format adapters and dictionary backends:

cargo add gukhanmun-html gukhanmun-markdown
cargo add gukhanmun-stdict gukhanmun-opendict
cargo add gukhanmun-fst gukhanmun-cdb

JavaScript/TypeScript library

Install the WebAssembly package for most JavaScript environments:

npm  add       @gukhanmun/wasm @gukhanmun/stdict-fst
pnpm add       @gukhanmun/wasm @gukhanmun/stdict-fst
yarn add       @gukhanmun/wasm @gukhanmun/stdict-fst
bun  add       @gukhanmun/wasm @gukhanmun/stdict-fst
deno add --jsr @gukhanmun/wasm @gukhanmun/stdict-fst

If you need better server-side performance and don't mind a native dependency, install the Node-API package instead:

npm  add     @gukhanmun/napi     @gukhanmun/stdict-fst
pnpm add     @gukhanmun/napi     @gukhanmun/stdict-fst
yarn add     @gukhanmun/napi     @gukhanmun/stdict-fst
bun  add     @gukhanmun/napi     @gukhanmun/stdict-fst
deno add npm:@gukhanmun/napi jsr:@gukhanmun/stdict-fst

Python library

Requires Python 3.11 or later. Use pip in your virtual environment, or add the package to a uv-managed project:

python -m pip install gukhanmun
# Or, in a uv project:
uv add gukhanmun

The base package has no required third-party Python dependencies and includes no official dictionary data. To install the optional Standard Korean Language Dictionary, use python -m pip install 'gukhanmun[stdict]' or uv add 'gukhanmun[stdict]', then explicitly add stdict() from gukhanmun.dictionaries to a converter. Installing an extra does not activate it automatically.

See the Python installation guide for supported wheels, source builds, and prerelease installation, and the Python quick start for conversion examples.

Quick start

echo "漢字 北京 標識" | gukhanmun
# → 한자 베이징 표지

echo "漢字" | gukhanmun --rendering hangul-hanja-parens
# → 한자(漢字)

echo "<p>漢字</p>" | gukhanmun --format text/html
# → <p>한자</p>
use gukhanmun_core::{MapDictionary, RenderMode, convert_plain_text};

let mut dict = MapDictionary::new();
dict.insert("漢字", "한자");

let output = convert_plain_text("漢字", &dict, RenderMode::HangulOnly);
assert_eq!(output, "한자");

The Python example supplies its own reading and needs no optional data:

from gukhanmun import Converter, DictionaryEntry, InputFormat, MapDictionarySource

converter = Converter(dictionaries=[
    MapDictionarySource(entries=[DictionaryEntry(hanja="漢字", reading="한자")]),
])

assert converter.convert("漢字를 한글로") == "한자를 한글로"
assert converter.convert("<p>漢字</p>", format=InputFormat.HTML) == "<p>한자</p>"

For the full guide, including HTML/Markdown adapters, rendering modes, presets, and the JavaScript and Python APIs, visit https://gukhanmun.org/.

Crates

The project is a Cargo workspace. All crates share the same version.

Crate Description
gukhanmun-core Format-neutral IR, engine, dictionary trait, lattice segmenter, fallback phoneticizer. no_std + alloc.
gukhanmun-html HTML fragment reader and writer; HtmlScopeData with lang inheritance and preserved-tag handling.
gukhanmun-markdown Markdown adapter over pulldown-cmark; inline HTML is re-scanned for lang attributes.
gukhanmun-fst FST-backed HanjaDictionary implementation for mmap-friendly on-disk dictionaries.
gukhanmun-cdb CDB-trie HanjaDictionary implementation; trivially auditable on-disk format.
gukhanmun-stdict The bundled Standard Korean Language Dictionary as an embedded FST byte array.
gukhanmun-opendict The bundled Open Korean Dictionary (우리말샘) data.
gukhanmun-loanwords NIKL loanword spelling examples with explicit reading choices.
gukhanmun-dict-extract Shared dictionary dump extraction helpers.
gukhanmun-mkdict CLI tool to build FST or CDB dictionary files from TSV, CSV, or JSON Lines input.
gukhanmun-cli The gukhanmun command-line binary.

npm/JSR packages

The project publishes nine JavaScript library and data packages, all sharing the same version as the Rust crates. The separate npm-only @gukhanmun/cli package installs the command-line tools, not a JavaScript conversion API.

Package JSR npm Description
@gukhanmun/types JSR npm TypeScript type declarations shared by the WASM and NAPI packages. No runtime code.
@gukhanmun/wasm JSR npm WebAssembly build. Runs in browsers, Deno, Node.js, and Bun.
@gukhanmun/napi npm Native Node.js addon via napi-rs. Faster than WASM for server-side use.
@gukhanmun/stdict-fst JSR npm Bundled Standard Korean Language Dictionary in FST format.
@gukhanmun/stdict-cdb JSR npm Bundled Standard Korean Language Dictionary in CDB format.
@gukhanmun/opendict-fst JSR npm Bundled Open Korean Dictionary categories in FST format.
@gukhanmun/opendict-cdb JSR npm Bundled Open Korean Dictionary categories in CDB format.
@gukhanmun/loanwords-fst JSR npm Bundled NIKL loanword spelling examples in FST format.
@gukhanmun/loanwords-cdb JSR npm Bundled NIKL loanword spelling examples in CDB format.

PyPI packages

All eight Python distributions share one version derived from the workspace, using PEP 440 spelling. The standalone gukhanmun-cli distribution is independent of the Python library. The six optional data distributions have platform-independent wheels and no dependency on the native package.

Distribution Description
gukhanmun Native converter, typed Python API, and optional dictionary loaders.
gukhanmun-cli Standalone converter and dictionary builder with bundled dictionaries.
gukhanmun-stdict-fst Standard Korean Language Dictionary data, FST format.
gukhanmun-stdict-cdb Standard Korean Language Dictionary data, CDB format.
gukhanmun-opendict-fst Four Open Korean Dictionary categories, FST format.
gukhanmun-opendict-cdb Four Open Korean Dictionary categories, CDB format.
gukhanmun-loanwords-fst NIKL loanword spelling examples, FST format.
gukhanmun-loanwords-cdb NIKL loanword spelling examples, CDB format.

Dictionary data retains its own license and attribution requirements, separate from the library's GPL license. See Official dictionaries and the LICENSE and ATTRIBUTION.md files in each data distribution.

Design documentation

DESIGN.md covers the full architecture: intermediate representation, lattice segmentation algorithm, dictionary trait design, middleware system, and format adapter internals.

Support the project

You can support Gukhanmun's ongoing development and maintenance through GitHub Sponsors or Open Collective.

License

Distributed under GPL 3.0. See LICENSE.

About

A Rust/JavaScript library and CLI tool that converts mixed-script Korean text into hangul-only text

Topics

Resources

Contributing

Stars

28 stars

Watchers

1 watching

Forks

Releases

Sponsor this project

Contributors

Languages