Memory footprint and startup time

[ssvb]

Hello! As an attempt to investigate why the plugin for Firefox has a high memory usage, I made the following git branch for testing purposes: https://github.com/ssvb/harper/commits/heaptrack/ with two commits. One commit introduces the use of the heaptrack tool, which is run in a github CI job on a simple harper-cli lint invocation. And another commit just erases almost all words from the English dictionary. Here are the results with the intact English dictionary:

   [...]
   Compiling harper-cli v0.1.0 (/home/runner/work/harper/harper/harper-cli)
   Compiling harper-ls v0.71.0 (/home/runner/work/harper/harper/harper-ls)
   Compiling harper-wasm v0.1.0 (/home/runner/work/harper/harper/harper-wasm)
    Finished `release` profile [optimized + debuginfo] target(s) in 2m 10s
heaptrack stats:
	allocations:          	2535728
	leaked allocations:   	43210
	temporary allocations:	894059
Peak heap usage: 130.37M
/home/runner/.config/harper-ls/dictionary.txt: No such file or directory (os error 2)
/home/runner/.local/share/harper-ls/file_dictionaries/test.txt%: No such file or directory (os error 2)
Warning: could not detect language ID; falling back to PlainEnglish parser.
Advice: 
   ╭─[test.txt:1:1]
   │
 1 │ Ths is a test.
   │ ─┬─  
   │  ╰─── Did you mean to spell `Ths` this way?
───╯


real	0m0.545s
user	0m0.493s
sys	0m0.053s

And this is after erasing all words from the English dictionary (except for "this", "is", "a", "test"):

   [...]
   Compiling harper-wasm v0.1.0 (/home/runner/work/harper/harper/harper-wasm)
   Compiling harper-cli v0.1.0 (/home/runner/work/harper/harper/harper-cli)
   Compiling harper-ls v0.71.0 (/home/runner/work/harper/harper/harper-ls)
    Finished `release` profile [optimized + debuginfo] target(s) in 2m 10s
heaptrack stats:
	allocations:          	1422507
	leaked allocations:   	38950
	temporary allocations:	351464
Peak heap usage: 7.89M
/home/runner/.config/harper-ls/dictionary.txt: No such file or directory (os error 2)
/home/runner/.local/share/harper-ls/file_dictionaries/test.txt%: No such file or directory (os error 2)
Warning: could not detect language ID; falling back to PlainEnglish parser.
Advice: 
   ╭─[test.txt:1:1]
   │
 1 │ Ths is a test.
   │ ─┬─  
   │  ╰─── Did you mean to spell `Ths` this way?
───╯


real	0m0.143s
user	0m0.133s
sys	0m0.011s

It looks like the lion's share of the memory footprint is contributed by the dictionary. It's reported as ~130MB for harper-cli (and ~170MB for the Firefox plugin) with a full dictionary, and merely ~8MB when the dictionary is stripped. Additionally, roughly ~400 milliseconds are added to the Harper's startup time on a rather powerful github CI machine to initialize the dictionary.

These numbers look very suspicious to me. Because the FST data structure used for the dictionary is supposed to be much more compact: #138

[elijah-potter]

Thanks for the investigation! I doubt it's the FST that's soaking up all of the memory. If anything, I would imagine it's the word map in the MutableDictionary, the trie in the TrieDictionary, or both. Since these are layered upon each other, there ends up being a good amount of duplication. Feel free to take a look at the layering. Maybe there is a simple solution I'm not seeing?

[braklo]

Hi @ssvb, a late follow-up in case Belarusian is still on your mind. The memory side has moved since this discussion: #3725 (large dictionaries materialized 2–3× in FstDictionary) was closed after #4030 removed a redundant copy, and #4071 tracks the remaining second copy in FstDictionary::curated(). We hit the same wall with Slovak, which is also heavily inflected and built from hunspell stems and affixes, much like your #2195.

On the language side, #3402 adds a module system where each non-English language sits behind its own cargo feature. It already has German, Portuguese, Polish and Slovak (the Slovak module is ours), so Belarusian could come back as one more module there rather than as a standalone branch. If you pick it up again, happy to compare notes on the hunspell conversion (https://github.com/sk-spell/harper-sk).