SubMiner/vendor/hachidori/third_party/hoshidicts

hoshidicts

This library implements a dictionary backend that works similarly to Yomitan. This was made for Hoshi Reader and was only tested with Japanese. Other languages might need their own deinflector or adjustments to the lookup strategy.

A MIT version of the library is available on the main-mit branch.

Reference

importer

ImportResult dictionary_importer::import(const std::string& source_path, const std::string& output_dir, bool low_ram = false)

Imports a Yomitan .zip dictionary file or an MDict .mdx dictionary into a custom format. The resulting folder is stored in output_dir/<dict_title>. Glossaries are compressed using zstd. Term, frequency and pitch dictionaries are generally supported, but only a small part of the pitch accent spec was implemented. Setting low_ram to true can reduce memory usage significantly at the cost of slightly lower import speed.

The format is detected from the file contents, not the extension.

MDX / MDD import

An .mdx file is imported directly, without an intermediate Yomitan archive: its entries are converted to Yomitan term banks of 10,000 rows as they are read, so memory use is bounded by the bank size, not the dictionary. The conversion follows manabitan's MDX importer so the result renders the same way.

• Container: MDict engine versions 1.x and 2.0 (GeneratedByEngineVersion). Version 3 files are rejected with a clear error.
• Encodings: UTF-8 and UTF-16. GBK, GB18030 and Big5 dictionaries are rejected (unsupported MDX encoding: <name>).
• Compression: none, LZO1X and zlib, per block, with every block's Adler-32 verified.
• Encryption: Encrypted="2" (ciphered key index) is supported. Encrypted="1" (registration-protected record blocks) needs a user key and is rejected.
• Resource files: X.mdd, X.1.mdd, X.2.mdd, ... next to X.mdx are read automatically (file name case does not matter); a missing MDD is not an error. Only assets the glossaries or stylesheets refer to are imported, under mdict-media/<path>; every *.css in the MDD plus inline <style> blocks become the dictionary's stylesheet. Keys containing .., a drive letter or NUL are dropped.
• Entries: @@@LINK=target redirects become extra headwords of the target (one hop; a redirect to a missing target is dropped). Duplicate headwords stay separate entries. Format="Text" definitions become plain string glossaries; HTML definitions become structured content.
• HTML fidelity: the MDX StyleSheet backtick markup is expanded; b/i/em/strong/u/s/sub/sup/h1-6/p/pre/font/... map to styled span/div; inline style keeps the properties Yomitan's structured content supports; entry://, bword://, d:, x: links search the term; sound:// links are disabled (rendered as #); javascript: and friends are neutralised; <script> is dropped; unsupported elements keep their text; nesting deeper than 20 is flattened. CSS from the MDD is passed through as the dictionary stylesheet, so selectors that depend on tags Yomitan does not render (<b>, <p>, ...) will not match.
• The key index of the MDX (and of each MDD) is held in memory during the import; records are streamed block by block.

hoshidicts-cli import path/to/dictionary.mdx

query

void DictionaryQuery::add_term_dict(const std::string& path)

Adds an imported term dictionary to the query.

void DictionaryQuery::add_freq_dict(const std::string& path)

Adds an imported frequency dictionary to the query.

void DictionaryQuery::add_pitch_dict(const std::string& path)

Adds an imported pitch dictionary to the query.

std::vector<TermResult> DictionaryQuery::query(const std::string& expression) const

Queries all added dictionaries for the given expression. TermResult includes glossary, frequency and pitch data in the order dictionaries were added. Glossaries are decompressed.

std::vector<DictionaryStyle> DictionaryQuery::get_styles() const

Returns CSS styles for all dictionaries, if present.

std::vector<char> DictionaryQuery::get_media_file(const std::string& dict_name, const std::string& media_path) const

Returns raw bytes for file originally stored at media_path in term dictionary dict_name or an empty vector if the file does not exist.

deinflector

std::vector<DeinflectionResult> Deinflector::deinflect(const std::string& text) const

Deinflects a given Japanese string using rules from the Yomitan deinflector. As this doesn't use any dictionary data, the result may include invalid deinflections.

static uint32_t Deinflector::pos_to_conditions(const std::vector<std::string>& part_of_speech)

Converts a vector of part-of-speech tags into a bitmask used for deinflection filtering.

lookup

Lookup::Lookup(DictionaryQuery& query, Deinflector& deinflector)

Creates a Lookup object using a given query with dictionaries added and a deinflector.

std::vector<LookupResult> Lookup::lookup(const std::string& lookup_string, int max_results = 16, size_t scan_length = 16) const

Follows a parsing strategy similar to Yomitan. Substrings of lookup_string are tested from length scan_length down to 1. Each substring is preprocessed, deinflected then queried using the query object.

Keys longer than scan_length are still found when scan_length is at least 8 and lookup_string is long enough to contain them: the importer records every key longer than 16 code points by its first eight code points in scan.idx, and when the input begins like such a key the scan extends to that key's length plus eight code points for an inflected ending. Inputs that do not begin like a long key keep the cost of scan_length. DictionaryQuery::max_long_key_length() returns the longest such key across the loaded term dictionaries so a caller can size lookup_string. Dictionaries imported before scan.idx existed simply never extend.

Results are filtered by part-of-speech tags defined in dictionaries, or added directly if none are present. The results are sorted by matched length first, then by preprocessing steps, then deinflection trace length and finally by frequency.

std::vector<LookupResult> Lookup::lookup_dictionary(const std::string& lookup_string,
                                                    const std::string& dictionary_path,
                                                    int max_results = 16,
                                                    size_t scan_length = 16) const

Runs the same lookup and ranking pipeline while restricting term matches to one already-added dictionary. Frequency and pitch metadata still come from every added metadata dictionary.

Acknowledgements

• Yomitan: Dictionary format, Japanese deinflection rules and descriptions, Japanese preprocessor | GPL-3.0
• glaze: MIT
• libdeflate: MIT
• xxHash: BSD-2-Clause
• zstd: BSD
• utfcpp: BSL-1.0
• unordered_dense: MIT
• utf8proc: MIT
• kanji-processor: MIT
• lzokay: MIT (vendored in external/lzokay)
• gumbo-parser (Nokogiri's fork of Google's gumbo): Apache-2.0, hashmap.c MIT (vendored in external/gumbo-parser)

License

hoshidicts (main) is licensed under the GNU General Public License v3.0. See LICENSE for details.