Publication details
UTFpatgen: Pattern generation without borders
| Authors | |
|---|---|
| Year of publication | 2026 |
| Type | Peer-reviewed scientific article |
| Magazine / Source | TUGboat: The Communications of the TeX Users Group |
| MU Faculty or unit | |
| Citation | |
| web | Journal paper page with fulltext and metadata |
| Doi | https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen |
| Keywords | UTFpatgen; hyphenation; dictionary problem; effectiveness; hyphenation patterns; Patgen; pattern generation; benchmark dataset; hyphenated wordlists; supervised learning; hyphenation benchmark |
| Description | Originally developed to power TeX’s hyphenation engine, hyphenation patterns generated by Frank Liang’s PatGen algorithm have transcended their original typesetting boundaries to become the universal backbone for automated text hyphenation across typesetting engines, web browsers, DTP software, and digital readers. Building upon such widespread and impactful deployment, UTFpatgen, a new addition to the TeXLive program suite, modernizes this foundational technology by removing legacy 8-bit limitations. It enables the seamless generation of compact, highly accurate hyphenation patterns for UTF-8 text---from complex scripts with large grapheme inventories (such as polytonic Greek and Tamil) to intricate conjunct characters and multilingual pattern sets. UTFpatgen has been programmed as a drop-in replacement for PatGen. Although PatGen was designed 44 years ago and the interface shows its age, we felt it was preferable not to disturb the many existing workflows built around it. Written from scratch in the CWEB literate programming system, UTFpatgen replaces fixed-capacity static arrays with dynamic memory allocation to eliminate constraints on dictionary sizes and hyphenation levels. Furthermore, by introducing a variable-length byte-indexing schema, it provides fully scalable support for UTF-8-encoded input alphabets while maintaining strict backward compatibility at the interface level. We present empirical results from stress testing UTFpatgen against the hyph-bench evaluation benchmark, offering a direct comparison of the time and memory efficiency of both programs. Ultimately, UTFpatgen provides the TeX ecosystem with a flexible, modern alternative capable of supporting complex script workflows, structured pattern sets for multiple languages and dialects, and new pattern uses such as ligature suppression, weighted hyphenation, and other uses of generated hierarchical patterns, without borders such as limitations on data structure sizes, allocation, and the number of generated levels of patterns. |
| Related projects: |