Tokenization with Split Trees
Published • May 26, 2026
Authors:,,
Craig W. Schmidt
Michael Krumdick
Adam Wiemerslage
Abstract
We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily splits each pretoken into a full binary tree using precomputed byte n-gram counts, independent of any vocabulary. Given a vo...
Subject
Finding related papers...
Discussions
(0)No comments yet
Be the first to share your thoughts!