This thesis addresses the challenge of accurately tokenizing Japanese sentences by improving a sole trader’s Python-based sentence-splitter system to better handle the splitting of Japanese sentences. Japanese poses unique difficulties during tokenization due to its agglutinative nature and due to the fact that spaces are not used to delimit words. We evaluate the baseline system, having a token lexicon of some 120 000 entries, against a Short Unit Word gold standard consisting of a sample of 10 000 sentences from Tatoeba, splitted using fugashi and the UniDic token lexicon. Precision, recall and F1-scores are calculated and an F1-score of 0.6537 is established for the baseline system. We enhance the token lexicon by (1) adding root verb and root adjective stems, (2) adding five additional verb stems that are used in UniDic and (3) incorporating 5 050 high-frequency tokens missing from the original lexicon. These additions raise the F1-score to 0.7235, demonstrating that while token lexicon expansion can yield some improvements in F1-score it is ultimately insufficient to reach the industry standard (96-99 %) without either pruning non-UniDic entries from the customer’s pre-existing token lexicon or by altering the phrase matching algorithm in the customer’s sentence-splitter system (which is currently a Minimum Forward Matching method). It is concluded that, while a base of Short Unit Words in the customer’s token lexicon is ultimately necessary, a move to a Long Unit Word standard may allow the customer to reach an industry standard, or somewhere close to it, while preserving the pre-existing data in the customer’s token lexicon.