Inefficient Regular Expression Complexity

Summary

A Regular Expression Denial of Service (ReDoS) vulnerability was identified in the huggingface/transformers library, specifically in the file "tokenization_gpt_neox_japanese.py" of the GPT-NeoX-Japanese model. The vulnerability occurs in the "SubWordJapaneseTokenizer" class, where regular expressions process specially crafted inputs. The issue stems from a regex exhibiting exponential complexity under certain conditions, leading to excessive backtracking. This can result in high CPU usage and potential application downtime, creating a Denial of Service (DoS) scenario. The issue affects transformers versions prior to 4.50.0.