Researchers at Meta AI may have developed a way to get around the “tokenization” problem with GPT models.
News Join us on social networks
Meta AI recently published pre-print research showing off a radical new “Megabyte” framework for building generative pre-trained transformer (GPT) systems.
Dubbed “promising” by OpenAI’s Andrej Karpathy, former director of artificial intelligence at Tesla, the new architecture is designed to process large volumes of data — such as images, novels and video files — without the use of a process known as tokenization.
Promising. Everyone should hope that we can throw away tokenization in LLMs. Doing so naively creates (byte-level) sequences that are too long, so the devil is in the details.
Trending: Huh? Chuck Schumer Slams ‘MAGA Supreme Court’ After A Unanimous Decision
Tokenization means that LLMs are not actually fully end-to-end. There is a
Join the conversation!
Please share your thoughts about this article below. We value your opinions, and would love to see you add to the discussion!