Unlocking Language Barriers: The Rise of a Slovene Generative Model with 1 Billion Parameters

In the domain of Natural Language Processing (NLP), it’s often said that large language models (LLMs) hold the key to the future. But what if that future is mostly accessible in English? Until recently, many languages, including Slovene, have been left in the dust due to the lack of suitable models. Enter GaMS 1B, a ground-breaking generative model tailored specifically for Slovene. Let’s unwind the complexity of this technological breakthrough and why it might just be the game-changer that less-resourced languages need.

Why Does Slovene Need Its Own Model?

Imagine trying to learn a dance routine with shoes two sizes too big. Sure, you can make do, but you’ll never be as graceful as you could be in a perfect fit. That’s what Slovene has been experiencing in the world of generative AI. English models? Everywhere. Slovene-focused models? Hardly any, until now.

Why is this a big deal? Because models like ChatGPT, Llama, and many others, mainly trained on English data, merely skim the surface when it comes to less-resourced languages. Even existing multilingual models rarely include enough Slovene data to be considered effective.

Making GaMS 1B: From English to Slovene

Creating a model like GaMS 1B involves more than just hitting the ‘translate’ button. The team behind this Slovene breakthrough adapted an English model, specifically the OPT model, to speak Slovene fluently. Given that starting from scratch with 15 trillion tokens is unthinkable due to data constraints, using an English model as a springboard made perfect sense.

The Language Boost: More Than Just Slovene

It’s not just Slovene fueling this model. Bro-side languages like Croatian, Bosnian, and Serbian also provided valuable data due to their structural similarities with Slovene. Think of it as a multilingual smorgasbord that amplifies the model’s capabilities.

The Tokenizer: Slovene’s New Linguistic Chip

A massive hurdle was tokenization—breaking down text into understandable parts, like chopping a vegetable before throwing it into the stew. Existing models used inefficient tokenization for Slovene, leading to cumbersome deciphering. So, a new, Slovene-friendly tokenizer was born, accommodating not just Slovene, but Croatian and English too!

And did it work? Absolutely! By re-jigging the whole process using nifty methods named WECHSEL and FOCUS, the team catapulted these language-specific tokens right into the model’s core.

Evaluating Success: From Challenges to Triumphs

Testing an LLM isn’t straightforward—especially in less-resourced languages where benchmarks are rarer than sightings of the Loch Ness monster. The GaMS 1B was put through paces with Slovene-centric tasks, from classification to sentence simplification. While it did face some bumps in the road when compared to finely-tuned Slovene BERT models in classification tasks, it held its own in generative tasks such as sentence simplification, rivaling even the GPT-3.5-Turbo.

What’s the Catch?

While GaMS 1B isn’t yet instruction-tuned—meaning it’s akin to a raw diamond waiting to be polished—it shows immense promise. The model sometimes got its wires crossed in understanding what kind of output was needed and suffered from a lack of task understanding in some situations, which likely could be cleared with further instruction-tuning.

Future Horizons: More Than Just Slovene

What does the future hold? The plan is to polish the GaMS model with an instruction-following dataset and fine-tune it for broader applications. There’s even talk of building an even bigger model in the pipeline, which would put many of these learning lessons into the high gear of execution.

Key Takeaways

1. GaMS 1B is a trailblazer: It’s the first open-source Slovene-specific generative LLM, making headway in less-resourced language processing.

2. Tokenization tackled: A superior, multilingual tokenizer is central to its success, especially in handling diverse languages like Slovene and Croatian.

3. Multilingual input: Incorporating texts from similar languages helps override the data scarcity issue, optimizing the model’s effectiveness.

4. Mixed achievements: While GaMS 1B excels in generative tasks, it needs more fine-tuning to perfect its classification skills.

5. Bright future: Upcoming instruction tuning might propel GaMS to new heights, offering robust multilingual LLM solutions.

In the evolving landscape of AI, GaMS 1B heralds a future where language isn’t a barrier but a bridge, enriching Slovene’s place in global AI developments and setting a benchmark for other less-resourced languages to rise to the challenge. And who knows? Maybe your next favorite AI-generated content could first see the light of day in Slovene!

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Generative Model for Less-Resourced Language with 1 billion parameters” by Authors: Domen Vreš, Martin Božič, Aljaž Potočnik, Tomaž Martinčič, Marko Robnik-Šikonja. You can find the original article here.