Danish Researchers Develop New Way to Build AI for More Languages

Today’s leading AI models rely heavily on English-language data, making it harder for less popular languages to keep pace.

Researchers in Denmark have developed a new AI architecture that could help countries and organizations build more capable AI systems in their own languages – without sharing sensitive data. FlexMoRE lets organizations train specialized modules on private data separately and then combines them into a single model without sharing the underlying data.

In this email Q&A, Peter Schneider-Kamp, a professor at the University of Southern Denmark and one of the researchers behind the project, discusses how the technology could help preserve local languages and culture while expanding access to AI.

The AI Innovator: What is FlexMoRE and why is it significant – not just for Danish institutions but as a new way of building AI models?

Peter Schneider-Kamp: FlexMoRE is an AI system that allows AI researchers to combine independently trained experts more efficiently, making it possible to run powerful Mixture-of-Experts (MoE) models such as FlexOlmo faster and on smaller GPUs (e.g., consumer GPUs with 16 GB instead of data center GPUs with 40 GB+ GPUs).

For MoEs such as FlexOlmo, each additional expert adds another 4 billion parameters to the size of the model. With FlexMoRE, this can be as low as 1.3 million parameters (or 300 times smaller), depending on the expert’s actual needs. In our experiments, the most powerful FlexMoRE model with seven experts uses just over 10 billion parameters in total and outperforms the MoE with full experts and 31 billion parameters across benchmarks.

What capabilities can a country- or industry-specific model provide that a general-purpose model cannot?

A country- or industry-specific model can be trained on data that is relevant to the country or industry but not publicly shareable or available for general-purpose model training. This can range from archives of national libraries to quality assurance documentation in the pharmaceutical industry.

Sharing data is harder than sharing experts. The interesting aspect here, which holds true for both FlexOlmo and FlexMoRE, is that experts trained independently on such data can be joined together in one model, where they are able to collaborate with each other.

As an example, an expert trained on Danish, another trained on Finnish, and one trained on pharmaceutical documentation might be able to successfully understand and translate pharmaceutical documents from Danish to Finnish.

What innovation in FlexMoRE excites you the most?

I am most excited about how FlexMoRE provides us with the opportunity to integrate many more independently trained experts into one model.

A FlexMoRE model could easily have 50 experts but run on the same hardware as a 7-expert FlexOlmo model. This allows for the idea of an app store-like ‘Expert Store,’ where language- and domain-specific experts can be made available and combined depending on needs into models that are just large enough to have the right set of skills.

Imagine a box of LEGO bricks – very Danish, by the way – that can be built into different shapes.

The underlying architecture of FlexMoRE can be applied anywhere. Do you envision a future where every country develops its own foundation models? Why or why not?

Yes and no. I do not envision that every country and industry trains their own foundation model. That would be wasteful.

I envision that each country and industry train one or more experts that can contribute particular domain knowledge or language understanding to a FlexMoRE model when integrated as an expert. Training such experts is comparatively cheap and technically not very demanding.

How would FlexMoRE fit into the broader movement toward sovereign AI?

FlexMoRE and an ‘Expert Store’ would provide the ability to build models that satisfy the demands of different sovereign AI use cases based on publicly available shared experts. The entities using the FlexMoRE models may judge the bias, legal concerns, value and other characteristics of each expert individually and include only those that they deem relevant and fit for inclusion.