Why Asking the Same Question 100 Times Might Be the Smartest Way to Build AI

What a simple experiment with an open-source model running on one of Europe’s largest supercomputers teaches us about the future of enterprise AI.

Most people assume that if you ask an AI model the same question twice, you should receive the same answer.

You don’t.

And that is precisely why businesses need to rethink how they evaluate generative AI.

Recently, I ran another simple experiment. The prompt was intentionally trivial:

“Describe Lisbon, the capital of Portugal, in 3 adjectives.”

Nothing more.

The prompt was executed 100 consecutive times using Kimi, one of the most powerful open-source language models currently available, hosted on the MareNostrum 5 supercomputer at the Barcelona Supercomputing Center. The infrastructure was configured with vLLM across four computing nodes, using 16 NVIDIA GPUs for distributed inference. The execution environment automatically deployed the model, exposed it via an API, and executed the entire experiment in a controlled manner.
The question never changed.

The answers did.

Sometimes Lisbon was luminous. Other times it became nostalgic, soulful, enchanting, undulating or historic. The overall meaning remained remarkably consistent, yet the wording varied naturally across executions.

For a human reader, this looks perfectly normal.

For AI engineering, it is one of the most important characteristics of modern language models.

AI Is Probabilistic, Not Deterministic

Traditional software behaves like a calculator.

Give it the same input, and you expect exactly the same output every time.

Large Language Models don’t work like that.

Instead of retrieving fixed answers, they generate the next token based on probability distributions learned from enormous amounts of text. Every generated word slightly changes the probabilities of the words that follow.

Generation is therefore a statistical process.

This is not a flaw.

It is the very reason these models can write, reason, explain, brainstorm and adapt to different contexts.

The consequence is that evaluating AI requires a completely different mindset from evaluating traditional software.

One answer tells you almost nothing.

One hundred answers begin to reveal the system's behaviour.

Why Volume Matters

One of the biggest misconceptions in enterprise AI is believing that testing a prompt once is enough.

It isn’t.

If a company wants to automate customer service, contract analysis, medical documentation, compliance reports or financial workflows, it isn’t enough to know that the prompt worked once.

It needs to work consistently.

Running the same prompt dozens—or even hundreds—of times allows engineers to measure something far more valuable than accuracy.

It allows them to measure stability.

Questions such as these become possible:

• Does the model always understand the task?

• Does the structure remain consistent?

• Does creativity stay within acceptable limits?

• Are there unexpected failures?

• How much variability is acceptable?

These questions matter far more than obtaining one impressive answer during a product demonstration.

Enterprise AI is about reliability, not magic.

What 100 Answers Tell Us About Lisbon

The beauty of this experiment is that it wasn’t really about Lisbon.

It was about measuring a model.

Across 100 independent generations, the model produced almost 300 adjectives (a handful of responses contained only two adjectives due to formatting differences), but only about 15 unique descriptive concepts emerged. That immediately tells us something important: the model is creative, but not random. It repeatedly converges on a relatively small semantic space.

Several interesting observations emerge.

First, “luminous” dominates the distribution, appearing in roughly 86% of all executions. That makes sense. Lisbon is internationally recognised for its exceptional natural light, a quality deeply represented throughout travel literature, photography, journalism and online content.

Second, “hilly” appears in nearly three-quarters of all responses. Unlike "beautiful" or "nice," it is a physical characteristic. The model has learned a factual geographic property rather than simply generating flattering adjectives.

Third, the third adjective becomes far more diverse.

Sometimes Lisbon is soulful.

Sometimes historic.

Sometimes vibrant.

Sometimes enchanting.

The model is essentially sampling from a family of highly compatible concepts rather than selecting a single “correct” answer.

That is exactly what a probabilistic language model should do.

In other words, the variability exists mostly where humans would also disagree.

Few people would argue whether Lisbon is hilly.

Many people would disagree on whether its defining emotional quality is historic, soulful or vibrant.

The model mirrors that uncertainty remarkably well.

The Hidden Role of Temperature

One of the reasons for this behaviour is a parameter called temperature.

Temperature controls how adventurous a language model becomes when selecting its next word.

At very low temperatures (close to zero), the model becomes highly deterministic. It repeatedly chooses the highest-probability token, often producing nearly identical responses.

Increase the temperature, and the model begins exploring alternative words that remain plausible but are slightly less probable.

That is why the experiment keeps returning variations within the same semantic neighbourhood rather than generating completely unrelated descriptions.

If the temperature were much higher, Lisbon might suddenly become mysterious, bohemian, romantic or even chaotic. Those descriptions are not necessarily wrong—they are simply less statistically dominant within the model’s learned representation.

Finding the right temperature, therefore, becomes an engineering decision rather than an artistic one.

For creative writing, higher variability may be desirable.

For legal documents or financial reports, businesses usually want lower variability and greater consistency.

Looking at AI Like a Statistician

Perhaps the most important lesson is methodological.

Most AI evaluations ask:

“Was this answer good?”

A better question is:

“What does the distribution of 100 answers look like?”

That small change completely transforms how we evaluate AI.

Instead of judging one output, we can calculate:

• the frequency of each concept;

• semantic convergence;

• lexical diversity;

• consistency between executions;

• outlier responses;

• confidence intervals;

• prompt stability.

This is much closer to how engineers validate aircraft components or pharmaceutical trials than how consumers typically use ChatGPT.

As frontier models become increasingly capable, businesses will need to think less like prompt writers and more like statisticians.

The future of enterprise AI is unlikely to be built on a single brilliant prompt.

It will be built on thousands of prompts, measured, compared, validated, and continuously improved until the system's behaviour becomes predictable enough to trust.

The Rise of Trillion-Parameter Models

The experiment used Kimi, an open-source frontier model belonging to a new generation of systems approaching the trillion-parameter scale through sophisticated architectures such as Mixture-of-Experts.

Although not every parameter is active for every request, these models contain an enormous amount of learned knowledge distributed across specialised expert networks.

The result is impressive.

They can write code.

Analyse contracts.

Summarise research.

Translate languages.

Reason through complex business problems.

Generate creative content.

And increasingly, they compete directly with the largest proprietary models.

Only a few years ago, this level of capability was available only through closed commercial platforms.

Today, some of the world’s most capable models are becoming open source.

That changes everything.

Businesses are no longer limited to a single vendor.

Researchers can inspect, optimise and deploy models on their own infrastructure.

National supercomputers can provide sovereign AI capabilities.

Innovation becomes significantly more accessible.

Why 16 GPUs?

People often hear that a model uses “16 GPUs” and imagine it simply runs faster.

Speed is only part of the story.

Models approaching one trillion parameters simply cannot fit inside the memory of a single GPU.

Instead, the model must be distributed.

In this experiment, four compute nodes, each equipped with four NVIDIA GPUs, worked together as a single inference engine.

Different parts of the neural network were executed across different GPUs while high-speed communication kept the entire system synchronised. The deployment combines tensor parallelism and pipeline parallelism to split the workload across multiple machines, allowing the model to operate as if it were running on one enormous computer.

This is one of the hidden engineering achievements behind today’s frontier AI.

The prompt itself may contain only a dozen words.

The infrastructure that answers it can span multiple servers, hundreds of CPU cores, and terabytes per second of communication bandwidth.

Open Source Changes the Economics

Perhaps the most exciting aspect of this experiment isn’t the hardware.

It’s the model.

Kimi is open source.

That means organisations can inspect it, benchmark it, optimise it, and deploy it in controlled environments rather than relying exclusively on commercial APIs.

For many businesses, this opens entirely new possibilities.

Better governance.

Lower operating costs.

Greater flexibility.

Control over data.

The conversation is no longer “Which AI should we buy?”

It increasingly becomes “Which AI should we build our business around?”

Prompt Engineering Is Becoming an Engineering Discipline

Prompt engineering has often been dismissed as simply “finding the right words.”

Reality is considerably more interesting.

A prompt is an interface between humans and probability.

Small structural changes can alter reasoning paths, output quality and consistency.

Recent academic research has already shown that prompt design alone can produce dramatic differences in performance.

But measuring those differences requires experimentation.

Not one prompt.

Hundreds.

Sometimes thousands.

The future belongs not to organisations with the longest prompts, but to those capable of systematically designing, testing and statistically validating them.

Prompt engineering is gradually evolving into prompt engineering science.

From Impressive Demos to Reliable Systems

The biggest lesson from this experiment is surprisingly simple.

Generative AI should not be judged by its best answer.

It should be judged by the distribution of all its answers.

That is the difference between creating a chatbot for a demonstration and building an AI system that supports real business operations.

As open-source frontier models become increasingly capable and high-performance computing becomes more accessible through research infrastructures and AI factories, organisations gain the opportunity to move beyond experimentation.

The competitive advantage will not belong to those who merely use AI.

It will belong to those who understand how to measure it, validate it and improve it at scale.

Sometimes, asking the same question one hundred times tells you far more than asking one hundred different questions.

by Gonçalo Perdigão

Disclaimer

Although I hold a degree in Computer Science Engineering, my day-to-day technical skills are relatively limited. I am not an HPC specialist nor a deep systems engineer. However, with the help of a custom-made GAIA solution, I was able to set up the entire workflow end-to-end on my own, from environment configuration to model execution to large-scale prompt experimentation. This experience demonstrates not only the power of MareNostrum 5 but also how accessible it can be when the right tools and support are in place.

I am part of the CNCA AI Factory (Centro Nacional de Computação Avançada in Portugal), a project designed to accelerate AI-related startups and provide them with access to the extraordinary power of MareNostrum 5. With FCT's support (Fundação para a Ciência e Tecnologia) - thanks to Diana Almeida and Susana Caetano (from FCCN - Serviços Digitais da FCT) and the CNCA team's commitment (thanks to Andreia Gaudêncio, Bernardo Malaca, Catarina Ortigão, Daniel Moraes, Larissa Santos and Pedro Marques).

Some more articles about HPC and experiments: