Run safetensors models locally with Ardana

The checkpoint has no GGUF

You have the checkpoint you want to run locally, but its weights are in safetensors and there is no GGUF. A runtime that expects GGUF leaves you with a conversion step before you can use those weights. In Ardana 0.3.2, we can load supported checkpoints directly if our runtime supports the model’s architecture.

Safetensors is a tensor storage format designed for safer loading than pickle. It handles how the weights are stored, while a runtime implements the architecture that uses them. We still need runtime support for that architecture to run a checkpoint.

We register two runtimes in Ardana: llama.cpp for GGUF files and Candle for supported safetensors snapshot directories. For a checkpoint supported by Candle, I can use the original snapshot without first making a GGUF.

Pull the original checkpoint

I use Qwen2.5-1.5B-Instruct for this example. Our Candle integration suite uses the same checkpoint and pull pattern. The command selects the chat layout for the instruct model and assigns qwen25-candle as its local registry name. Models without decider_config.json already default to chat, but I make that choice explicit here.

ardana pull hf.co/Qwen/Qwen2.5-1.5B-Instruct --layout chat --name qwen25-candle

After the pull, we can inspect its registry record with the first command below. That record identifies the Candle runtime and includes the model source, weights, tokenizer, and profile. The second command lists the models we have pulled.

ardana show qwen25-candle --json
ardana list

The Qwen repository contains config.json, model.safetensors, tokenizer.json, and tokenizer_config.json. For a sharded checkpoint, we read the shard filenames from model.safetensors.index.json. Those safetensors shards must sit at the repository root. An empty or invalid shard map is an error.

For causal models, we need tokenizer.json alongside the weights. The registry also fetches available tokenizer configuration and chat-template files. If another checkpoint lacks a tokenizer, we can supply one from another repository through the tokenizer override.

--tokenizer hf.co/<org>/<repo>

Repository contents decide which pull path we take. An unqualified Hugging Face repository reference selects safetensors detection only when the listing contains config.json and no GGUF file anywhere. This Qwen repository meets that rule. A repository containing both formats takes the GGUF path, and adding a quantization suffix explicitly requests GGUF selection.

Ask for a probability

I can now ask the pulled model a concrete question. The state is a customer message asking for a duplicate charge to be refunded. I pass that message alongside a yes-or-no question, and Ardana returns the probability of yes:

ardana run qwen25-candle "Please refund the duplicate charge." --noul "Does the customer request a refund?" --json

The question flag requests that probability. The JSON flag prints the API response, which gives me a structured answer to use in application code. To make the same decision over HTTP, I start the local server in another terminal:

ardana serve

Our HTTP request keeps the same state and question. I use the pulled alias as the model value and give the question a name, refund, so its answer has a place in the response. I send this complete request to the local endpoint:

POST http://127.0.0.1:8000/v1/systemone
Content-Type: application/json

{"model":"qwen25-candle","state":"Please refund the duplicate charge.","questions":{"refund":{"type":"noul","instructions":"Does the customer request a refund?"}}}

The response identifies the model and gives the answer type. I read the probability of yes from the named answer’s noul field. Input and output token counts sit alongside the answer, under usage, so my application can inspect both the decision and its token accounting through these fields:

model
answers.refund.type
answers.refund.noul
usage.input_tokens
usage.output_tokens

We report zero output tokens for this path. Candle reads hidden states at answer positions and projects only the requested vocabulary-label rows in F32. Our engine converts those logits into probabilities without autoregressive text generation. The model still processes the prompt; zero output tokens describes the absence of generated text.

The architecture decides what can run

I match checkpoints by the exact architecture name in their configuration. We read the field below during pull and reject an unsupported architecture before downloading its weights. The model’s marketing family does not decide which implementation we load.

config.json#architectures

Our causal loader recognizes these ten names. The spelling is the dispatch key, so I use this list when checking a checkpoint’s configuration. That check tells us whether we have an implementation for the architecture it declares.

Qwen3ForCausalLM
Qwen2ForCausalLM
SmolLM3ForCausalLM
LlamaForCausalLM
Phi3ForCausalLM
GraniteForCausalLM
Lfm2ForCausalLM
Gemma3ForCausalLM
Gemma3ForConditionalGeneration
Olmo3ForCausalLM

We also support THXDecisionModel, through a separate decision-model implementation. The Gemma conditional-generation entry above loads its text tower only. I would use it for text input here; that entry does not add image inference.

For causal Candle models, we compute in F32 on CPU. On native Apple-silicon macOS, we use BF16 with Metal. Other targets use CPU, and our Candle builds do not provide a CUDA backend. I check that hardware path before choosing a checkpoint for the machine that will run it.

I select CPU with the setting below. On Apple silicon, a value of negative one or any positive count puts the entire Candle model on Metal. The count does not choose how many layers to offload. If opening Metal fails, we return an error.

--gpu-layers 0

The same question can produce different probabilities

I treat the returned probability as something to check against the task. Our stock causal profiles use temperature 1.0, with no separate temperatures for different question types and listwise score levels. Those defaults define how we turn logits into probabilities. They do not establish calibration for the requests I want to automate.

Execution takes one forward pass per planned prompt row. That row normally represents one question or one isolated score level. I would not describe a request containing several questions as a single model forward pass.

Our regression suite compares causal checkpoints with paired Q8_0 GGUFs across three request fixtures. It requires equal choice picks, yes/no probabilities on the same side of 0.5, and probability gaps of at most 0.1. Identical probabilities are not required by that comparison. These checks define agreement within the regression fixtures; they do not prove task accuracy.

Regression agreement does not establish calibration for the workload I want to automate.

For a decision threshold, I still need examples from the workload where that threshold will be used. I would compare the probabilities with known outcomes before wiring them into an automation.

Try a checkpoint you can inspect

Once you have successfully pulled the Qwen checkpoint, I would start by inspecting it. Its registry record shows where the model came from and which runtime and profile it uses. We can check that record with:

ardana show qwen25-candle --json

The Qwen configuration sets a 32,768-token context ceiling. Ardana takes the smaller of the requested context and that ceiling. With the CLI default of 40,960, our effective limit for this checkpoint is still 32,768 tokens. A prompt beyond that limit is refused instead of truncated.

Now I would try the refund request below. It asks for the probability of yes and prints the API response as JSON. Try it as written, then replace the message with one from your own automation.

ardana run qwen25-candle "Please refund the duplicate charge." --noul "Does the customer request a refund?" --json