Requantization of a fine-tuned and fused model with [mlx-lm]

July 20, 2026NLP

Motivation

In the last post, I laid out an issue of fusing a fine-tuned model with mlx-lm's fuse (mlx_lm.fuse) method.

To be specific, using the mlx_lm.fuse, unlike using a separate local adapter, the performance even with simple evaluation samples degraded noticeably. Unlike this thread, dequantizing and requantizing to the same bits (i.e., 4bit) didn't change the result.

Therefore, I had to settle to the base model + local adapter workaround.

New Developments

While not being urgent, since this has bugged me for awhile, I've always wanted to tackle the issue.

After consulting Claude in a fine-tuning session, we identified that upscaling the bits (i.e., 4bits -> 8bits) worked as expected. To be precise, 8bits quantization performed better than the original 4bits + adapter method.

Here is a few summary comparison between the original 4bits + adpaters, dequantized and 8bits quantized evaluation results.

Metric	Original (4bits + adapter)	Dequantized (4bits)	8bits quantized
Perfect records	51.5%	54.5%	54.5%
Entity Precision	89.7%	89.7%	89.7%
Entity Recall	91.2%	91.2%	91.2%
Entity F1	90.4%	90.4%	90.4%
Polarity Accuracy	82.7%	86.5%	86.5%
Entity Type Accuracy	98.1%	98.1%	98.1%
================================================================================

While the small (33) samples wouldn't represent the full accurate picture, it at least presents a promising result that local adapter is not always needed while this is not ideal (would you like to upscale when you can use half the bits), though.

Here is the overall process.

As can be seen, the base model is Gemma-3 4B 4bit quantized model and the fine-tuned adapter is in the adapters_gemma directory.

Dequantize from the quantized (4bits)

Base model (4bits): mlx-community/gemma-3-4b-it-4bit
Adapter directory: adapters_gemma
Dequantized directory: mlx_model_gemma_dequant

uv run mlx_lm.fuse \
  --model mlx-community/gemma-3-4b-it-4bit \
  --adapter-path adapters_gemma \
  --save-path mlx_model_gemma_dequant \
  --dequantize

Requantize to an upscaled bits (8bits)

Dequantized directory: mlx_model_gemma_dequant
Quantized (8bits) directory: mlx_model_gemma_8bit

uv run mlx_lm.convert \
  --hf-path mlx_model_gemma_dequant \
  --mlx-path mlx_model_gemma_8bit \
  -q \
  --q-bits 8