Unsloth Qwen3.8-27B 양자화 – 동적 v3.1 업데이트

For the complete documentation index, see llms.txt. This page is also available as Markdown.

Qwen3.8-27B is a 27B parameter dense model Qwen, featuring vision and reasoning capabilities, a 256K context window, and runs locally on 17GB RAM/VRAM setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and Unsloth Desktop.

Nov 11 Update: Qwen3.8-27B GGUFs now use Unsloth Dynamic V3.1 which improves average accuracy by 5-10% accuracy at comparable sizes including Terminal Bench, DeepSWE and more benchmarks. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and future models.

Run Qwen3.8 GuideDownload Unsloth

Thank you Qwen for day zero access. Unsloth quants also include:

• Developer Role Support for agentic tools like Codex

• MTP enabled for fast inference

• Tool calling: Improved parsing nested objects to make tools succeed more

Unsloth quants: Qwen3.8-27B-GGUF

Dynamic V3.1 Update (November 11th 2026)

We're excited to release new Qwen3.8-27B GGUFs using our updated Unsloth Dynamic V3.1 (preview) algorithm which improves Unsloth quants by 5-10% accuracy at the same size on most benchmarks and is the best in class for most tasks including coding, agentic workloads and general chat.

Most quants from 2bit until 6bit are updated. We found 3bit UD3 quants to do on par with UD3.1, so we do not update them. We are actively working on making all quants better!

We ran agentic benchmarks including Terminal Bench 2.1, DeepSWE v1.1, and general benchmarks like LiveCodeBench v6, GPQA Diamond, MMLU-Pro, AIME25 and many others. We used 256K context lengths for all agentic benchmarks, and 128K for the rest. We followed the same methodology for Qwen3.8-27B as DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next which also utilized our V3.1 preview algorithm.

Dynamic V3.1 Full Breakdown

Click below to see full details of benchmarking and more:

Click to expand Full details and breakdown

Exllama-v3 considerations

To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.

If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, the below plot is the true reflection of actual usage.

We have not investigated why Exllama-v3 keeps token embeddings in BF16, but as the lm_head is still quantized heavily, the embedding layer might have "absorbed" most of the quantization damage from it. We would be glad to benchmark new Exllama-v3 non BF16 token embedding quants like using 8-bit for example, and see if benchmarks change.

GSQ-RCO results are misleading

For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well.

We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.

On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse.

Through out investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do BETTER on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.

We also checked and graded our answers using Math Verify Dynamic 3.1 GGUFs and also LightEval and we find exactly the same conclusions. We also tried greedy decoding (temperature of 0), and this could not get the 100% accuracy recovery ISTA found across multiple greedy runs, as GPUs themselves are non deterministic. That is why you benchmarks should be run multiple times. Greedy decoding is also not representative of real world use cases.

We also checked 96K (which is most likely ISTA's budget) and 128K, and our quants still do on average better.

KLD of Unsloth v3.1 vs v3

We did KLD for multiple domains for Unsloth UD3.1 quants and UD3 quants - we reduce KLD throughout most quants and most domains - we did not replace the 3-bit ones, as we found UD3 to do mostly great, and the accuracy recovery on benchmarks was not significant enough with UD3.1 to warrant an update, so we did not update them.

Benchmark details and setups

We follow the same procedure for all benchmarks as for Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash as well as discussed in Dynamic 3.1 GGUFs. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning for Qwen3.8-27B.

⚙️ Usage Guide

Qwen3.8-27B Requirements:

Qwen3.8-27B 4-bit quants work on 16-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM. Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory)

1-bit2-bit3-bit4-bit6-bit8-bitBF16

7-8 GB

9-11 GB

12-14 GB

16-19 GB

23-26 GB

31 GB

56 GB

If you want to use MTP for faster inference, prepare to have 1-2GB extra headroom.

Recommended Settings

Qwen3.8-27B is a hybrid thinking model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can adjust the thinking effort:

ParameterThinking ModeInstruct (non-thinking) Mode

temperature

1.0

0.7

top_p

0.95

0.80

top_k

20

20

min_p

0.0

0.0

presence_penalty

0.0

1.5

repetition_penalty

1.0

1.0

• Maximum context window: 262,144 (can be extended to 1M via YaRN)

• Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

• Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

💡 Thinking + Preserve Thinking

Qwen3.8 has Preserve Thinking which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right):

Qwen3.8-27B comes with support for reasoning_effort, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:

• xhigh (default): for complex tasks demanding thorough analysis

• medium: balancing accuracy and speed

• low: efficient reasoning optimizing for speed and cost

• none

To change thinking / reasoning effort in unsloth run or llama-server, use --chat-template-kwargs '{"reasoning_effort":"medium"}'

If you're on Windows Powershell, use: --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"

Change medium to your desired reasoning level.

Run Qwen3.8 Guide

You can now run Qwen3.8-27B in llama.cpp and Unsloth Desktop. Feel free to change quantization type.

• Hugging Face: Qwen3.8-27B-GGUF • Qwen3.8-27B-NVFP4

• ModelScope: Qwen3.8-27B-GGUF • Qwen3.8-27B-NVFP4

Run in Unsloth DesktopRun in llama.cppNVFP4 GuideQwen3.8-2.4T-A95B Guide

🦥 Run Qwen3.8 in Unsloth Desktop

Qwen3.8 can run in Unsloth Desktop, an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups. With Unsloth Desktop, you can run models locally on MacOS, Windows, Linux and:

• Search, download, run GGUFs, MLX and safetensor models

• Self-healing tool calling + web search

• Code execution (Python, Bash)

• Automatic inference parameter tuning (temp, top-p, etc.)

• Fast CPU + GPU inference via MLX and llama.cpp

• Train LLMs 2x faster with 70% less VRAM

1

Install Unsloth

The easiest way to get started is by downloading the Unsloth Desktop app. Works on macOS, Windows, and Linux.

Download Unsloth

• Download for macOS

• Download for Windows

• Download for Linux

Or, if you prefer to install manually:

MacOS, Linux, WSL:

Windows:

The Unsloth Docker image unsloth/unsloth is available on Docker. Read guide.

Install Unsloth Desktop:

Update Unsloth Desktop:

Install Unsloth Desktop:

Update Unsloth Desktop:

2

Search and download Qwen3.8

Go to Unsloth Chat or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant.

3

Run Qwen3.8

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our Unsloth inference guide.

For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.

4

Serve Qwen3.8 with Unsloth API

You can use unsloth run command and serve Qwen3.8 via an API using llama-server runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our API docs or unsloth start.

5

Unsloth is now ready

You can also do many other things with Qwen3.8 via Unsloth Desktop like:

• Connect tools: Claude Code, Codex, web search, MCP and more

• Train models: Fine-tune text, diffusion, embedding, and more

• Generate media: Create and train images, video, TTS locally

🦙 Run Qwen3.8 in llama.cpp

1

Download the latest version of llama.cpp. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.

2

Download the model via (after installing pip install huggingface_hub). If downloads get stuck, see: Hugging Face Hub, XET debugging

3

To run the model in llama-cli, follow the code snippets below: Remember to change settings according to your use-case.

⚡️NVFP4

Like Qwen3.6, we’re also releasing new dynamic NVFP4 Qwen3.8-27B quants that run ~1.5× faster than BF16 checkpoints, with better performance and comparable file sizes. Run Qwen3.8-27B NVFP4 1.5x faster on 24GB VRAM. We also added FP8 KV cache calibration for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see Qwen3.8-27B), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in vLLM only for now (SGLang is not supported).

• Qwen3.8-27B-NVFP4 quant

batchBF16 total tok/sNVFP4 total tok/sspeedupBF16 per-userNVFP4 per-user

1

89.8

133.7

1.49x

89.8

133.7

8

649.4

938.8

1.45x

81.2

117.3

32

1983.0

2787.0

1.41x

62.0

87.1

64

3048.5

4407.2

1.45x

47.6

68.9

See below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations:

All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s.

For accuracy benchmarks we ran KLD and Top-1 agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16

corpusKLD meantop-1 agreement

zh

0.01628

93.55%

code

0.02600

96.68%

refgen

0.03993

94.46%

chat

0.05818

92.15%

ja / ko / ru / es

0.0124-0.0155

94-95%

For accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:

For more information, you can read our Dynamic NVFP4 quants blog.

To run NVFP4 quants, see below for commands to run Qwen3.8-27B in vLLM or SGLang:

vLLM:

To install vLLM in a separate venv:

Then to serve the 27B variant:

To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

SGLang:

If using SGLang, you must use SGLang version v0.5.19 otherwise it will not work since we quantize the lm_head to FP8.

vLLM has a CompressedTensorsW8A8Fp8 kernel which supports this, whilst SGLang v0.5.19 cannot load the FP8 lm_head. v0.5.19 works now!

For sglang after installing the latest:

You can then serve it:

To enable MTP, use:

🤯Quantization Analysis

NVFP4 quants are 1.5x faster than BF16 and retains 92 to 97% top-1 accuracy.

We used Dynamic 3.0 GGUFs to make Qwen3.8-27B much better!

Top-1 accuracy plot as shown in UD-3 fro Qwen3.8-27B:

And mean KLD for Qwen3.8:

📊 Benchmarks

For GGUF quantization benchmarks you can see above for our quantization analysis or Dynamic V3.0 article.

Qwen3.8-27B

See further below for table benchmarks:

Text Performance

Click to expand full tableBenchmarkQwen3.8-27BQwen3.6-27BQwen3.7-PlusMuse Glimmer-30BOpus4.6 Max

Coding

Agentic terminal codingTerminal Bench 2.1 (Terminus)

73.0

63.4

64.0

51.7

78.2

Agentic codingSWE-bench Pro

61.7

53.5

57.6

51.2

53.4

Repo-level code generationNL2Repo-Bench

42.3

36.2

41.1

--

47.6

Agentic codingDeepSWE 1.1

42.2

13.3

14.2

--

--

Software engineeringQwenSWEBench

79.0

49.3

59.2

--

63.8

Agent

Long-horizon office workCoWorkBench

70.7

61.0

65.1

--

68.2

Professional job tasksJobBench

33.4

21.8

27.6

--

--

Frontier agentic tasksAgents' Last Exam

Pass@120.4Score42.9

[email protected]

[email protected]

--

--

General

Instruction followingIFBench

79.5

69.1

79.1

77.0

62.5

Scientific reasoningGPQA Diamond

89.2

87.8

90.3

83.5

91.3

Multidisciplinary reasoningHLE

30.8

24.0

34.7

22.0

40.0

Competitive codingLiveCodeBench v6

90.3

83.9

89.6

--

88.8

Qwen3.8-2.4T-A95B

Qwen3.8-2.4T-A95B is a 2.4T parameter (95B active) model with rivaling GPT-5.6 Sol. Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit Unsloth Dynamic GGUFs takes 397GB (91% smaller), and larger IQ1_S takes 508GB.

⚙️ Usage Guide

Qwen3.8-2.4T Requirements:

• Qwen3.8-2.4T-A95B-GGUF

Dynamic 1-bit XXXSDynamic 1-bit StandardDynamic 2-bitQ8_0BF16 (Lossless)

397GB

508GB

657 GB

2.6 TB

4.9 TB

Recommended Settings

Qwen3.8-2.4T is thinking-only, while Qwen3.8-Max is hybrid.

Default

temperature = 1.0

top_p = 0.95

top_k = 20

min_p = 0.0

presence_penalty = 0.0

• Context length = up to 1,010,000

• temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

If the model fits, you will get ~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.

Run Qwen3.8-2.4T Guide

You can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB IQ1_XXXS quant (named Q1_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type.

• 2.4T-A95B: Qwen3.8-2.4T-A95B-GGUF

Run in Unsloth DesktopRun in llama.cpp

🦥 Run in Unsloth Desktop

Qwen3.8 can run in Unsloth Desktop, an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups. With Unsloth Desktop, you can run models locally on MacOS, Windows, Linux. Go to Unsloth Chat or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant. Then run it.

Qwen3.8-2.4T-A95B New 1-bit data-types

We extended IQ1_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation). Qwen3.8-2.4T-A95B-GGUF

Click for detailed analysis

Due to naming issues, we used TQ2_0, TQ1_0 and Q1_0 otherwise it won't pop up in the HF repo.

DtypeNamingBPW# entriesIndex bitsBlock

IQ1_S

IQ1_S

1.5625

2048

11

50 B

UD-IQ1_XS

TQ2_0

1.4375

1024

10

46 B

UD-IQ1_XXS

TQ1_0

1.3125

512

9

42 B

UD-IQ1_XXXS

Q1_0

1.1875

256

8

38 B

We are still running benchmarks for the new data-types, but for other large models, we get good results without any QAT / QAD:

DtypeGiBPPLKLDtop-p

IQ1_S

553.204

2.578876

0.564553

78.882

UD-IQ1_XS

513.583

2.931261

0.690161

75.726

UD-IQ1_XXS

473.961

3.540383

0.876007

71.284

UD-IQ1_XXXS

434.340

4.488796

1.109944

66.257

🦙 Run Qwen3.8-2.4T in llama.cpp

1

We need to use the specific IQ1_XXXS branch here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.

If you just want to run the standard IQ1_S and other quants, then compile llama.cpp normally:

2

Download the model via (after installing pip install huggingface_hub). You can choose Q1_0 for IQ1_XXXS or other quantized versions like Q8_0 . If downloads get stuck, see: Hugging Face Hub, XET debugging

3

To run the model in llama-cli, follow the code snippets below. Remember to change settings according to your use-case.

4

To run the general UD-IQ1_S you can do:

5

Then to run it:

Qwen3.8-2.4T Benchmarks

PreviousDeepSeek-V4.1-FlashNextFine-tune Qwen3.8

Last updated 2 days ago

Was this helpful?