Flavius Burca

Before you raise your LoRA rank, check which layers LoRA trains. Leave target_modules unset in Hugging Face PEFT, and on Llama or Qwen3 it adapts only q_proj and v_proj. That's 2 of the 7 linear layers in each block. The MLP layers get no adapter at all. The QLoRA paper found the number of adapted layers mattered most. In their LLaMA tests, matching full fine-tuning needed LoRA on all linear layers. Rank r made little difference. Try: target_modules="all-linear" It trains more parameters, so watch memory. Run print_trainable_parameters() to confirm the change.

Print the first two token ids of a Llama 3 prompt. If both are 128000, the model sees the start token twice. This happens when you build prompts in two steps: apply_chat_template(tokenize=False), then tokenizer(text). The template already adds <|begin_of_text|>. The tokenizer adds it again. Hugging Face's docs warn that duplicated special tokens can hurt model performance. Fix it one of two ways: tokenize=True in apply_chat_template add_special_tokens=False when you tokenize later

Temperature 0 can push Qwen3 thinking mode into endless repetition. Qwen's model card says it in bold: do not use greedy decoding. Greedy means always picking the single most likely next token. That's what temperature 0 does. Use the settings the card lists instead: Thinking: temperature 0.6, top_p 0.95, top_k 20, min_p 0 Non-thinking: temperature 0.7, top_p 0.8, top_k 20, min_p 0 Your client or server can override them, so log what each request actually sends. Before you swap models over a loop, read the sampling section of the model card.

Your short prompts in a batch generate garbage? Decoder-only models keep writing from the rightmost token. With default right padding, that token is often a pad. Hugging Face shows the failure on a short batch item: "1, 2, 3" becomes "1, 2, 33333333333". Before you blame the model, set `padding_side="left"` on the tokenizer for generation batches. Quick check: print one padded batch. Pads should sit before the prompt, not after it.

Your chat model finishes your message instead of answering it? Check one argument before you blame the model: add_generation_prompt=True Chat models still just continue text. This flag adds the tokens that open the assistant's reply, like <|im_start|>assistant Leave it out, and the model may keep writing your message. Quick check: print the prompt with tokenize=False. If your template has an assistant header, it should be the last line.

Your fine-tuning code may be hiding the token that tells your model to stop. Set `pad_token = eos_token`, and Hugging Face’s language-modeling collator can ignore both padding AND real end-of-text tokens during training. Before training, inspect one processed example: Padding labels should be `-100`. Real end-of-text labels shouldn’t. If both are hidden, mask padding by position using the attention mask, not by token ID.

Your open model ignores your tools in vLLM? Check two server flags before you switch models. vLLM only turns model output into real tool calls when you start it with: --enable-auto-tool-choice --tool-call-parser <name> The parser must match how your model writes tool calls. Model families use different formats. Qwen2.5 writes Hermes-style calls, so it uses hermes. Qwen3-Coder uses qwen3_xml. Pick the wrong parser and the call can come back as plain text instead of a tool call. Find your model in vLLM's tool calling docs and copy its exact flags.

Apache 2.0 next week. Closed weights today. Thinking Machines just shipped Inkling 2 Small: 594B total / 17B active MoE, vision, on Tinker only. They say full weights land under Apache 2.0 after external safety testing. That's the most permissive license class any major lab has promised this cycle — if the drop actually ships. Waiting for the download, or trying it on Tinker first?

80.5 on Terminal-Bench 2.1. Dense 65B. Ripple-65B Preview is out from Ricursive: Apache 2.0, 12T tokens, 1M context. Their table has it edging Beam (80.1) and beating Muse Spark Open (76.4), Inkling (63.8), Nemotron 3 Ultra (56.4) on that bench among U.S. open weights. Caveat: lab comparisons, 0.4-point lead over Beam. DeepSWE: 44.3 vs Muse Spark Open's 60.0. Pull it today for coding agents, or wait for a third-party run?

87 ms. That's how long the engine pauses to swap a 2.5 GB rank-16 LoRA under eight live generations. No restart. Staging that adapter took ~16 s in host RAM while traffic kept the current weights. In-flight requests continue on the new policy from their next token. Do you drain adapters between RL steps, or swap them under traffic?