All pull requests

Pull requests

• Omit SAE decoder bias for additive CSV steering

  #4366·sdivyanshu90 opened Oct 12, 2026Contributor·

  ·
  1

  1

• Preserve model temperature when sampling is explicitly enabled

  #4365·sdivyanshu90 opened Oct 12, 2026Contributor·

  ·
  1

  1

• feat(tasks): add AboveBoard 52 sovereign Canadian marine benchmark

  #4362·Rolosolo opened Oct 11, 2026·

  ·
  1

• fix(aclue): restore dataset loading without remote scripts

  #4360·ChaceQs opened Oct 11, 2026·

  ·
  1

• fix(afrobench): rename nollysenti columns after the dataset dropped its loading script

  #4359·chrikrah opened Oct 11, 2026·

  ·
  1

• Preserve nested generation options when batching requests

  #4358·AaravMehta-07 opened Oct 10, 2026·

  ·
  1

• Add PT-BR typed decisions (Brazilian Portuguese closed-question tasks)

  #4357·antonio-mello-ai opened Oct 10, 2026·

  ·
  1

• Preserve leaf names for group reference overrides

  #4356·sdivyanshu90 opened Oct 10, 2026Contributor·

  ·
  1

  1

• Fix direct-task bypass metric results

  #4355·sdivyanshu90 opened Oct 10, 2026Contributor·

  ·
  1

  1

• fix(models): preserve None values in undistribute

  #4352·joshduplaa opened Oct 10, 2026·

  ·
  1

  1

• fix(mgsm): match the native exemplars' answer format in native_cot strict-match

  #4351·usmanmateen opened Oct 10, 2026·

  ·
  1

  1

• fix(arabic_leaderboard): include Arabic MMLU in the leaderboard groups

  #4349·samycc777 opened Oct 9, 2026·

  ·

• fix(arabic_leaderboard): list the answer choices in the SciQ prompt

  #4348·samycc777 opened Oct 9, 2026·

  ·
  1

• Add LLMSQL 2.0 Text-to-SQL task

  #4347·DzmitryPihulski opened Oct 9, 2026·

  ·
  1

• fix(llama3): keep question whitespace in mmlu_cot_llama prompt

  #4346·mertkayacs opened Oct 9, 2026·

  ·
  1

• Add Saraiki LLM Benchmark (saraiki_bench)

  #4344·themohal opened Oct 9, 2026·

  ·
  1

• fix(utils): default temperature to 0.0 when do_sample is unset (vLLM sampled greedy tasks)

  #4343·avifenesh opened Oct 8, 2026·

  ·
  1

• Make direct Python task names deterministic

  #4341·sdivyanshu90 opened Oct 8, 2026Contributor·

  ·
  1

  1

• Fix empty sample selection in task document iterator

  #4340·sdivyanshu90 opened Oct 8, 2026Contributor·

  ·
  1

  1

• fix(turkishmmlu): take the last standalone letter in the CoT get-choice filter

  #4339·Anri-Lombard opened Oct 8, 2026Contributor·

  ·

• fix(mmlu-redux): only take a standalone answer letter in the generative filter

  #4338·Anri-Lombard opened Oct 8, 2026Contributor·

  ·

• fix(arab_culture_completion): return the gold index so it matches the choices

  #4337·Anri-Lombard opened Oct 8, 2026Contributor·

  ·

• fix(score): make options_consistency_rate swap labels instead of collapsing them

  #4336·Anri-Lombard opened Oct 8, 2026Contributor·

  ·

• Warn when a group's declared aggregate metric matches no task results

  #4333·jurayh opened Oct 7, 2026·

  ·
  1

• fix(janitor): make the C++ janitor match Python semantics on non-ASCII text

  #4332·zstring opened Oct 7, 2026·4 of 5 tasks·

  ·
  1

  1