Issues

• CascadeEvaluator._get_llm_correctness counts a judge reply that starts with "Correctness" (or "Correctly") as correct, even when the verdict is negative

  open-compass/opencompass#2657 · Filter by author shaurya416 opened on Sep 25, 2026
  

• [Bug] FollowBench post-processing runs eval() on model output (arbitrary code execution)

  open-compass/opencompass#2655 · Filter by author linhongyu510 opened on Sep 25, 2026

• JudgeEvaluator/RMBEvaluator score judge replies that never state a verdict, by reading the first character of the raw unparsed text

  open-compass/opencompass#2654 · Filter by author shaurya416 opened on Sep 24, 2026
  

• gsm8k_postprocess breaks numeric answers at thousands separators (\boxed{$9{,}500} → 500, \boxed{8,000} → 000)

  open-compass/opencompass#2647 · Filter by author laa1991 opened on Sep 18, 2026
  2

• TopkRetriever crashes on transformers 5.x: tokenizer.encode_plus was removed

  open-compass/opencompass#2635 · Filter by author rrrxxx0510 opened on Sep 8, 2026

• Grading integrity: model code rewrites ground truth in three benchmarks; CRUXEval splices prediction into assertion; unparseable verdicts exit denominator; LCEvaluator eval() on raw prediction (follow-up to #2535)

  open-compass/opencompass#2631 · Filter by author AUTHENSOR opened on Sep 3, 2026
  

• [Bug] ToxicEvaluator: expected_max_toxicity uses built-in max() on a NaN-containing array, so the metric depends on sample order

  open-compass/opencompass#2627 · Filter by author IgorVoytyuk opened on Sep 1, 2026
  

• OpenAI-compat: OpenAISDK openai_api_base /v1, not the requests OpenAI full-path default

  open-compass/opencompass#2615 · Filter by author cursor opened on Aug 28, 2026
  

• Feature request: EvalPort import/export for CustomDataset (adapter already built and tested)

  open-compass/opencompass#2604 · Filter by author adhabnr-ux opened on Aug 21, 2026
  

• CompassAcademic: 4 of 7 objective boards have an unseparated #1, two are exact ties broken by display order

  open-compass/opencompass#2574 · Filter by author ipezygj opened on Aug 1, 2026
  1

  

• [Bug] Qwen3.5 cannot run when using HuggingFacewithChatTemplate

  open-compass/opencompass#2573 · Filter by author hobbitlzy opened on Jul 31, 2026
  

• MedFailBench integration proposal for clinical AI safety evaluation

  open-compass/opencompass#2516 · Filter by author goktugozkanmd opened on Jul 8, 2026
  2

• [Feature] DIK-structured Turkish syntactic dependency annotation dataset for structured linguistic evaluation

  open-compass/opencompass#2508 · Filter by author nane100503-gif opened on Jul 3, 2026
  1

  

• [Feature] 集成 ThinkCheck 作为 OpenCompass 的“推理健康度”评估维度

  open-compass/opencompass#2475 · Filter by author luoxuejian000 opened on Jun 8, 2026
  

• [Feature] TruthfulQA mcq模式评测支持

  open-compass/opencompass#2427 · Filter by author IgorZan opened on Mar 31, 2026
  1

  

• [Bug] [Feature] 评测框架代码过于耦合，失误得重新跑推理，文档中没有看到可以复用推理结果从新评测的命令

  open-compass/opencompass#2424 · Filter by author mozhang-ah opened on Mar 30, 2026
  1

  

• [Feature] opencompass公开学术多模态榜单公布qwen3.5模型的结果

  open-compass/opencompass#2413 · Filter by author cobraheleah opened on Mar 13, 2026
  

• [Feature] Question: where would a long horizon "tension crash test" TXT pack fit in OpenCompass?

  open-compass/opencompass#2396 · Filter by author onestardao opened on Feb 6, 2026
  1

  

• Feature Request: Add Judge Calibration Diagnostics for LLM-as-Judge Evaluation

  open-compass/opencompass#2392 · Filter by author elandesberg opened on Feb 3, 2026
  1

  

• [Feature] Add another OpenAISDK model using prompt completion API

  open-compass/opencompass#2386 · Filter by author tongl2 opened on Jan 14, 2026
  2

• [Bug] mmbench 文档不可访问

  open-compass/opencompass#2349 · Filter by author lll123github opened on Dec 8, 2025
  

• [Feature] 使用飞书/QQ群来进行交流

  open-compass/opencompass#2348 · Filter by author lll123github opened on Dec 8, 2025
  

• 期望开通slack方便大家交流分享合作

  open-compass/opencompass#2341 · Filter by author wenba0 opened on Dec 3, 2025
  1

• [Bug] 关于评测集社区-daily benchmark模块AI解读的问题

  open-compass/opencompass#2339 · Filter by author cobraheleah opened on Dec 1, 2025
  

• [Bug] MedBench模型答案上传文件报错

  open-compass/opencompass#2338 · Filter by author Visualsf opened on Dec 1, 2025