Measuring Whether Black-Box LLM APIs Know When They Might Be Wrong

Reliable deployment of large language model (LLM) APIs requires not only accurate answers, but also measurement protocols that reveal when a model's stated uncertainty is meaningful. This paper presents a lightweight black-box measurement study of three Qwen API models on factual question answering and hallucination detection. We operationalize model uncertainty through self-reported confidence and confidence-conditioned abstention, then evaluate these signals with accuracy, hallucination-detection F1, expected calibration error, Brier score, abstention rate, and selective accuracy versus coverage. Across 1,800 cached API responses on TruthfulQA multiple choice and HaluEval QA, stronger model tiers improve factual accuracy and calibration on TruthfulQA, with qwen3.6-max-preview reaching 0.93 accuracy and 0.055 ECE under confidence prompting. On HaluEval, models achieve high detection accuracy, but self-reported confidence remains over-optimistic, with ECE between 0.099 and 0.135. Simple instruction-only abstention rarely triggers. These results show both the usefulness and limits of prompt-based uncertainty measurement for black-box LLM APIs, and provide a reproducible protocol for low-cost AI measurement science.