The data is set aside before the model is trained or tuned and is used only to measure the final result. Because the model has not seen it, the score shows how well it generalizes, and not how well it memorized.
It can be spoiled. If test material leaks into training data, which is a risk when models are trained on large amounts of text from the internet, scores become too optimistic. Repeatedly tuning a model against the same test set has a similar effect, since the test slowly turns into training data.
For security benchmarks, this matters because published vulnerabilities and exploits are widely available online, and a model may simply have seen them.
Keep the set private or newly written where possible, use it sparingly, refresh it over time and ask how a published score handled contamination.