A trial is one attempt at a task in an evaluation. Agents are run over several trials because results vary: the same agent given the same task can succeed on one run and fail on the next, owing to randomness in the model and in the environment.
It matters because a single run says little. Measures such as pass@k and pass^k are calculated from several trials, and the number of trials affects how reliable the result is. Too few trials can make a good or a bad result look more certain than it is.
Running more trials costs more time and money, so there is a trade-off. Trials should also be independent, with the environment reset between them, or earlier runs can affect later ones. Different studies use different numbers of trials, which makes comparisons harder.
When reading or designing an evaluation, check how many trials were run per task, whether the environment was reset and whether the result includes some measure of uncertainty.