Hi Malte,
Listbuilder may be OK to render the list of 20 or so results for each page, but I would highly disencourage the use of the pagination feature of listbuilder. That's the main point in the last part of this discussion: You don't want to spend time checking the permissions of items that you are not going to display in the first place.
Project/Open has its standard way of pagination using "limit" and "offset", which would avoid the issue above. But I could even imagine a different approach with eliminating the object_ids of the previous pages by passing them around on the HTML level... (well, not that in serious, but you get the point with my preocupations in terms of performance?)
Bests,
Frank