OK, I wasn't really speaking on disk access terms but rather "pokes" at the structure itself. Hopefully we've got the entire index cached in RAM :)
In practice there's some overhead in the index structure, you need to differentiate between a leaf and a branch and other stuff. I'm not quite sure how big the entries are but it is bigger than 4 bytes.
On the other hand the overhead for tuples in the table itself is a lot higher than metadata overhead in an index in PG's case, so the penalty for a sequential scan is worse than you might imagine ...