My experience with IE 5.5 tells me that IE could be caching everything and serving it up on domain access, but actually contacting your server on IP accesses. IE has, in the past, been fairly ruthless and heavy-handed on local caching, and totally ignores the Reload/Refresh functions. Perhaps it always takes a long time, but you have not yet noticed?
OTOH, without any logging information, we can't really help.
