There are objective metrics you can use to determine how effective an interface is. Choose the top ten most common tasks on your system, and measure how long it takes a newcomer to accomplish them. That's one.
Then change your system, and measure again.
You'll need a sample size that's decent, but that works.
You can also define "user errors", and count the number of user errors. For example, clicking on the wrong link might be considered a user error.
Watching newcomer use your system can be very enlightening.
HCI folks have developed methodologies for dealing with these problems; unfortunately, computer scientists aren't given these skills during our formal education. How many computer scientists do you know that had to design a program FOR A PERSON while in school? How many people do you know that had to actually design something for person and get feedback, and then work on it some more. Unfortunately, most of us learn this in the trenches, so we lack a theoretical framework for understand how to best approach these problems.
