I'm working on the design document right now. I should have something fairly detailed drafted by early next week at the latest.
I like the idea of a floating toolbar; I've been thinking along the same lines myself. What I'm going to propose is putting a "Report A Usability Problem" on every page but allow the problem entry window to float and check what page it's at when "Submit" is pressed. That way users who want the floating palette will have it while those with limited screen real estate can call it up when they need it.
Regarding the idea of having the users rate the severity of the problem at the same time they are entering it, Neilsen separates the two steps, and for good reason, I think. You want people to be focused on entering the particular usability problem *and the heuristic it violates*. Once you have a running list of proposed problems, you'll get a more statistically significant response if you have the entire group of testers rate each alleged problem and then average the ratings.
Keep in mind that we're talking about implementing a very particular, empirically tested method of usability testing. We've got to follow Neilsen's prescription if we want the system to have credibility.
As for implementing it in ACS Classic, I'd love to see that too. I suppose that's up to the folks who have the coding skills.
