I also feel out of place when attending a meeting, it hasn't been always like that but is the way I feel now.
There's a problem with irc meetings at least in my case, it's not something I get paid to do and lately I have been pretty short in time for things I don't get paid for doing, so I just pop in whenever I'm around at the time of the meetings. Don't get me wrong, I'm not saying I want to get paid for attending meetings, but maybe you shouldn't expect too much on people with a lot of different things to do.
While I like the idea of the IRC meetings I think the bulk of discussion and decission making should go to the forums and the honchos list in the rare cases in which is better to keep it private. Having stated this I don't feel you should drop members for not attending irc meetings. What I think this community needs is not less members involved in decission making, is encouraging people to participate, and to do so we should encourage people to commit their works, I think 5.5 is a big step forward for that, but right now both core and leadership team seem close groups from the outside, we should fix that or the current situation won't change.