David: after moving PG to a separate disk (see my thread about forces unknown hitting my root fs :) both this and my original query run fast with enable_seqscan turned off. I didn't think that would be such a big factor but apparently it was.
Russel: yes, this is how it's supposed to be. W/o iscachable, PG's query planner recomputes function calls for each row (which is what David was working around with the subselect above). Even if you set enable_seqscan=off in such a case, EXPLAIN will still output a seq scan plan. I'm not sure how PG knows to make this function not "globally cachable", but if you do select ago('1 minute') twice, you will get two different values.