For scenario 2., check out tclwebtest. Filling out forms is easy (see examples in documentation). Looping through the links of a html page works for example like that:

do_request $url
foreach link [link all] {
  array set a_link $link
  # do something with $a_link(url) ...
}

Unfortunately running it from within aolserver is a bit complicated - you need to make its commands as well as the tcl http package available to the running thread. There was some discussion recently either here or on the aolserver mailing list on how to use external libraries with aolserver but I did not follow it closely.

I have a quick-and-dirty solution lying around somewhere that was done for the acs-automated-testing package, which checks on each request if the libraries are already available and if not then it "source"s them. If you are interested in it I can look that stuff up for you.