recently i had to download various pdf files from a vendor website, which list them according to different criterias. since i'm lazy, i wanted to write a script that would download all those files for me.
that would be quite easy, but the list was generated by some ajax when i was changing the criterias... so, what to do to pally this problem?
there are some solutions out there for the perl programmer.
the same author, leveraging his knowledge, also wrote www::scripter that's supposed to achieve the same result.
enters http::proxy which is a configurable proxy written in perl by book. among the various examples, it comes with a logger.pl script that displays all the urls accessed by your browser (with cookies). run it as is, configure your browser to use a proxy on localhost port 3128, and check the output of logger.pl while tinkering with your ajaxy application.
chances are that you'll see a url such as http://www.examplecom/path/to/script/list/?crit1=foo&crit2=bar&limit=10
well, that was the case for me, and i could just download the list according to my criterias, parse it with the excellent html::tree and download each of the files separately... \o/