Stefan Petrea has written up a summary of his building of an MP3 website crawler using WWW::Mechanize and an RDBMS. It’s a good write-up, and good overview of the issues of crawling beyond the obvious “open a page, get the links, follow the links.”.