Using WWW::Mechanize to get my scratchy 45s

| 3 Comments

I'm a big fan of WFMU's Beware of the Blog. So much music geekery and arcana in one handy source!

Sometimes there will be a post with lots of MP3s for download, like this one today with long-forgotten 45s of songs paying tribute to Merle Haggard. I don't want to listen to the mp3s in my browser, and I don't want to manually do the Save As dance in the browser.

Perl and WWW::Mechanize to the rescue! If you have WWW::Mechanize installed, you also have the mech-dump utility. mech-dump started as a tool to make it easier to create WWW::Mechanize programs by showing what form fields exist, but it does more than that. By default, mech-dump will fetch a page and display the forms and fields on the web page. If you call it with --links, you'll only get back the links, like so:

alester:~ $ mech-dump --links http://blog.wfmu.org/freeform/2012/04/a-tribute-to-the-hag-mp3s.html 
http://blog.wfmu.org/freeform/styles.css?v=6

http://static.typepad.com/.shared:v20120403.02-0-g1ba1fe9:typepad:en_us/themes/common/print.css

http://blog.wfmu.org/freeform/2012/04/you-cant-put-your-arm-around-a-memory.html?no_prefetch=1
... etc ...

Filter that output through grep and pass it to xargs and wget, and you've got a handy MP3-only downloader.

alester:~ $ mech-dump --links [big URL] | grep mp3$ | xargs wget

--2012-04-04 09:27:32--  http://blogfiles.wfmu.org/GG/Skeeter_Harmon_-_A_Tribute_To_The_Hag.mp3

100%[=======================================================>] 4,935,323   4.08M/s   in 1.2s

... etc ...

I suspect very few Mech users are aware of mech-dump, and how handy it can be from the command line. I wish I'd done a better job of publicizing it.

3 Comments

Yes, you're right. I wasn't aware of mech-dump. Thanks for the tip.

If you're a Firefox user, you should look at the DownloadThemAll! extension. It will list all links and you can quickly select e.g. mp3-links and download them. See http://www.downthemall.net/

TIL about mech-dump. Thanks, Andy.

If you want to make mech-dump more visible, perhaps link it in a SEE ALSO section of WWW::Mechanize's POD?

Mech-dump looks great! Seconding Steve's suggestion to update the POD. Also, you're able to save yourself some time/processes by skipping xargs altogether and sucking straight from STDIN with wget's -i switch:

mech-dump --links http://blog.wfmu.org/freeform/2012/04/a-tribute-to-the-hag-mp3s.html | grep mp3$ | wget -i-

Great article!

Leave a comment

Job hunting for programmers


Land the Tech Job You Love, Andy Lester's guide to job hunting for programmers and other technical professionals, is available in PDF, ePub and .mobi formats, all DRM-free, as well as good old-fashioned paper.