After sitting on my CBC Radio 3 metadata for just over a week, I finally got around to throwing together a decent downloading script. Actually, the scraper/downloader is a loose federation of scripts, deliberately kept in separate modules so as to allow nice things like, say, running multiple copies thereof concurrently. I'll post a link to the source in the near future, along with a few words of explanation. Maybe I'll even write a README - after all, although CBC Radio 3 is afloat for now, there's no telling how long it will survive the budget axes of doom.
(And, to prevent the inevitable smartasses from chiming in with "you forgot wget, n00b" - nope, it's in there somewhere. That said, I think you'll find these scripts go a tiny bit further...)
Showing posts with label crawling. Show all posts
Showing posts with label crawling. Show all posts
April 3, 2009
March 26, 2009
2017 Songs and Counting
Your guess as to what this does:
(Yes, I've left out some details - like what exactly those scripts do under the covers. I'll post about that when it's finished!)
Better make that 3156 songs. And counting.
#!/bin/bash
for i in `seq 0 25`; do
echo "http://radio3.cbc.ca/nmc/artists.aspx?offset=${i}"
done | tee -a artists.log |\
./url-dumper 1.0 |\
egrep -o "/bands/[^\"]*" | uniq |
while read line; do
b=`basename "$line"`
echo "/play/band/${b}"
done | uniq | tee -a bands.log | ./cbc3-get-music-info 1.0
(Yes, I've left out some details - like what exactly those scripts do under the covers. I'll post about that when it's finished!)
Better make that 3156 songs. And counting.
Labels:
bash,
crawling,
music,
programming,
python
Subscribe to:
Posts (Atom)
