Author Topic: Python script to download all xkcds  (Read 10132 times)

0 Members and 1 Guest are viewing this topic.

Offline TC01

  • LV6 Super Member (Next: 500)
  • ******
  • Posts: 344
  • Rating: +9/-0
    • View Profile
Python script to download all xkcds
« on: November 20, 2010, 05:21:03 pm »
Well, actually, this is a Python library for accessing xkcd.

It does contain five scripts- one of which does download all xkcds onto your computer. (I haven't actually downloaded all ~850 yet though). There are two others, for opening up random/last xkcds in your web browser, and two others for downloading the random/latest xkcd to your computer.

Plus, you can use the functions available in the library for doing other things.

You need Python 2.x (2.7 is the latest version) to install: you can get it here.

As of version 1.1, you no longer need the feedparser module. I've modified it to use xkcd's JSON interface, which is good for two reasons: the RSS feed sometimes has things in it that aren't comic) which break the old implemention, and secondly: it removes the only third-party dependency.

The download of xkcd.zip includes seven files (minus the readme, which has all this text):

-dowallxkcd.py: script to download all xkcds
-dowlastxkcd.py: script to download latest xkcd
-dowrandxkcd.py: script to download random xkcd
-lastxkcd.py: script to open a random xkcd in browser
-randxkcd.py: script to open the latest xkcd in browser
-xkcd.py: the library itself
-xkcd-1.0.zip: a zipped Python package - if you know how to install Python packages just install this

To install manually on Windows, you'd put xkcd.py in C:\Python27\lib\site-packages (wherever Python is installed, then \lib\site-packages). The scripts can go anywhere.

To install manually on Linux... it goes to the same lib/site-packages folder, but I'm not sure where this is- either /lib/python/ or /usr/lib/python, probably.
« Last Edit: February 06, 2011, 12:33:02 pm by TC01 »



The userbars in my sig are links embedded links.

And in addition to calculator (and Python!) stuff, I mod Civilization 4 (frequently with Python).

Offline yunhua98

  • You won't this read sentence right.
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2718
  • Rating: +214/-12
  • Go take a dive in the River Lethe.
    • View Profile
Re: Python script to download all xkcds
« Reply #1 on: November 20, 2010, 05:23:26 pm »
Nice!  /me will download once he frees up some disk space.  :P

Spoiler For =====My Projects=====:
Minor setback due to code messing up.  On hold for Contest.
<hr>
On hold for Contest.


Spoiler For ===Staff Memberships===:






Have you seen any good news-worthy programs/events?  If so, PM me with an article to be included in the next issue of CGPN!
The Game is only a demo, the code that allows one to win hasn't been done.
To paraphrase Oedipus, Hamlet, Lear, and all those guys, "I wish I had known this some time ago."
Signature Last Updated: 12/26/11
<hr>

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: Python script to download all xkcds
« Reply #2 on: November 20, 2010, 05:33:59 pm »
Nice, I hope if this is popular that this doesn't overload their servers, though. O.o (eg: if someone decided to post the script on 4chan or another ultra-busy community site)
« Last Edit: November 20, 2010, 05:34:20 pm by DJ Omnimaga »

Offline TC01

  • LV6 Super Member (Next: 500)
  • ******
  • Posts: 344
  • Rating: +9/-0
    • View Profile
Re: Python script to download all xkcds
« Reply #3 on: November 21, 2010, 06:54:08 pm »
Reuploaded with a readme (that I realized I omitted previously)- it contains everything in the first post as well as some legal disclaimers.



The userbars in my sig are links embedded links.

And in addition to calculator (and Python!) stuff, I mod Civilization 4 (frequently with Python).

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: Python script to download all xkcds
« Reply #4 on: November 21, 2010, 07:24:55 pm »
Ah cool! :D

Offline ruler501

  • Meep
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2475
  • Rating: +66/-9
  • Crazy Programmer
    • View Profile
Re: Python script to download all xkcds
« Reply #5 on: January 17, 2011, 01:27:33 pm »
very nice I will have to use this. Is there any way to change the website it accesses and download from others?
I currently don't do much, but I am a developer for a game you should totally try out called AssaultCube Reloaded download here https://assaultcuber.codeplex.com/
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCM/CS/M/S d- s++: a---- C++ UL++ P+ L++ E---- W++ N o? K- w-- o? !M V?
PS+ PE+ Y+ PGP++ t 5? X R tv-- b+++ DI+ D+ G++ e- h! !r y

SirCmpwn

  • Guest
Re: Python script to download all xkcds
« Reply #6 on: January 17, 2011, 02:02:07 pm »
Does this handle this?
« Last Edit: January 17, 2011, 02:02:44 pm by SirCmpwn »

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: Python script to download all xkcds
« Reply #7 on: January 17, 2011, 02:14:54 pm »
Lol SirCmpwn XD

It would be funny if the script actually found a comic hidden there. :P

Offline jnesselr

  • King Graphmastur
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2270
  • Rating: +81/-20
  • TAO == epic
    • View Profile
Re: Python script to download all xkcds
« Reply #8 on: January 17, 2011, 03:03:36 pm »
Lol SirCmpwn XD

It would be funny if the script actually found a comic hidden there. :P
Nope, none hidden there.  But there is a comic hidden deep within xkcd.

Offline Builderboy

  • Physics Guru
  • CoT Emeritus
  • LV13 Extreme Addict (Next: 9001)
  • *
  • Posts: 5673
  • Rating: +613/-9
  • Would you kindly?
    • View Profile
Re: Python script to download all xkcds
« Reply #9 on: January 17, 2011, 03:21:20 pm »
There is? o.O I bet this script would fail for some of the newer interactive comics ;)

SirCmpwn

  • Guest
Re: Python script to download all xkcds
« Reply #10 on: January 17, 2011, 03:26:42 pm »
graphmastur, you will now share all knowledge you have on the subject, or face my wrath.

Offline jnesselr

  • King Graphmastur
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2270
  • Rating: +81/-20
  • TAO == epic
    • View Profile
Re: Python script to download all xkcds
« Reply #11 on: January 17, 2011, 05:04:28 pm »
graphmastur, you will now share all knowledge you have on the subject, or face my wrath.
lol, figure it out. I'll give you a hint.  You must see the source for uni.xkcd.com, and your scouter must be broken, since obviously the konami code is broken as well.

SirCmpwn

  • Guest
Re: Python script to download all xkcds
« Reply #12 on: January 17, 2011, 05:07:13 pm »
Oh, I've seen the unixkcd source before, but I'm not able to see it from my phone.  Could you do me the favor of putting the beautified code in a spoiler, seeing as it may be quite a while before I can see it?

Offline jnesselr

  • King Graphmastur
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2270
  • Rating: +81/-20
  • TAO == epic
    • View Profile
Re: Python script to download all xkcds
« Reply #13 on: January 17, 2011, 05:37:32 pm »
Oh, I've seen the unixkcd source before, but I'm not able to see it from my phone.  Could you do me the favor of putting the beautified code in a spoiler, seeing as it may be quite a while before I can see it?
sure:
Spoiler For Spoiler:
THE GAME
Click if you dare. :D

Offline TC01

  • LV6 Super Member (Next: 500)
  • ******
  • Posts: 344
  • Rating: +9/-0
    • View Profile
Re: Python script to download all xkcds
« Reply #14 on: January 17, 2011, 05:41:05 pm »
To answer the original question(s). :P

It does handle 404, because all Internet access runs in a try-except block, and it simply skips over 404 if it fails.

It can't be changed to use other websites, because it depends on the way xkcd works- the page encoding saying "direct link to comic", the RSS feed,  and the page numbers for comics: "xkcd.com/843".



The userbars in my sig are links embedded links.

And in addition to calculator (and Python!) stuff, I mod Civilization 4 (frequently with Python).