How to make GIFs of sites using WayBackMachine

So… I like following fivethirtyeight’s interesting 2016 Election Prediction page. It shows the ups and downs and the general mood of the election. I’ve been staring at it for so long that I wanted to collect the daily changes and make a nice GIF. I know the Internet Archive’s WayBack Machine collects archives of popular websites, so I went there and found that the Election Prediction page is on there too.

So, I started looking for ways to make a GIF from the WayBack machine. There were some node and ruby scripts and applications which didn’t really work. But then I landed on waybacklapse. Its developer – Kyle Purdon – works for bitly and has built two versions of waybacklapse. The older one is python, node, imagemagick and then some. The newer one is python3 and docker. Eww. I followed the steps of the tutorial for the older version, with a few notable exceptions –

1. The tutorial is for OS X and is a little dated. What I have on hand is an Ubuntu 15.04 VM, so I went ahead and used apt-get install instead of brew
2. The tut tells you to use the command “git checkout -t v1.1.0”, but it should be “git checkout -b v1.1.0”. Technically v1.1.0 is a tag, not a branch, but I didn’t know that and just used -b, which worked, so why mess with a good thing, amiright?
3. You need to have node installed, but not the new node. Install old node with “apt-get install nodejs-legacy” and use the command “nodejs app.js” when you’re running screenshot-as-a-service
4. The tut doesn’t mention that you need to actually *run* screenshot-as-a-service. I went to the github page for the service and found out that I need to run the above “nodejs app.js” command in order to run a server on the localhost. Technically, waybacklapse has code in it to warn you that the server isn’t running. But that didn’t work so well for me.
5. The user prompts for waybacklapse only allow for monthly or yearly snapshots. But fivethirtyeight has only been running the site for about 3 months, with daily updates, so those didn’t make sense to me. I wanted to get all the changes. So, after installing waybacklapse with pip, I went ahead and modified the code inside /usr/local/lib/python2.7/dist-packages/waybacklapse/waybacklapse.py with one small change to get all the screenshots instead of just monthly or yearly ones –
   1. In the create_payload function, I commented out the collapse variable as follows –

[gist https://gist.github.com/nitinthewiz/260780defd28739c50c05e1c1f83df53]

All was well and good, but not really. Turns out, screenshot-as-a-service pulls a screenshot of the entire page, not just above the fold. Which is great, and not so much. I was looking at a GIF that was way too long to be palatable. So, I needed a way to extracts parts of the screenshots so I could make a nice, clean and small-ish GIF. Luckily, waybacklapse made me install imagemagick. So I looked around and made the following script.

[gist https://gist.github.com/nitinthewiz/d6bebb2e1dc3b39df0dee915f3de0cbc]

It must sit inside the screenshot folder. It parses through the screenshots and converts them into smaller versions of themselves. Finally, I found the command inside waybacklapse which creates the GIF. I modified it a bit and used it to recreate the GIF.

convert -delay 30 /root/fivethirtyeight/2016081011081470853418/final-*.png /root/fivethirtyeight/2016081011081470853418/timelapse/2016electionforecastss.gif

Now, I could go about changing waybacklapse and submitting the code to the author, but he’s moved on to docker and in-house solutions for the dependencies, so I doubt it’ll be a benefit to anyone. Instead, I’ll just leave these notes here so I can reference them in the future. If they helped you, shout out in the comments section. Oh, and I’ll leave you with the GIF I made. –

FiveThirtyEight's Election Forecast in a GIF