So… I like following fivethirtyeight’s interesting 2016 Election Prediction page. It shows the ups and downs and the general mood of the election. I’ve been staring at it for so long that I wanted to collect the daily changes and make a nice GIF. I know the Internet Archive’s WayBack Machine collects archives of popular websites, so I went there and found that the Election Prediction page is on there too.
So, I started looking for ways to make a GIF from the WayBack machine. There were some node and ruby scripts and applications which didn’t really work. But then I landed on waybacklapse. Its developer – Kyle Purdon – works for bitly and has built two versions of waybacklapse. The older one is python, node, imagemagick and then some. The newer one is python3 and docker. Eww. I followed the steps of the tutorial for the older version, with a few notable exceptions –
- The tutorial is for OS X and is a little dated. What I have on hand is an Ubuntu 15.04 VM, so I went ahead and used apt-get install instead of brew
- The tut tells you to use the command “git checkout -t v1.1.0”, but it should be “git checkout -b v1.1.0”. Technically v1.1.0 is a tag, not a branch, but I didn’t know that and just used -b, which worked, so why mess with a good thing, amiright?
- You need to have node installed, but not the new node. Install old node with “apt-get install nodejs-legacy” and use the command “nodejs app.js” when you’re running screenshot-as-a-service
- The tut doesn’t mention that you need to actually *run* screenshot-as-a-service. I went to the github page for the service and found out that I need to run the above “nodejs app.js” command in order to run a server on the localhost. Technically, waybacklapse has code in it to warn you that the server isn’t running. But that didn’t work so well for me.
- The user prompts for waybacklapse only allow for monthly or yearly snapshots. But fivethirtyeight has only been running the site for about 3 months, with daily updates, so those didn’t make sense to me. I wanted to get all the changes. So, after installing waybacklapse with pip, I went ahead and modified the code inside /usr/local/lib/python2.7/dist-packages/waybacklapse/waybacklapse.py with one small change to get all the screenshots instead of just monthly or yearly ones –
- In the create_payload function, I commented out the collapse variable as follows –
[gist https://gist.github.com/nitinthewiz/260780defd28739c50c05e1c1f83df53]
All was well and good, but not really. Turns out, screenshot-as-a-service pulls a screenshot of the entire page, not just above the fold. Which is great, and not so much. I was looking at a GIF that was way too long to be palatable. So, I needed a way to extracts parts of the screenshots so I could make a nice, clean and small-ish GIF. Luckily, waybacklapse made me install imagemagick. So I looked around and made the following script.
[gist https://gist.github.com/nitinthewiz/d6bebb2e1dc3b39df0dee915f3de0cbc]
It must sit inside the screenshot folder. It parses through the screenshots and converts them into smaller versions of themselves. Finally, I found the command inside waybacklapse which creates the GIF. I modified it a bit and used it to recreate the GIF.
convert -delay 30 /root/fivethirtyeight/2016081011081470853418/final-*.png /root/fivethirtyeight/2016081011081470853418/timelapse/2016electionforecastss.gif
Now, I could go about changing waybacklapse and submitting the code to the author, but he’s moved on to docker and in-house solutions for the dependencies, so I doubt it’ll be a benefit to anyone. Instead, I’ll just leave these notes here so I can reference them in the future. If they helped you, shout out in the comments section. Oh, and I’ll leave you with the GIF I made. –