Dat Rats

But if a YouTube channel disappears, it’s gone to us.

I worry about this too, sometimes, because it would seem that we’ve created and destroyed more content on the Internet than the entire Greek civilization produced for us.

But other times, I’m ambivalent to the idea. Some of the most important ideas survive and move on to the next level or the next civilization and there’s always progress.

So while yeah, it would suck if these cool/weird/fun sites disappear, and if YouTube one day loses all content from a period of time. But how much would it be a loss for civilization? The ideas would have been absorbed by the people of the time and the most important ones move on with artists and consumers in different ways.

How to make GIFs of sites using WayBackMachine

So… I like following fivethirtyeight’s interesting 2016 Election Prediction page. It shows the ups and downs and the general mood of the election. I’ve been staring at it for so long that I wanted to collect the daily changes and make a nice GIF. I know the Internet Archive’s WayBack Machine collects archives of popular websites, so I went there and found that the Election Prediction page is on there too.

So, I started looking for ways to make a GIF from the WayBack machine. There were some node and ruby scripts and applications which didn’t really work. But then I landed on waybacklapse. Its developer – Kyle Purdon – works for bitly and has built two versions of waybacklapse. The older one is python, node, imagemagick and then some. The newer one is python3 and docker. Eww. I followed the steps of the tutorial for the older version, with a few notable exceptions –

The tutorial is for OS X and is a little dated. What I have on hand is an Ubuntu 15.04 VM, so I went ahead and used apt-get install instead of brew
The tut tells you to use the command “git checkout -t v1.1.0”, but it should be “git checkout -b v1.1.0”. Technically v1.1.0 is a tag, not a branch, but I didn’t know that and just used -b, which worked, so why mess with a good thing, amiright?
You need to have node installed, but not the new node. Install old node with “apt-get install nodejs-legacy” and use the command “nodejs app.js” when you’re running screenshot-as-a-service
The tut doesn’t mention that you need to actually *run* screenshot-as-a-service. I went to the github page for the service and found out that I need to run the above “nodejs app.js” command in order to run a server on the localhost. Technically, waybacklapse has code in it to warn you that the server isn’t running. But that didn’t work so well for me.
The user prompts for waybacklapse only allow for monthly or yearly snapshots. But fivethirtyeight has only been running the site for about 3 months, with daily updates, so those didn’t make sense to me. I wanted to get all the changes. So, after installing waybacklapse with pip, I went ahead and modified the code inside /usr/local/lib/python2.7/dist-packages/waybacklapse/waybacklapse.py with one small change to get all the screenshots instead of just monthly or yearly ones –
1. In the create_payload function, I commented out the collapse variable as follows –

[gist https://gist.github.com/nitinthewiz/260780defd28739c50c05e1c1f83df53]

All was well and good, but not really. Turns out, screenshot-as-a-service pulls a screenshot of the entire page, not just above the fold. Which is great, and not so much. I was looking at a GIF that was way too long to be palatable. So, I needed a way to extracts parts of the screenshots so I could make a nice, clean and small-ish GIF. Luckily, waybacklapse made me install imagemagick. So I looked around and made the following script.

[gist https://gist.github.com/nitinthewiz/d6bebb2e1dc3b39df0dee915f3de0cbc]

It must sit inside the screenshot folder. It parses through the screenshots and converts them into smaller versions of themselves. Finally, I found the command inside waybacklapse which creates the GIF. I modified it a bit and used it to recreate the GIF.

convert -delay 30 /root/fivethirtyeight/2016081011081470853418/final-*.png /root/fivethirtyeight/2016081011081470853418/timelapse/2016electionforecastss.gif

Now, I could go about changing waybacklapse and submitting the code to the author, but he’s moved on to docker and in-house solutions for the dependencies, so I doubt it’ll be a benefit to anyone. Instead, I’ll just leave these notes here so I can reference them in the future. If they helped you, shout out in the comments section. Oh, and I’ll leave you with the GIF I made. –

FiveThirtyEight's Election Forecast in a GIF

Save yourself from the Ephemeral

As users of the Internet, we change a lot. We move email IDs, we jump from one social networking fad to another, we change bookmarking and read-it-later sites and even crash, delete or just forget blogs that we write on.

Most of the stuff I’ve done in the past 10 years or so on the Internet has been pretty personal. Emails, Orkut or Facebook where privacy settings allowed me to block external users or bookmarking sites that were private by default. But recently, most of my contribution to the Internet has been public – twitter and App.net, my blogs and even my bookmarking has been public. So is true for most of us out there. With the shift in social networks’ view of what data should be totally private, there’s a lot of data that’s in the public domain. This also means that there’s equally that much data that can be lost or can stagnate when an eventuality occurs – a web service shuts down because of acquisition or drying up of funds, your blog crashes and you have to start from scratch, you leave a social network and even though you download all your data and invite all your connections to the new one, some don’t join or you can’t upload any of that data anywhere else (how many social networks out there are interchangeable? None.) or maybe you just stop using a site or service and that data just sits there, alone and forgotten (just ask my bookmarks on del.icio.us). Continue reading →