Building a Better API

How and why I modified the official xkcd API

Comedy. I love comedy in all of its forms. One of my favorite forms is the comic strip. Growing up I used to spend countless hours reading Garfield, and Calvin and Hobbs comic collections.

I stumbled across xkcd.com a few years ago, like you do on those sleepless nights trolling the internet. (Trolling in the fishing boat sense.)

I have been obsessed with it at times and I think I have read almost all of them and committed them to the internal memory of my brain and can recall them with almost perfect precision.

In search of an iOS pet project

As part of my eventual transition from C#.NET developer to iOS developer I was searching for a project I could build that would show off a pretty complete set of skills. I settled on building an xkcd reader app. I looked on the App Store and there were already a few of them but they all seemed to be flawed in some way or another. So I decided to start the app.

They, (who is they right?), say that “A week of planning can save a month of coding”. In my case that would have been very true. After I had the app about 90% of the way built I noticed a fundamental problem. I was going to be fetching the comics from the APIs located here and here.

Identifying the problem

If you look closely at the second url: ”http://xkcd.com/641/info.0.json” you can see that you have to supply the number of the comic you want to view. I thought, “No problem. I’ll just grab the most recent comic from the first url: ’http://xkcd.com/info.0.json’ and then parse that to find the comic number and then use that upper bound in a loop to get every one of the comics.

Now the more experienced developers in the audience may have already noticed the flaw in this idea. In fact, as a junior developer I should have noticed and I probably would have had I sat down and planned for 20 minutes.

There are currently 1,880 comics on xkcd so in order to get all of them I would have to have a loop that runs 1800 times and makes 1800 requests. These are small requests but lets say there are 100 people using my app worldwide. That would be 18,000 requests every time they open the app, (barring any local storage of course).

This is the problem and as a solution I decided to use/hone some of my C#.NET chops and build a WebAPI project in ASP.NET Core 2.0 and host everything in Azure. It was a fun project but it was some of the toughest stuff I have done to date. Here is what I did:

Solving the problem

I’m not going to talk much about the iOS side of my project here. It’s a pretty straight-forward TableView app with a detail page. I’m mostly going to focus on the difficulties I faced with this project from the .NET Core side.

I wanted to use the latest tech and design patterns and I wanted my code to be as tight as possible. I went with ASP.NET Core 2.0 that had just been released a few weeks before I started. I also made the decision at the start that I wanted to host everything in Azure.

I started with the Azure piece first because I thought that was going to be my biggest stumbling block. All in all it was easier than I expected it to be.

Azure piece

I setup the entire pipeline from start to finish. When I commit and push my code it goes to VSTS and once there the entire Continuous Delivery process kicks off. The app is built, load tested, deployed to a “staging” deployment slot and the previous deployment slot is swapped to production. The whole process takes about 20 minutes which is a little longer than I like but it’s acceptable.

The really cool thing is that I can always publish an emergency hot fix from Visual Studio in a few seconds.

After setting up the App Service in Azure I setup a Server and Database. Then I connected to the database via Sql Server Management Studio. Once I had that all up and running I was ready to start writing code.

Code piece

I used all of the hot new design patterns. I used dependency injection, the repository pattern, etc. The only thing I neglected to do to make this a really robust project is writing unit tests. I know that this is heard all of the time but, "I plan on getting back to them eventually."

I can honestly say that I learned a lot by doing this project. A lot of the code that I see on a daily basis at my day job didn’t really make sense to me. I know the steps when adding functionality to a project but for the first time I am starting to understand them.

So What? Who Cares?

This project was really hard for me. As a junior developer I spend a lot of time reading code, modifying code or copying existing code to add a new page or a new checkbox to a page. However, I rarely get an opportunity to write software from start to finish by myself.

That is part of the reason why I wanted to do this project. You can’t grow as a junior developer unless to get out of your comfort zone from time to time.

I realize that I haven’t really given a lot of information on what I did. If you were looking for a full tutorial, I’m Sorry. What I thought I would do though in this closing section is just highlight some of the difficulties I had. This whole project was straight-forward minus the few issues I encountered below:

• Getting started was honestly the hardest part. Trying to figure out how to create the project and start writing code took me about 4 hours. After all of that was done though I had the project built and fetching data within another 4 hours. I think as junior developers we want to start writing code and have everything come out the first time. This is never the case. Even highly trained software teams have false starts and redos and direction changes. The most important thing is to just get started ASAP and when you code, code with the mindset that you may have to change things.
• Playing of the first item, you need to not be afraid to try stuff. Playing around with the code and trying new and different things is how we learn. I just created new branches for any experimental things I wanted to try and if they didn’t work out I just backed the project out and tried something else.
• Scheduling was a doozy. One of the only really tough problems with this project was scheduling. I had a method in my API that I needed to call periodically to get new comics. Randall posts new comics every Monday, Wednesday, and Friday. I needed to be able to pull those down along with any additional comics he posted or changed. I tried using a Timer but with all of the asynciness it was proving difficult so I found a tutorial here and created my own scheduling framework. Easy. 😃
• Take nothing for granted. Early on I had the code working and I was in the process of seeding my database with all 1,880 comics. While writing code and debugging I had hard coded my loop to only grab 10 items. It was all working perfectly so I added the variable that was the id number of the most recently published comic. I started the project running and I was giggling as I watched my database populate, (23 records, 55 records, 234 records, 389 records, 403 records), CRASH!!! I ASSUMED that it had to be something wrong with my code, (imposter syndrome is a bitch), so I spent a bit of time trying to figure it out. It was a 404 error so I was certain that there was a problem with the JSON that was returned, then I was sure it was the fact that seeding was taking too long and something was timing out. NOPE!!! One of the reasons I like this comic is because it is tech/science/computer based. Here is the link to comic #404.

Lessons Learned:

• Have fun
• Don’t give up
• Check for 404 errors when fetching JSON
• Just because you are a junior developer doesn’t mean you can’t tackle big projects…it just means it might take you longer.

Project

Thanks for reading. If you would like to take a look at the project here is the link to the GitHub page where it is located.