D'Hondt on Google Cloud

02 September 2024
One of my micro-sites is an on-line calculator intended for working out election results using the D'Hondt greatest-average formula and for some time I had been planning on migrating it to the cloud as a learning exercise, the main point of interest being whether it is more economical to run as an on-demand web-app rather than an always-on VPS. Getting things working on cloud hosting has its quirks and irritations but overall having left it to run for a month overall I am impressed with the experience using Google's App Engine, although it is a good solution for only a select range of situations.

Example results

The site started out as a place for the D'Hondt application which was one of the first two Windows95 programs that I wrote back in the late-1990s and since then it was rewritten as a dynamic website and went through several revisions as both web technologies and my understanding of them improved: PHP, Ajax, Javascript, Django, and now Flask. They all used to live in a section of this website but after I noticed people linking to it around the time of the 2009 Euro-elections I decided to spin it out to its own micro-site, and at one point someone offered to buy the dhondt.eu domain which encouraged me to spend some more effort on the site, which turned out to be a means-to-an-end for learning various bits of client-side web development, which is what once again has happened here.

Writing the web application

A while back I experimented with putting Django into Google Cloud but I found it a bit involved and I was soon stung by charges because I did not realise things like having to delete/release resources rather than just switch them off. The previous VPS-hosted site did not use any database functionality since all the past election data-sets are built right into the Python source code so the original idea was to strip out anything that needed a database migration, but disabling some of the built-in apps that have migrations caused Django itself to break. Due to the Django and Flask URL controller and template renderer not being vastly different the whole site was rebuilt using Flask. One benefit of Flask compared to Django is static files are deployed in-place rather than using collectstatic on the server which bring with it the added benefit of not needing to use any cloud storage.

URL controller: Flask's preferred way of specifying URL rules is via decorators although I preferred to use add_url_rule which is very close to how urls.path operates. Flask's blueprint seems to roughly mirror Django's apps although with the former it is a case of just write the code rather than doing something equivalent to manage.py startapp polls. In hindsight Django was far too heavy-weight for the task in the first place but at least I was familiar with it.
Templating engine: Flask's templating engine is Jinja which apart from the subtle thing of using {{ loop.index0 }} rather than {{ forloop.counter0 }} is practically identical to Django's templating engine, and the HTML for the voting & results table was the one chunk of code bought across almost unchanged. While not quite as bad as PHP the mixing of HTML and template control tags is a choice between the template looking a mess or the resulting HTML looking ugly, and in my case it was a mixture of both.
Change of styling: The styling of the menus was a bit broken so I tweaked the site banner but my general feeling is that it really needs more work done to it. I am not the artistic type but having textured wallpapers on websites is rather old-fashioned these days and the image file this website uses is was made something like 14 years ago. One thing that really needs to be done is to make it more mobile friendly since these days mobiles and tablets account for more web browsing than traditional desktop computers.
Bye bye breadcrumbs: I decided to do away with the navigation breadcrumbs because they basically duplicated almost identically the URL and with the site style changes I was not sure where to put them.
Election archive dataset: Some missing Welsh assembly results were added but the whole results archive could do with an overhaul. The choice of colours I have long forgotten the reason for — I think maybe the BBC used white for UKIP back in 1999 — and having the likes of the Scottish National Party and Plaid Cymru listed for English regions is a left-over from previous iterations. It could also do with some non-British datasets.

All in practically the entire code-base was rewritten in the space of two days including changing the way past election data is structured internally which increased flexibility but at the cost of efficiency, although some features such as colour and name customisations were dropped for sake of simplicity. It definitely needs more work but overall the site is in a better state than it was before the migration.

Going live

Although I would later soon find out that the code had bugs the application was at least functional enough that I decided to go live with the newly written site, and it was helpful that Google actually used a Flask application as a demonstration of a Python-based service. Compared to the Django tutorial I had tried a few months prior getting Flask working requires very little effort, especially since I was able to skip all the stuff related to data storage. In the past I had used AWS but I find that I get on better with how Google does and documents things rather than Amazon, but that might be because I've used Google for over about a decade whereas AWS I've only used a few times things setup by other people.

Project structure for Engine Apps

Google has some off quirks such as automatically picking up main.py rather than main.py but the idea is if their suggested project layout is used then the App Engine finds everything it needs to know — the one thing I forgot was to include a app.yaml but apart from that it all worked first time. One thing that is not mentioned in the tutorial is a .gcloudignore which stops certain files being included in the deployment:

.gcloudignore .gitignore .git envFlask __pycache__ dhondt/__pycache__ envFlask

The app.yaml file

The app.yaml contains information the Google App Engine needs to get up-and-running and the snippet shown below is mostly based on the stock one included in Google's tutorial. An addition is the automatic_scaling section which limits the number of instances and hence cost, with max_instances being set to the highest number of active instances seen so far rather than the default value of 20. Reducing it to five also means in the worst case it would take almost six hours to blow an entire day's free quota. However the more important parameter is max_instances which is set to one because the traffic profile does not justify leaving more than one instance idle.

runtime: python39 automatic_scaling: max_instances: 5 max_idle_instances: 1 handlers: - url: /static static_dir: static - url: /.* script: auto

While setting max_idle_instances to one can notionally cause performance issues that is not going to be a concern with a site where a significant portion of the time all instances are stopped. I might up it if traffic becomes heavier but that is a bridge I will cross when needed.

Deployment to Google Cloud

I already had some stuff like billing accounts setup because this was not the first time I had used some of Google's services but their instructions for creating the App Engine project are concise enough, although I skipped using their installer script and instead ran the Google Cloud client in-place via a manually updated PATH variable. My preference is to source a setup script as-needed rather than putting this into by Bash config. From there it is running gcloud app create in the local project directory, and using gcloud app deploy to upload the code to the cloud — any parameters it needed such as which region to use it asked for interactively. Only thing that I will add is it is useful to know about gcloud auth login and gcloud auth revoke for logging in and out of your account respectively. It then provided me with the App Engine URL which in this case was dhondt-eu.nw.r.appspot.com

Attaching dhondt.eu

The next stage is pointing the D'Hondt domain name at the cloud project, and I went down the route of using a custom domain for which Google's App Engine has a pool of shared IP addresses that all applications use and I guess it is using some form of SSL-enabled virtualhosts — it is possible to get a dedicated IP address by using a load balancer but this comes with usage charges. Google recommends implementing egress control so that people cannot use the app engine URL directly, but personally I don't care if people do so as I see no harm in it. Only pit-fall I had was kicking this off while a change in nameserver from DynaDot to CloudFlare was still in progress, so it took longer than expected for SSL to get up and running.

The bottom line

While one motive for doing this conversion was to get experience of using cloud computing, the driving force is finding a hosting solution that is more cost effective than my traditional approach of setting up a virtual computer system with web-server software. The App Engine free tier allows for 28 hours of instance run-time per day and an instance shuts down after 15 minutes idle, and assuming I have interpreted the graph below correctly there are a lot of periods with no instances at all running; As for outbound traffic the first 1GB per day is free and this sort of site is not going to even remotely come close to using it all up. a Python script that was quickly hacked together calculated that under 12 hours of run-time was being used in this 24-hour sample.

Run-time trace

This plot was taken before the automatic scaling parameters were added to the app.yaml file. With instances taking 15 minutes to shut down when idle charges have more to do with the distribution of traffic than the traffic load itself Cost-wise the D'Hondt site was running on a VPS that was coming in at about €20 a year which is unusually cheap but in the month since moving D'Hondt to the cloud I was billed 1¢, which was attributed to artefact registry storage and was deducted from a few dollars of credit still hanging around from a Startup Weekend I attended about a decade ago. An important note is the App Engine together with static HTTP hosting being the only two Google Cloud systems that allow custom domains without needing a load balancer, and the latter comes with usage charges that completely destroy any economic case for choosing cloud over a VPS.

Remarks

Using Google's App Engine is a perfect fit for a low-traffic dynamic micro-site like D'Hondt.EU and Google themselves have stated that it is fine to use App Engine for static sites, although for static websites there are better solutions and in time I will likley migrate my one remaining static App Engine micro-site. However as soon as average traffic rather than the odd spike becomes high enough that more than one instance needs to be running all the time attracting the 6¢ per instance-hour charge, App Engine will basically become uneconomic compared to a VPS pretty quickly. Having to use up a finite number of projects that a Google account starts out with is also a major negative.