Today I put Lovable to the test to see just how fast I could go from a rough sketch to a functioning landing page: fully styled, responsive, and hosted on my own custom domain. Spoiler: it took less than an hour.

🖍️ First Prompt

I started with a simple wireframe sketch, a few reference images for the visual style, and some assets I wanted to include. My first prompt was pretty direct:

Use this attached image 'PXL_20250725_080435921.MP.jpg' as a rough wireframe. Keep the style minimalistic. Have the language used on the page be in Dutch.

Have the landing page contain the following sections:
- Welkom bij Bijenherder Mark (introduction)
- Over mijn imkerij (Since when I have been a beekeeper, where I have my bees and my philosophy how I treat the bees)

Use the following images as assets on the page:
- PXL_20240509_125708558.MP.jpg - image of queen bee, can be used as image at the top of the page
- IMG_20200730_114701.jpg - bonus image of beecomb

I uploaded the files, hit "Generate," and within minutes Lovable delivered a first version. Not bad at all!

🔁 Iterating

Once the initial version is ready, Lovable makes it easy to iterate. You can invoke additional prompts or make local edits by just clicking on a part of the design and typing something like:

“Vertically align the text center, make the font bigger, and use Arial for the heading.”

Changes are rendered instantly, and you can either tweak further or roll back to a previous version. It’s fast, intuitive, and surprisingly fun.

🚀 Publishing

Lovable’s built-in publishing flow lets you deploy your project directly to their infrastructure. If you have a bit of engineering experience, you can also:

• Connect the source to a GitHub repo - it auto-syncs the solution
• git clone the repo locally and run npm build to generate a Vite-based distribution
• Deploy it where you like: AWS S3 + Route53, Netlify, Vercel, you name it

Super straightforward.

🛡 Something about security

For me this was out of scope since I was creating a static page without user interaction. The amount of attack vectors for such a solution are very limited. When you are creating a more expansive, complicated prototype you can also do an audit using Lovable to identify Security caveats to take care of.️

✅ The Verdict

With just a sketch, a few images, and a clear idea, I went from concept to live prototype in under an hour. For non-engineers, Lovable even handles hosting and custom domains out of the box. The experience was smooth, efficient, and genuinely enjoyable.

Sure, it might not be the tool for highly complex or bespoke designs. But for quick idea-to-prototype workflows, it’s a fantastic addition to the toolkit.

👉 Try it yourself on Lovable.dev!

Often when you are compiling your sourcecode in Gitlab you will want to push some changes back to the repository, for instance a created commit or tag . By default Gitlab does not allow the gitlab runner to write back to the repository. Adding that behaviour has been on the backlog of Gitlab for a long time. But there exists a feature called Deploy Keys in gitlab which we can leverage to fix this 🎉

This process involves some shuffling with sensitive files so be mindful where you store these. We are going to generate a private SSH key (+public key), upload that pair into gitlab and use it from within our pipeline to have a trust between the cicd pipeline to be able to push code.

One time setup: generate SSH key

We assume that the following directory exists and is empty: ~/development/gitlab-deploy-token/ (MacOS).

To generate the public/private key pair we use ssh-keygen with the following command:

ssh-keygen -t rsa -b 4096 -C "your@email.address" -f ~/development/gitlab-deploy-token/id_rsa

This will give you a id_rsa file (private key!) and id_rsa.pub which is the public key. the your@emaill.address is only a comment added to the public key, tweak as you desire.

Upload private SSH key to Gitlab

To have the Private SSH key available in your build pipeline we need to add it as a CI/CD variable. You can add it to every project as you require it but to centralise your secrets its better to set them at the root level of your organisation/team/group/...

Because you want to keep your secrets secure it is also wise to think about the layering of permissions that you utilise within your organisation:

• Variables can only be updated or viewed by project members with maintainer permissions (source: Gitlab).

To upload the SSH key as CI / CD variable:

• Navigate to Settings → CI / CD → Variables
• Add key with name SSH_PRIVATE_KEY_TOOLKIT and paste the contents of your private key as the value. (trick: cat id_rsa | pbcopy to copy the file contents directly to your clipboard on MacOS)

All projects underneath the place where you have declared the variable SSH_PRIVATE_KEY_TOOLKIT will now have access to this variable during the build pipeline:

Create Deploy Token in your project

To start trusting the uploaded SSH key we need to upload its public key as 'Deploy Key' into your project.

• Navigate to Settings → Repository → Deploy Keys within your project
• Add a new Deploy Key and add the public key as contents (cat id_rsa.pub | pbcopy)

• ⚠️  Do not forget to tick the box Write access allowed or you will not be able to use your Deploy key to push back to the repository.

After you have added this Deploy Token you can now push to your repository from your build pipeline with the SSH key configured.

Use deploy token in your CI/CD pipeline

We are using node:12 as our docker container which comes pre-loaded with git installed. If your runner does not have git installed you will have to do this manually yourself in your .gitlab-ci.yml

We are going to configure git to be able to push back to the origin with our SSH key as credentials in our build stage as well as configure git to use the original committers email address  to improve traceability.

image: node:12

"Tag version":
  stage: "Build"
    before_script:
      # put SSH key in `.ssh and make it accessible
      - mkdir -p ~/.ssh
      - echo "$SSH_PRIVATE_KEY_TOOLKIT" > ~/.ssh/id_rsa; chmod 0600 ~/.ssh/id_rsa
      - echo "StrictHostKeyChecking no " > /root/.ssh/config
      # configure git to use email of commit-owner
      - git config --global http.sslVerify false
      - git config --global pull.ff only
      - git config --global user.email "$GITLAB_USER_EMAIL"
      - git config --global user.name "🤖 GitLab CI/CD"
      - echo "setting origin remote to 'git@$CI_SERVER_HOST:$CI_PROJECT_PATH.git'"
      # first cleanup any existing named remotes called 'origin' before re-setting the url
      - git remote rm origin
      - git remote add origin git@$CI_SERVER_HOST:$CI_PROJECT_PATH.git
      # have the gitlab runner checkout be linked to the branch we are building
      - git checkout -B "$CI_BUILD_REF_NAME"

There are some magic variables in here: CI_BUILD_REF_NAME (branch name), CI_PROJECT_PATH  (repository), CI_SERVER_HOST (gitlab.com unless self-hosted) GITLAB_USER_EMAIL (email of commit owner) which are all made available as environment variables by default by the gitlab build pipeline. For a complete list you can find them here https://docs.gitlab.com/ee/ci/variables/

Now we are able to tag our commit and push that back to the origin:

script:
    - VERSION=1.2.3
    - git tag v$VERSION -m "🤖 Tagged by Gitlab CI/CD Pipeline" -m "For further reference see $CI_PIPELINE_URL" -m "[skip ci]"
    - git push origin v$VERSION --no-verify

When doing 🧙‍♀️ magic ✨ I always like to add as much clarity and traceability as possible. This is why the commit contains the robot emoji to indicate that it was automation and also why the commit contains a traceback to the gitlab pipeline ( "For further reference see $CI_PIPELINE_URL") which leverages the CI_PIPELINE_URL environment variable to create a bi-directional relationship between the commit and the point of origin where it was created.

⚠️  We use [skip ci] in the tags commit message to prevent a recursive build pipeline which tags itself to build and tag itself and.. well, that wouldn't be enjoyable...

You are now able to push commits/tags/.. to your repository from within your gitlab-ci pipeline! 🎉  The only thing which now remains is to add the deploy token to every repository where you want to leverage this same solution and adapt your .gitlab-ci.yml build pipeline.

🧐 Downsides to this approach

• You are uploading a SSH key which has write access to the repositories which have uploaded its accompanying public key as Deploy Token in Gitlab. This token is retrievable by everyone within the scope where you define it who has enough permissions (maintainer+) to manipulate the CICD settings of gitlab.
• It is possible to leak the private SSH key by mistake by echo'ing the contents in your .gitlab-ci.yml script. It will then end up in the logs of your build pipeline.
• In theory you could generate a SSH key pair per project to scope down the impact when a key gets leaked. Although that would add a lot of manual labor.

One of my co-workers created a Proof of Concept to transform an old code Java codebase into a mavenized setup and split it up into smaller pieces. To do so he started with copy-pasting the original files into a new git repository and slowly committing changes to in the end get the maven setup work 🎉 . Only the thing is: by doing so we had lost all git history of our original files.. 🤔  Git surgery to the rescue! 🩺

➡️ This article assumes you are quite comfortable with day-to-day usage of git and does not explain the basics.

Our situation 😓

original repository containing multiple applications without maven (simplified):

README.md
JavaSource
Configuration/application-1
Configuration/application-2
Configuration/application-...n

target mavenized repository for application-1 (simplified):

README.md
pom.xml
mvn/
src/main/configuration/application-1 <-- containing the application sourcefiles

Rough steps taken to get it working with maven:

• Contents of JavaSource was packaged and moved to a maven dependency
• Application configuration application-{1..n} was split up into separate repositories, files copied into it without history

The plan 🩹🥳

Ideally you would have changes be based on a branch of your original codebase so you can apply them as a changeset. Since that was not the case we needed to bring in the serious git tools to rewrite history to our liking.

• We needed to remove the files inside /src/main/configuration/application-1 copy-pasted into the mavenized setup to prevent conflicts and confusion
• We needed to extract only the history + changes of the files application-1 in our source repository
• merge the mavenized changeset over the original files so they become complete again

These steps will result in retaining the full file history of all application-1 files. Downside is that the commits where the mavenizing setup was created are not atomic - the source files are absent. This was taken as an acceptable tradeoff.

Put on your scrubs 👩‍⚕️

First things first; make two local checkouts in a working directory so we do not break anything unrelated. Because git is distributed we can do all the following operations on a local copy on your machine and only when you are satisfied push it back to your version control system (Github, Gitlab, ..)

mkdir git-surgery && cd git-surgery
git clone git@github.com:crunchie84/blogpost-git-surgery-source.git source
cd source

lets check our git history:

Extract our application from the source repository

Okay, we need to clean this up so we only have the application-1 part which we need. To do so we can use a very handy command existing in git: git substree split  which allows us to extract a folder out of our repository and place only those changes in a separate branch:

git subtree split -P Configuration/application-1 -b rewritten-history-application-1

After executing this command the history now looks as the following:

The newly created branch rewritten-history-application-1 only contains comimts (or the part of a commit) which involved the folder Configuration/application-1. It is noteworthy to observe that the new branch does not share a common ancestor with the original master branch because it is a total rewrite of history.

Prepare our new repository

We are going to take the rewritten history of our source repository and use this as the master branch of our 'new' repository (which we are going to mavenize).

# back to the root directory `/git-surgery`
cd ..
mkdir mavenized-solution
cd mavenized-solution
git init -b master
git pull ../source rewritten-history-application-1

We now have pulled the local filesystem based git repository source with specified branch rewritten-history-application-1 into our (empty) master branch and as a result a clean history without any dangling commits:

A side effect of subtree split is that all the files in the extracted folder are placed top level in your commits which we are going to address in a bit:

As a final preparation we are going to move the files as if they have always lived in src/main/configuration/application-1 to make the history a bit more readable. To do so we can use git filter-branch to rewrite where the files have been all their lives:

# we are in the /git-surgery/mavenized-solution folder
git filter-branch --force --prune-empty --tree-filter '
dir="src/main/configuration/application-1"
if [ ! -e "${dir}" ]
then
    mkdir -p "${dir}"
    git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files "${dir}"
fi'

⚠️  You will get a big warning about the side effects and possible gotchas with filter-branch but for our task it will suffice:

Now onwards to prepare our mavenized-poc to merge onto our prepared repository!

Cleaning up the mavenized PoC repository

We are going to clone the repository and remove the copy and pasted files of application-1 from the commit history.

cd .. # back to the git-surgery root folder
git clone git@github.com:crunchie84/blogpost-git-surgery-poc-target.git mavenized-poc

Original history:

We are going to use the community plugin filter-repo which you can install using brew install filter-repo (on MacOS) to ease our syntax what we want to do:

cd mavenized-poc
git filter-repo --path src/main/configuration/application-1 --invert-paths

We are filtering out all (parts of) commits which have anything to do the folder containing the copied application-1 sourcefiles. The result is a clean history of only the steps to get the mavenized setup working:

Only thing which we need to do is apply the maven setup to our extracted application-1

Merging our cleaned up mavenized setup onto our application repository

We are going to re-use the git pull trick we have used before to get commits from a different repository into ours. But with a twist:

# working dir = /git-surgery/mavenized-poc
git checkout -b mavenizing-application
cd ../mavenized-solution
git pull ../mavenized-poc mavenizing-application --allow-unrelated-histories

First we make a branch in our clean mavenized setup repository because the name will end up in our commit history and this is an important piece of information. The true magic resides in --allow-unrelated-histories Given that git pull is a short hand for git fetch && git merge this option allows us to merge unrelated histories. Normally git will always look for a common ancestor when merging but that does not mean it can not merge without!

After invoking the git pull you should be presented with a git merge commit message dialog in which you can add as much extra information that you deem relevant.

Now we can observe in our git commit history what happened:

Two unrelated histories have been merged together retaining the commit history of both. All that is left is to test it out locally. When satisfied you can now (force) push it to your origin as the new repository 🎉

Surgery success, patient dismissed! 🚀

Parting thoughts

• It would have been easier if the proof of concept had been directly created as a branch on the original repository. When we were at this point that was already water under the bridge.
• Taking the original application repository, creating a branch and then copy-pasting the mavenized proof-of-concept codebase over it was also an option but then we would loose the history of those changes ⚖️... We opted for a solution which tried to retain both histories as good as possible

References

• filter-repo community package - https://github.com/newren/git-filter-repo
• merging unrelated git histories - https://developpaper.com/git-pull-merges-two-branches-without-common-ancestor/
• git subtree explained on stack overflow - https://stackoverflow.com/questions/359424/detach-move-subdirectory-into-separate-git-repository/17864475#17864475

Git repositories used as example in this blogpost

• https://github.com/crunchie84/blogpost-git-surgery-poc-target
• https://github.com/crunchie84/blogpost-git-surgery-source

The complete script explained in this article:

• https://gist.github.com/crunchie84/a070f66a3af57d918e4c345d0ec5ea9b

I have taken some time to play around with AWS CodeArtifact (https://aws.amazon.com/codeartifact/) as Intermediate/private npm package registry. It ties into the AWS ecosystem very well and provides a transparent proxy for packages of npmjs.org to your application. You can keep using your regular tooling (npm) with only an additional aws-cli based login in front of it.

One time setup

1. Create a domain within your AWS account (often: your-company-name, or if scoped your-team-name)
2. Create a repository for your app, optionally linking to an upstream package source
3. Start using it!

Usage

To use your new AWS CodeArtifact package repository you can easily configure its endpoint in your .npmrc file globally or in your project. You can publish private packages to it yourself (unless configured otherwise in AWS)

export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain my-org-name --domain-owner 12345678 --query authorizationToken --output text`

// put in the .npmrc file
registry=https://my-app-name-12345678.d.codeartifact.eu-central-1.amazonaws.com/npm/npm-store/
//my-app-name-12345678.d.codeartifact.eu-central-1.amazonaws.com/npm/npm-store/:always-auth=true
//my-app-name-12345678.d.codeartifact.eu-central-1.amazonaws.com/npm/npm-store/:_authToken=${CODEARTIFACT_AUTH_TOKEN}

Why do you want to use it?

• 🔑 Private packages (internal code sharing)
• 📝 Auditability within AWS of package used for application code
• 💵 Pricing is pay-as-you-go: data stored, data in, data out so very scalable solution. instead of per-user like npmjs which starts at 35 USD/user/month

Summary

• 👮‍♂️ You can create multiple repositories - one for each team (or product) -  this leads to (more) visibility (within AWS) which packages (+version) are used where (auditability: security)
• 🥳 You can publish private packages to your own repository within AWS CodeArtifact
• 🤯 You can even publish a private ‘hotfix’ version of a public npm package without having to do manual hacks in your application codebase or wait for the maintainers to merge your code
• 🧐 I got a 404 during npm install of a package - most likely because the transparent request to npmjs took longer then expected. On retry everything was fine. Not sure if this is just one-time or a common theme
• 👍 Upstream package sources & tooling supported: maven-central, google-android, gradle-plugin, pypi and npm.

I have always been intrigued of the chromebook solution. Zero maintenance to the operating system and very decent hardware at an affordable pricepoint. But my concern always was "will this also work as a primary personal laptop?" Every time I reviewed the Chromebook option the answer was Not yet because the only way to gain access to a terminal was to configure the device in Developer Mode effectively rooting the device. But lately there have been two major improvements in ChromeOS:

• The feature to install and use Android Apps;
• The feature to enable first-class terminal access without having to jailbreak the device.

So I decided the proof is in eating the pudding and bought a Chromebook.

Which device to buy

There are three big concerns in choosing the right Chromebook:

• Storage - Most devices only have 32/64gb SSD internal storage
• CPU - Basic Chromebook usage does not require a hefty CPU as development would require
• Compatibility with Crostini (terminal) since not all chipsets support this

If you have money to spend the Google Pixelbook is the way to go. It has enough storage, is the primary target for Crostini and a fast i7 processor. Current price is ~$1000

Since I was a bit more on a budget I choose the Lenovo Yoga C630 (touchscreen, 64gb SSD & decent i5 processor).

So how did I fare?

The trivial stuff just works(tm). You can use the GSuite for your documents and Google Drive for storing files. I was able to mount my Dropbox as another datasource in ChromeOS which made cross-device much easier. For a lot of applications you can either use their Chrome extension, website of Android application.

Let us dig into some more advanced usecases.

Web Development

The terminal is bliss. It works as expected for doing software development using Visual Studio Code. A delighter for me was that software that you install in the Terminal gets a shortcut in ChromeOS automatically.

Code signing using pgp also works and is as easy as:

• gpg --gen-key
• gpg --armor --export
• Upload the gpg public key to github.
• Configure git to utilise the signing key: git config --global commit.gpgsign true

Even though the terminal runs in a separate sandbox you are able to proxy-pass localhost from chrome to this sandbox environment to reach your web application.

Downloading torrents

You can use JSTorrent as an extension in Chrome to download torrent files. It even works with magnet links too. Downsides are the download speed which was underwhelming.

Alternative route which I have tried is to install Deluge (or any other popular Torrent client) in the terminal but found that the terminal currently has no access to USB connected storage. So you are stuck downloading to the Downloads folder of your chromebook which can contain not much data. USB support is being worked on so expect this to be resolved in the upcoming months.

Cryptocurrency

Unfortunately I have not found any chromeOS compatible wallet. I tried to use my physical wallet in the form of an USB device called Nano S The android app for this device called Ledger Live is not able to detect the key once plugged in; most likely this is an issue with USB pass-through to Android applications from chromeOS.

Luckily I can trade using my phone using Android with OTG support so for now there is no urgent need to have this fixed.

Gaming

You can install Steam using the Unix support via the terminal. But at the moment hardware acceleration is not supported so playing games is not really feasible for now. This might change in the future ofcourse.

Flashing SD Cards

You can use the Chrome OS Recovery Utility and instead of flashing a backup of ChromeOS to an SD card you can flash any arbitrary image. This allows you for instance to flash a Raspberry PI image for your own needs.

3D Printing, IoT Development

Unfortunately since USB support of the terminal is currently not working (ChromeOS 74) a lot of applications are not useable yet. On the bright side; ChromeOS 75 should have this fixed (https://www.androidheadlines.com/2019/04/usb-support-chrome-os-75-versatile.html)

E-Book management

Previously I relied on Calibre to manage my Ebook collection and the books on my reader. I have not found a way to do so with my Chromebook for now.

In conclusion

Using a Chromebook as primary laptop has not been an easy decision. But it has paid of tremendously. For now I enjoy what works and have not come across a lot of usecases which I use with high frequency and are not able to do so. With the imminent release of USB support via Unix/the terminal a lot of goodness will also come to make the Chromebook offering much more versatile.

Past week I finally attended the Lead Developer Conference in London and what a conference it has been. Everything was provided: from quality talks, good and healthy nourishments to even provide childcare for those requiring so.

There were a lot of sessions about the soft skills involved with being a team lead which gave a lot of insights in what motivates peoples, makes people excell at their job and overall feel welcome at their team.

Some of my personal takeaways and talks which most resonated with me were the following.

Lara Hogan - Navigating team Friction  

Learn to decypher the needs of others and what motivates them. There are lot of ways to do so for instance using active listening, asking open ended questions etc. Use the six axis of core needs (Belonging, Improvement, Choice, Equality, Predictability)

Do not require change of your reports, elaborate to them what needs attention and use facts. Use something like the Feedback Equation to create actionable feedback. People are capable to develop their own ideas how to solve their fallacies so ask them to voice these. "How would you like to approach this?"

Melinda Seckington - Level Up: Developing Developers

The analogy with how you learn to play a game vs how you grow as a developer was very striking. Topics like onboarding, starting small, practice skills, repeat your skills and having a mentor.

Pat Kua - Flavours of technical leadership

Pat argued that anyone can be a leader. All it takes is a single action.

“To demonstrate technical leadership, you don't have to be an expert in a particular technical area. You need to understand enough to be able to facilitate the right conversations.”

There are multiple ways to lead:  

• Knowledge cultivator (creates a learning environment)
• The advocate (listens to painpoints, finds buy-in for his vision, creates the community)
• The Connector (knows who you need to solve your issues)
• The Storyteller

There are also a lot of ways to share knowledge and let your leadship become visible. Speak at conferences, blog, interviews, book reading clubs, newsletters, meetups etc.

The last interesting thing which he discussed was career growth and the three paths you can grow towards:

You can read more on his own blog about the trident model of career development.

Neha Batra - Facilitation techniques 202

Neha shared a lot of practical experience how to facilitate sessions:

• Have a clear agenda with the goals, timeline, expectations when to speak & when you can have a break (recharge);
• Engage the group with pairing activities, sharing of input and creating individual contributions;
• Make sure every personality can make their voice heard (Steam Rollers vs Quiet Folks);
• Finish with a positive bang!

The Philips Hue cloud infrastructure is running on Google Container Engine (Kubernetes) since 2014. We have done many Kubernetes upgrades without incidents but I want to share with you the learnings we have attained in our most recent upgrade which failed spectacularly:

It resulted in a production outage which took multiple hours to resolve. But to understand what went wrong and what we have learned from it we first need to discuss a bit how we normally handle Kubernetes upgrades.

New Kubernetes release, now what?

Every 12 weeks a new version of Kubernetes is released to the stable channel. It takes some time for Google to produce a GKE compatible version which is mature enough to be used in production.

Our regular approach is to wait for a bit longer after the new GKE release gets available to let the dust settle. If no bugs are raised by other Google customers we will start our upgrade by manually upgrading the master node and after this is completed we will follow suit with the upgrade of our worker nodes. This process is tested on our development clusters before being executed on the Philips Hue production environment.

On a beautiful morning in January 2019

I'm going to upgrade the production environment's nodepool today. Minor impact (elevated error rate) is to be expected due to application pods getting cycled excessively.

Famous last words. We started the upgrade by using the GKE UI and have it be done in an automated fashion by Google. It would remove one VM from the nodepool, upgrade it to Kubernetes 1.10 and return the new VM into the nodepool until all machines had been rotated. Due to the size of our cluster (~280 machines) this process takes hours.

After a few hours we observed an increase in message delivery failures to one of our partners (Amazon Alexa) which seemed to only increase. In hindsight we now know that this was caused by kube-dns somehow starting to get DNS requests from within our cluster but only having three pods effectively DDoS'ing ourselves with DNS requests.

We aborted the roll-out of the new nodes and after some discussion decided to undo our partial roll-out by creating a second nodepool with 150 nodes of the original Kubernetes version and then move the workload back to these nodes, scaling further up when needed.

The wheels were set in motion. We created the nodepool with our previous kubernetes version nodes and waited for all nodes to become healthy. Only once they came online our master became unresponsive! Even though the master should only be used for the management control plane our metrics started to indicate that things just became dramatically worse, we faced a major outage. #panic

It took us multiple hours together with help from Google Support to get our GKE master node back online, deduce what caused the original error rate to go up and have our service get stable again. Unlucky achievement unlocked; get Google support engineers on our case from every geographical timezone to receive round-the-clock support.

Post Mortem

After the dust had settled we came to the conclusion that our cluster was pretty old - created somewhere in 2016 causing our DNS configuration to be incorrect. Due to the upgrade of the nodes this configuration became somehow corrected which caused DNS lookups to now hit kube-dns first - which was only running with three pods and became overloaded. Scaling kube-dns from three to 110 pods resolved a lot of that issue together with properly configuring stubDomains and upstreamNameservers.

To rollback our upgrade we created a second nodepool which (in hindsight) was of sufficient size to have our master node be automatically upgraded by Google to accommodate such amount of nodes. This made the master be unresponsive for some time and when it finally came back receive a flood of events to process and propagate to all nodes causing it to be even longer unresponsive.

What did not help in this flood of events is that one of our services contained a lot of backends (> 500 pods) - causing the size and propagation of iptables to become cumbersome. This is one of the many dimensions in which Kubernetes scalability can become an issue as discussed in "Kubernetes Scalability: a multi-dimensional analysis" (Presented at Kubecon 2018):

In the end we were able to resolve our outage and have our cluster become stable again. In doing so we have learned a lot about scalability and performance characteristics of our cluster. In the weeks since this outage we have been able to successfully upgrade our nodes to Kubernetes 1.10 but this again surprised us in many ways. Stay tuned for part 2 in this series!

I had this coming for a long time. My blog was originally hosted on Tumblr.com. When I wrote articles I would utilised the public folder of Dropbox to upload and host my images. This worked beautifully until they disabled this feature in september 2017 effectively making all my images become broken. Combined with the sub-optimal editor of Tumblr this suggested it was time to move. Time to search for an alternative. Enter the battle of the blogs.

If you Google "Best blogging platform 2019" the shortlist will include Tumblr, WordPress, Medium and much lower on the list Ghost.

Requirements

• Good writing experience
• Free or low cost
• Ability to write in Markdown
• Copyright of my thoughts retained
• Bonus: ability to import my current articles from tumblr.com

WordPress

Almost everybody knows Wordpress. You can download and host the software yourself which also make you responsible for upgrades & backups to prevent data loss. If you do not want to do this you can also take a subscription for as little as $5 a month to have wordpress.com do this so you can focus on writing high-quality content.

Due to the large install base it is also a big target for CVE exploits so I decided to take the Paas offering for a spin. Luckily, getting some content into it was trivial because they support importing your blog from Tumblr.

But the downside is the editor which was still a pretty basic HTML WYSIWYG editor. UPDATE - After writing this blog I found out that they now have a blog based editor alike Medium and Ghost called Guttenberg which was not enabled by default in their hosted offering.

Medium

Medium is a big platform with a large userbase, cross-referring blog articles to keep users engaged and give your articles more exposure. The downsides of this are that the brand is Medium so they first made it difficult and finally unable to have your own custom URL. The UI is also very much fixed and you as an editor have very little influence over this. But the writing experience is o so sweet. It is the sirens lure of blog writing.

In the end I chose not to go with Medium. There has already been written a great deal by others why Medium in the end might not be the best place to put your content. For example:

• https://sendcheckit.com/blog/why-you-should-put-your-content-on-medium-and-your-own-domain
• https://m.signalvnoise.com/signal-v-noise-exits-medium/

But the final straw which resonated deeply with me was "OWN YOUR WORDS". If you let somebody have control over where you write they effectively have control over what you write.

Ghost

Finally I reviewed Ghost and found out that they not only have paid tiers but the software is actually open source and you can host it yourself. It supports the same nice editing experience as Medium does and even has a native App to write articles on your Mac or even on your mobile device on the go.

Importing my existing content into Ghost was not trivial because in Tumblr the articles are stored as simple objects while Ghost uses the rich block-based approach. But having the Titles and metadata like creation date & tags is enough to polish my old content back into shape in the upcoming weeks.

After researching a lot about Ghost and also discovering that I would be able to host it for free using a Google Cloud Platform virtual machine I took the leap. And you are reading the results. Welcome on my new blog, hosted by Ghost.

Every year Q42 organises a hackathon - W00tcamp. We take two days with all colleagues, invite some of our friends, form teams and try to create something that will amaze you, make you laugh but above all will be an awesome learning experience. This year I enhanced the slide at our The Hague office to have sound & light effects. Read this blog for more details how it came to be.

How do you approach a hackathon?

The goal at W00tcamp is to present your project to a jury at the end of the second day. It will be judged based on level of polish, impact & business value.

What I have learned from taking part in multiple hackatons and w00tcamps in the past years is that finishing something in just two days is hard. Taking the time after the hackathon ends to deliver a working result is even harder. This is the reason why I always have very clear requirements for the project which I bring forwards or team I'll join:

• Focus on building the MVP (you can always expand);
• Treat the night (aka party time 🍻) as if you are working overtime;
• New tech to explore? Do a spike before the hackathon;
• Going to use hardware? Make sure you have all components and did a PoC;
• Slim down the MVP to have the gut feeling it is doable in one day (it never is).

Lets talk about augmenting the slide

A few years back I played around with a sonar sensor + raspberry pi + camera to have the slide in our Amsterdam office take a picture when you slide. You can visit http://www.superslideqam.nl for a pretty detailed write up how it came to be. This gave me the trust that we could do something alike for our The Hague office in a similar time frame.

So for this years W00tcamp I wanted to enhance the sliding experience at our office in The Hague with Lights & Sound effects. You can find the initial pitch here (dutch).

The idea

What if you take the slide and by doing so trigger sound & light effects?

We had all sort of additional ideas if the time would allow it. We wanted to have a start & end sensor to measure your speed. We thought about using an iPad to capture a video of the slide and use facial recognition to make a high score per user (anonymous or mapped to known colleagues). But all these ideas were prioritised as non-mvp.

Hardware required

We needed to get a lot of components from local vendors or AliExpress which takes anywhere between 14-60 days to arrive:

• LED strip of 5m @ 60 LEDS/meter (ws2812b chipset)
• Power supply for led strip (60mA*5m*60 -> 18A@5v) => 100W220v
• 1000uf capacitator (to prevent damaging LEDs on power surges)
• Raspberry Pi for sound effects
• Simple speaker
• Laser
• Laser-diode to receive signal
• Arduino Uno for sensor readings + LED animations

Note; Initially we used an Arduino for the Analog sensor readings + LED animations. A few weeks after W00tcamp we also found that we could directly plug the sensor + LED strip in the Raspberry Pi to further simplify the design using an Analog to Digital converter chip (MCP3002).

Designing the hardware

To determine that somebody is using the slide we made a simple tripwire by having  a laser reflect back to a diode at the entrance of the slide:

Prototyping the casing & mirror assembly

We created a lot of iterations of the casing containing the laser and diode as well as the kinetic mirror mount to reflect the laser back to create the tripwire. Luckily we have a 3d printer at our office.

For reflecting the laser we created our own kinetic mirror mount so that we could adjust the angle at which the laser beam was reflected to have it directly hit the diode. These are pretty expensive to buy but again the 3d printer was a life saver.

The kinetic mirror mount design my team member created is hosted on Thingiverse so that you can use this for your own project.

The result

#success!

EVBox is a producers of electric vehicle charging stations and has these come bundled with a managed software solution. Originally founded in 2010 in the Netherlands they are experiencing a tremendous growth. This has also resulted in them needing to redesign their cloud infrastructure to scale towards the future.

Given the fact that the Philips Hue Cloud is somewhat alike (hardware + cloud offering) and also being ran on Kubernetes they reached out to us to request a workshop to address specific needs and get their engineering team up to speed.

Challenges

Given the fact that EVBox is expanding rapidly they have experienced a lot of growing pains in their DevOps way of working. Getting code shipped to production became more difficult and assessing that it is working as expected using metrics instead of tapping logs quickly becomes cumbersome.

We have split the day into two parts: the morning was geared towards broad Kubernetes knowledge, how to use it and be able to observe the system to decide upon the right course of action. This to facilitate the OODA loop in the development process using Kubernetes.

In the afternoon we had more specific topics to discuss like how to do global rollouts (multi-region), improve network-based security using networkpolicies and ofcourse discuss the hot and upcoming: Istio service-mesh. What it contains and why you want to use it.

All in all we had a great day of sharing knowledge, discussing unsolved issues and how to approach this. It was great to see the passion in the team at EVBox and quality they strive to deliver.

Slides

Got interested in the challenges that EVBox is solving? They are growing rapidly and can always use skilled software engineers.

Because we always want to improve our coding skills and the Philips Hue team has been growing significantly in the last 3 months we decided it was time to flex our developer muscles and practice on our teamwork. A great way to do so is during a Coderetreat which i was glad to be able to facilitate for my team.

A coderetreat is a day-long, intensive practice event, focusing on the fundamentals of software development and design.

During a coderetreat multiple sessions will be held in which people form pairs and try to implement a problem, most times Conway’s game of life. Each session just takes 45 minutes and will enforce certain restrictions to get you really out of your comfort zone. And the best thing is that when the session ends you throw away the code to absolve you of your moral obligations to it. People tend to find this really hard during the first session but will get comfortable with it during the day. Then to wrap up a session we do a short retrospective to share our experiences, take a break for coffee and hack away at the next session!

Schedule of the day

Being it that most of my team had never done a coderetreat before i decided upon the following constraints per session:

1. Getting to know the problem domain & TDD using Ping-Pong in which person A writes a test and person B implements it before switching roles;
2. Force people to think small using Baby-Steps in which we restrict ourselves to create a test and implement its code in just 2 minutes or revert all changes to try again;
3. Challenge ourselves to write very explicit tests using Mute Evil Coder where no speaking is allowed and the test may be implemented as evil as possible before writing the next test while still trying to get all business rules implemented;
4. Experience legacy code by making the pairs implement as fast as possible without tests before reorganizing the pairs and making them fix everything.

And to wrap things up we did a group programming exercise called Mob Programming in which everybody works together with only one computer to implement the solution. We took a different coding kata about a LCD panel to have a fresh challenge:

While mobbing one person is the driver at the computer and only may write code as directed by the navigator. All other persons are co-navigators / researchers. Every few minutes switch roles.

Learnings

We had a blast doing the code retreat. Of all the things we learned the following were most noteworthy:

• We experienced that using a programming language we aspire to use (Go) was hindering our abilities to reach the sessions goals and thus we reverted towards NodeJs in which everybody is fluent;
• Restricting yourself to just 2 minutes is very frustrating at first; people really hated reverting their code. But still almost everybody found small enough steps to make progress;
• Most people were initially hesitant that they could get anything done within 2 minutes but found out that it was actually pretty easy;
• The Mute Evil Coder session ended up being a true game of chess for some in which all tests passed but nothing worked as expected;
• Randomisation of tests was a big solution to sidestep evil coding practices;

But above all: everybody learned a lot and got to know each other much better. Mission accomplished!

Since starting to use RxJs I have been searching for the best and easiest approach for testing. After getting a setup which worked pretty fine they introduced RxJs5 which is largely incompatible with their previous testing approach. This blogpost documents my search for a solution of a way to keep testing in RxJs5 simple.

Testing in RxJs4

The documentation about how to test with RxJs4 is pretty good. Depending on the case i tend to take one of two roads to testing Rx code: dumb it down to promises or full-fledged testing using the TestScheduler.

using Promises

Depending on your needs you can cast the Rx Observable stream toPromise() and verify the output. The pro of this approach is that not much scaffolding is required. The con is that you do not have access to the timings of emissions.

it('can be done using .toPromise()', () => {
  const rxPromise = Rx.Observable.from([2,4,6,8])
    .filter(val => val > 4)
    .toArray() /* because only the last value or error will be passed to .toPromise() */
    .toPromise();

  return rxPromise
    .then(
      res => expect(res).to.equal([6,8]),
      err => { throw new Error(`expected result but got error: ${err.message}`)}
    );
});

By returning the promise to the it() operator you let the test framework (in my case Mocha) handle the promise without having to pass the done callback to your tests and invoke that yourself.

Using the TestScheduler

You can also create an instance of the TestScheduler which gives you acces to virtual time testing. This is also required when you start doing tests with operators which are time-bound like .Delay or .Interval. All these operators take an optional scheduler as last argument for you to replace the default implementation.

Because the testScheduler invokes your Rx stream in virtual time you will need to advance it yourself procedurally or call .startScheduler and let it run its course:

const onNext = Rx.ReactiveTest.onNext;
const onCompleted = Rx.ReactiveTest.onCompleted;

it('can be done using the TestScheduler', () => {
  const scheduler = new Rx.TestScheduler();

  const results = scheduler.startScheduler(
    () => Rx.Observable.interval(100, scheduler).take(3),
    { created: 100, subscribed: 200, disposed: 1000 } /* note, this are the default settings and i only include these for clarity */
  );

  collectionAssert.assertEqual(res.messages, [
    onNext(200 + 100, 0),
    onNext(200 + 200, 1),
    onNext(200 + 300, 2),
    onCompleted(200 + 300)
  ]);
});

The collectionAssert basic implementation can be found in the RxJs4 documentation

Testing in RxJs5 the RxJs4 way

When you migrate your codebase from RxJs4 towards 5 you will find out that a lot of things have been moved, renamed and above all that the implementation of the TestScheduler is no longer available. RxJs contributor kwonoj has created a compatibility shim to help migration towards RxJs5. You can install it using npm npm install @kwonoj/rxjs-testscheduler-compat. Not all features of the TestScheduler are implemented but the most important .startScheduler is working.

const TestScheduler = require('@kwonoj/rxjs-testscheduler-compat').TestScheduler;
const next = require('@kwonoj/rxjs-testscheduler-compat').next;
const complete = require('@kwonoj/rxjs-testscheduler-compat').complete;

it('works in RxJs5 with the compat package', () => {
  const scheduler = new TestScheduler(); // Note; no longer the Rx.TestScheduler

  const results = scheduler.startScheduler(
    () => Rx.Observable.interval(100, scheduler).take(3),
    { created: 100, subscribed: 200, unsubscribed: 1000 } // NOTE: disposed is now renamed to unsubscribed
  );

  collectionAssert.assertEqual(res.messages, [
    next(200 + 100, 0),
    next(200 + 200, 1),
    next(200 + 300, 2),
    complete(200 + 300)
  ]);
});

I have adapted my own version of the collectionAssert to work with RxJs5 in combination with the compatiblity package and is available as a gist

Testing in RxJs5 using the new Marble testing syntax

The RxJs team has introduced marble testing syntax to more visually define how your operator or custom code should operate.

    var e1 = hot('----a--^--b-------c--|');
    var e2 = hot(  '---d-^--e---------f-----|');
    var expected =      '---(be)----c-f-----|';

    expectObservable(e1.merge(e2)).toBe(expected);

At the time of writing this post they have not yet made this approach really easy to use outside of the RxJs5 library itself. There are implementations available to see how to do it yourself. You can also look around in the codebase of RxJs5 to see how to setup your testing framework to do your own marble tests. There is an open issue about documenting testing with RxJs5. I have not yet succeeded to get my testing framework setup to do marble testing in this way.

Conclusions

I would love to start using the marble diagram syntax for my testing because they are less verbose in the setup and communicate the emission of values over time way better than the emissions array filled with onNext/onError/onCompleted statements in RxJs4. But for now it seems just a bit out of reach.

Since the compatibility shim is a great workaround i will stay using it to write tests for RxJs5 as i did for RxJs4 until the marble diagram syntax becomes easier to start using.

This year I got the change to give a talk at the NDC Oslo 2015 about using StatsD to measure your application's performance. Giving a really long talk to a big crowd of non-native dutch attendee's was quite the feat! I had a load of fun and learned a ton.

The NDC

The NDC (Former: Norwegian Developer Conference, Now: New Developer Conference) is an annually hosted conference originating from Oslo but now with spinoffs in London and Australia next year. The focus of the conference is mainly .NET developer oriented but with the increasing popularity for F# now is also expanding the type of talks. The days are long (9:00-19:00) and the diversity in talks is huge (9+ parallel tracks!). Together with the speaker dinner first day, the party the second day it makes for a really intense three day long conference packed with fun and brilliant idea's.

The talk which i gave at NDC:

Knowledge is power! The guide to measure what matters

How do you monitor the key performance indicators of your application? Do you know if signups are decreasing versus last week? Have you adopted agile principles but also a hard time to monitor the improvements of your continuous deployments? In this talk we will briefly discuss multiple measuring solutions before diving into the nitty-gritty details of measuring with the help of StatsD. We will implement a few counters and timers and graph these so we can start to make sense of the data. Then we will use powerful functions to analyse the data and spot trends before your users do.

After this talk you will be empowered to create your own data metrics with help of StatsD and have basic knowledge how to plot these metrics into meaningfull graphs. Be empowered!

Code examples will be in C# but technology demonstrated is not limited to this.

Slides & Recording