The EuroCC/CSCS Kubernetes Hackathon took place from November 28 to December 2 in Lugano. Participants worked closely with experienced mentors throughout the immersive coding session to work on tasks from their home institutions. Knowledge about container orchestrators like Kubernetes will become increasingly important in HPC, say the organisers.

5 min read

by Santina Russo

“You can start!” As soon as Mario Valle, co-organiser of the Kubernetes Hackathon, gave this go-ahead, the room began to buzz. The assigned mentors introduced themselves to the nine participating teams at the round tables, and everyone started to log in to their assigned cluster, eager to get going. The hackathon, jointly organised by EuroCC and ETH Zurich/CSCS — the National Compentence Centre (NCC) in HPC Switzerland — and funded by CASTIEL, got underways as the teams began the work to migrate services of their home institutions into the container orchestrator Kubernetes.

“With Kubernetes, containerized software can be controlled and balanced,” explains Mario Valle, a CSCS computational scientist. Nowadays, software functionalities are frequently organised as a set of services — this is increasingly true also in the world of HPC. Kubernetes, also known as k8s, allows for the management and orchestration of a large number of services. Most importantly, Kubernetes enables software engineers to easily scale up and coordinate services on different machines, and to manage services in a tight feedback loop and redundant fashion. This reduces delivery time and cost and prevents system overloads. As a result, the services become more flexible, stable and reliable.

Mario Valle, computational scientist at CSCS and co-organiser of the Kubernetes Hackathon. (Image: Marco Abram, CSCS)

A race against yourself

The week-long hackathon took place from November 28 to December 2 at Hotel Dante in Lugano, Switzerland. The teams came from CSCS and other institutions in Switzerland, as well as from NCCs of other European countries like Austria, Norway and Portugal. For them, the event provided something that is not easily organized in everyday working routines: a time and place where the participants could work on specific problems or tasks in a very focused way — supported by mentors with in-depth knowledge of Kubernetes. “It’s a sort of race, in which everybody wins,” commented Valle.

Some of the mentors are based at CSCS, others work at CERN, the Square Kilometre Array Obervatory (SKAO), or LuxProvide, the Luxembourg national HPC organisation.

Hard at work: Team CTA aims to migrate the management of telescope data into Kubernetes. (Image: Marco Abram, CSCS)

Coding for a better user experience

Each team planned to solve a set of specific tasks connected to services offered by their home institutions. For instance, the team from the Norwegian Research Infrastructure Services (NRIS) aimed to extend their current Kubernetes platform with a specific software bundle, the “NIRD Toolkit”, as team member Siri Kallhovd explained. The toolkit provides researchers with access to European computational resources. The overall goal is to automate more and more services and improve users’ access to these services.

Meanwhile, mentor Rohini Joshi from SKAO was helping the DWDI team from ETH Zurich/CSCS to work towards their goals: to import Elasticsearch, a stack of tools to search and analyse data such as metrics or logs, into Kubernetes; and to connect the search to the message broker Kafka, a software for exchanging data between applications. “Using Kubernetes, we can bring the search engine to HPC in a flexibly expandable way, so it can handle the high quantity of data managed,” explains team member Massimo Benini, a software engineer at CSCS. “This will greatly benefit our users.”

Hackathon in high demand

The MeteoSwiss team, in turn, aimed to import a machine learning (ML) pipeline that predicts horizontal visibility at Zurich Airport into Kubernetes. To achieve this, the team used Kubeflow, a platform dedicated to making deployments of ML workflows on Kubernetes simple, portable and scalable. The team had a busy schedule, since the goal was to import an entire workflow of multiple machine learning steps.

Eirini Koutsaniti and Rafael Sarmiento from CSCS team My Jupyter. (Image: Marco Abram, CSCS)

Interest to participate in the hackathon was high, said Pablo Fernandes, service/business manager at CSCS and managing co-organiser of the hackathon. Nine teams were the maximum that the location could accommodate but many more had applied. “We received 20 applications from institutions all over Europe,” confirms Fernandes. “We were overwhelmed by this high level of interest, really.”

New paradigm in HPC: software-definded infrastructure

The attention toward the event is understandable given the strong trend towards cloud-like infrastructure management that is now taking hold in HPC. “From the traditional hardware-centered view of managing infrastructure, supercomputing is moving more and more towards a software-oriented paradigm,” says Maxime Martinasso, Head of Engineering at CSCS and managing co-organiser of the event. “This calls for tools like Kubernetes, with which HPC providers can deploy entire service stacks in a much more agile, fast and robust way.”

Concurrently, from a scheme with service building blocks that are equal and unchangeable for everyone, the trend points towards an HPC usage that is programmable. “Users will increasingy ask for this, which means that in the future, HPC providers and scientific platform developers will have to be able to deploy auxiliary services with a container orchestrator like Kubernetes,” states Martinasso.

Team CSCS_IAM working towards moving their services for identity and access management to Kubernetes. (Image: Marco Abram, CSCS)

Happy coding for all

The coding week was not only a professional affair, however, but a social one as well. Exchange between participants was promoted by joint coffee breaks and lunches, a social dinner, regular updates from the teams, and a guided tour of CSCS. A dedicated Slack channel helped participants to get in thouch with each other — during coding hours, but also in their leisure time to meet for dinner or a stroll through Lugano’s Christmas market.

The hackathon group visiting CSCS …           and learning about its HPC infrastructure (Image: Marco Abram, CSCS)

On the last day of the hackathon, the teams presented their results. “Sharing the problems that people encountered, how they solved them and what they learned in the process is an important part of the event,” says co-organiser Mario Valle. The feedback from the participants confirmed the hackathon’s success: Before the event, 18 percent of the participants had rated themselves as having no knowledge of Kubernetes; 40 percent had basic, and 17 percent intermediate knowledge. But by the end of the week, this had shifted to a 20 percent basic and 50 percent intermediate experience level. All respondents stated that they were satisfied with the learning experience. One comment specifically mentioned the usefulness of having a dedicated week, group of people, and infrastructure to experiment with something as complex as Kubernetes. And unsurprisingly, there is already talk of the next Kubernetes Hackathon.

The Kubernetes Hackaton was supported by the EuroCC and CASTIEL projects.