NCC Switzerland: FirecREST makes it easy to integrate HPC into web-based applications

In computational science, there is a strong trend towards automated workflows. Now, FirecREST provides an easy way to integrate access to HPC systems into such workflows. The tool is already […]

In computational science, there is a strong trend towards automated workflows. Now, FirecREST provides an easy way to integrate access to HPC systems into such workflows. The tool is already in use at CSCS and ready to be adapted for other European HPC facilities. We asked the developers of FirecREST, Felipe A. Cruz and Erini Koutsaniti, about the tool’s capabilities, its basic design, and the necessary culture change in HPC that it promotes.

June 28 by Santina Russo

Felipe and Eirini, what does FirecREST offer to developers and users?

Felipe A. Cruz: FirecREST brings HPC to web applications. Basically, any web application can use it as a gateway and interface for managing HPC resources. With this functionality, FirecREST fulfills a need in the community and, simultaneously, promotes a culture change towards modern web-based technologies that has not really happened yet in HPC. Today, most users still connect to HPC systems via SSH and command lines — using technologies that date from the 1990s. This is clearly not the most user-friendly way to connect applications and systems. Even more pressing is the fact that, nowadays, research groups using HPC increasingly collaborate in large, interdisciplinary projects that frequently have complex workflows. A typical scenario can involve a party that develops the scientific model, a second party that provides the data, a third that runs simulations, a fourth that handles processing of the results.

Eirini Koutsaniti: Until now, if you wanted to integrate HPC in such a web-based workflow, things would get extremely challenging. FirecREST solves this and acts as an easy-to-use bridge from web-clients to HPC services and resources. In fact, it opens up web access to HPC from any device.

How does FirecREST achieve this integration with HPC?

FAC: Simply put, FirecREST presents a bundle of services that a given HPC provider like CSCS offers, and lets HPC users and developers know how they can access those services. In a way, FirecREST resembles a collection of lego bricks with each brick accessing a specific HPC functionality. Developers can utilize these bricks to manage and run HPC computations at CSCS over the web, and to build customized web solutions to integrate HPC power into scientific workflows.

How does this work in terms of technology?

FAC: As a technological base, FirecREST uses REST APIs. It provides a REST API interface that defines a set of functions, which, in turn, interact with the HPC system through HTTP protocols — so in a straightforward way for developers. One must remember that HPC environments are usually unique: They consist of dedicated and specialised computing resources, networks, and storage, as well as specific libraries and software tools, like the job scheduler. FirecREST handles this complexity for developers and the user community by serving as a gateway that transforms the users’ requirements and workflows into the appropriate HPC infrastructure requests.

EK: For instance, FirecREST provides functions for authentication and authorization, retrieving system status, data management and transfer to and from the HPC facility, execution of jobs, and so forth. Since these functions are exposed by FirecREST’s web-enabled APIs, their response is structured, which is very useful for web applications. This way, communities can release their members from the hassle of having to perform the single steps involving HPC manually. Previously complicated workflows can be simplified, structured, and automated.

How difficult is it for users and developers to adopt FirecREST?

FAC: We have made it easy by design. This is not hard if one uses standard technologies such as REST APIs and HTTP protocols to begin with. In addition, with our python wrapper pyFirecREST, any HPC user can develop a simple python program that facilitates HPC access — that connects users to their HPC provider via the web to submit jobs or transfer data in and out of the HPC system.

Can you give a use case example?

EK: A nice example is the SELVEDAS project, a co-development by CSCS and the Paul Scherrer Institute (PSI). PSI has an immense raw data output of several petabytes per year from its large research facilities, such as the synchrotron or the free electron laser, that needs to be post-processed in real-time. We are talking about an online workflow, meaning that computation power has to be guaranteed at the time the users of PSI’s facilities need it. Thanks to FirecREST, the project team could build an online workflow that includes this specific use of CSCS’s HPC resources.

FAC: The advantages of this approach are evident: By building their own web client, PSI has maintained full control over its entire workflow and its users’ identities.

What if users need HPC services that are not yet exposed via FirecREST?

EK: This question actually points to one of FirecREST’s main advantages. If users or user communities do require a set of business logics or commands that are not yet exposed in FirecREST, this can easily be remedied. We can create a corresponding service and provide an API for this service. For instance, for the SELVADAS project we created a reservation service. This way, when a PSI user reserves a timeslot at one of the PSI facilities, corresponding computation time is reserved at CSCS for the date of the shift — via FirecREST.

FAC: Similarly, FirecREST can easily be adapted for usage at other HPC facilities. For now, FirecREST is in use at CSCS, but with its modular design, it can be extended to access many further technologies, such as specific file systems and services characteristic of other European HPC centres. Our goal is to disseminate the tool so that other HPC providers might adopt it and offer it to their users as a service as well. This will help to standardise HPC access and allow the portability of scientific workflows across HPC infrastructures.

How do users get access to FirecREST?

FAC: The initial step is to request client credentials for the users’ use case, after which they can develop code using the API. A good source of information to get started is the user portal: A simple way for users to develop code with FirecREST is by using the python package mentioned before, pyFirecREST, which is a collection of wrappers for FirecREST’s functionalities. It simplifies the usage of FirecREST by taking care of multiple requests in the background for more complex workflows, or by refreshing the access token before it expires.

How advanced is FirecREST compared to similar existing solutions?

FAC: I think it’s fair to say that FirecREST is leading when it comes to including HPC in web workflows, certainly in Europe. In the US, colleagues at NERSC (National Energy Research Scientific Computing Center) in Berkeley built their “Superfacility API” for the US market with the same idea and technology on which FirecREST is based. If we look at Europe, there are a few initiatives that, while using a different technological concept, offer some of the same functionalities as FirecREST, but not in such a simple and standardised way.

EK: Another important point is that with FirecREST, we are also making life easier for us as HPC providers. To date, individual research groups have had their own way of accessing CSCS’s HPC infrastructure. This renders support and upscaling of services more difficult than necessary for us at CSCS, and I imagine for other HPC providers as well. Improving this situation was an important motivation for us to put forward FirecREST as a standardised way of connecting to the HPC environment.

FAC: In the end, it’s really simple: If we want to continue to grow, if we want to provide more services to more scientific communities and industries, we as an HPC community need to bring web technologies in. Together with colleagues at US HPC facilities, we are currently working on building a community to support and promote this transition — this culture change towards web-based technologies in HPC.

FirecREST is a platform-agnostic gateway and interface to manage HPC resources and services through web applications. The tool handles the complexity of HPC systems for developers and the user community by transforming users’ requirements and workflows into the appropriate HPC infrastructure requests. Developers can utilize FirecREST to run HPC computations over the web, to build customized web tools, or to extend existing tools to integrate HPC into scientific workflows. Please find further information on the product’s website. The dissemination of FirecREST is part of the EuroCC project.