t-online.de Weather successfully modernized with cloud-native functions

t-online is Germany’s biggest news site - with more than 49 million monthly users via desktop, smartphones, tablets, social media and public video. Ströer Digital Publishing GmbH (SDP) combines publishing expertise with innovative technology, intelligent products and efficient marketing solutions.

More than 150 editors report from the newsroom in Berlin on politics, business, sports and entertainment as well as extensive advice topics.

In addition to the news portal t-online.de and smartphones and iPads apps,
Ströer Digital Publishing GmbH’s portfolio also includes the t-online browser and the t-online weather app.

t-online.de is the biggest German news site with over 1 billion page views per month. The “old” t-online website originates from the earliest days of the German Internet. Back then, portals such as Yahoo, were the major source of new information. In 2015 t-online.de was bought by Ströer, best known for its outdoor advertising and being the biggest ad publisher in Germany.

Even though t-online was a profitably running platform, its technology stack showed its age. Still hosted in its own data centers, scaling the site at sudden traffic pikes posed a constant challenge for the company. Moreover, the need to provision resources in advance created significant costs. The legacy code base slowed down the company from innovating as well. Therefore, Ströer decided to migrate its entire technology stack into the AWS Cloud to benefit from improved scalability and cost-efficiencies of hyperscalers.

A massive undertaking was started that took dozens of engineers, designers and product managers over three years until launch day on May 3rd 2022. As part of this migration, the opportunity was taken to modernize one of the major legacy applications: t-online Wetter. To get support in this endeavor and to help meet the May deadline, Ströer reached out to PCG to modernize the Wetter website and associated backend. In the process PCG performed a complete overhaul of the architectural design utilizing AWS’s Cloud services to deliver a modern, scalable solution that allows for future development and ease of maintainability.

At its core, t-online Wetter shows accurate weather forecasts for German as well as international locations for up to 14 days in advance. It includes the website, mobile iOS and Android apps as well as special purpose screens in malls and metro stations. If you’re waiting for a metro in a German city, chances are you’re glancing at a t-online screen showing you the current weather forecast. T-online Wetter serves over 14 million page views a month. But weather data isn’t free of charge and each page visit has a unit cost associated. Adding to the challenge of scale and keeping the cost associated as low as possible, the engineering team had no prior experience with running workloads in a cloud native fashion.

The solution was to embrace a cloud-native architecture and leverage the benefits of modern open source software. PCG designed the architecture utilizing AWS services consisting of two parts: The first part is the Smart Caching API, responsible for caching and providing the weather data in a smart and cost-effective manner. The second part is the user-facing website optimized for fast browser-experience as well as SEO.

Smart Caching API

More than 10 million people use t-online Wetter per month. Each user-request results in the backend asking the proprietary weather API for respective data points - data t-online Wetter has to pay for. In order to ensure that costs stay low, an API caching / transformation layer was introduced relying on a combination of AWS services such as AWS Lambda, Amazon CloudFront, Lambda@Edge and Amazon DynamoDB.

• Lambda@Edge is used to normalize the requested weather forecast location coordinates to one decimal. This has drastically increased the cache-hit-ratio and thus reduced requests to the origin. Less requests to the origin means less costs for accessing proprietary weather data.
• CloudFront is used to cache API responses for up to 15 minutes
• For requests that can not be served via the cache, a Lambda is used to make requests to the proprietary weather data API
• To circumvent the need to reauthorize on every request, and to reduce the load on the external API, authentication tokens are stored for reuse in DynamoDB.

The provisioning of the architecture was achieved using Terraform embracing IaC (infrastructure as code) allowing for automatic deployments and reducing the need for manual operation management.

Production-Grade frontend

Over the years, significant effort has been invested in having the t-online website rank high for relevant keywords. This resulted in the majority of the traffic of t-online Wetter being organic traffic coming from search engines. When migrating an entire website with tens of thousands URLs however, there’s always the risk of losing a good ranking.The chosen strategy was to optimize the new frontend for search engine crawlers by delivering server-side rendered HTML, as well as to optimize for client-side performance so the user enjoys a fast browsing experience, which in turn is also rewarded by search engines.

While evaluating different options for a suitable solution, Next.js emerged as the best choice rather early on as it provides server-side rendering out of the box and is built on top of React, which allows for building production-grade frontends for high-traffic websites.

Choosing the right rendering strategy

A feasibility study was executed in order to decide on the best rendering strategy. After all, the use case was not as straightforward as a news site for example. A news article is written once and usually doesn’t change after that. This makes caching easy. Weather forecasts on the other hand change frequently. Users of t-online Wetter want to see the most up-to-date forecast data in their locations and want the site to load fast.

On the business side however, one of the objectives is to reduce costs incurred by loading new weather data. To resolve this dilemma between end user and business interests, PCG had to find a good balance between data costs, currentness and latency.

The study concluded that using ISR would be the right approach to balance latency and cost, while providing up-to-date content to the users. The team started with ISR and set the cache invalidation of the weather forecast pages to 15 minutes. In the worst case scenario, a user would be served 15-minute old content. During the prototype phase it showed that the regeneration by Next.js was generating an unacceptable high latency. To keep the benefits of the ISR approach, the team decided to implement their own version of ISR using a combination of server side rendering and CloudFront as a CDN caching layer.

Cloud-natives Server-Side Rendering

Since the Smart Caching API already ran 100% serverless, the initial intention was to also run the rendering of websites using serverless technologies only. A promising candidate was the Serverless Next.js Component which allows running Next.js in a Lambda@Edge. The component is a port of Next.js and is therefore playing constant catch up with the official version’s feature set. Combined with almost 200 open issues was a dealbreaker.

The next best option was to design a render farm of orchestrated ECS Fargate container instances running Next.js. With this approach, the team was able to implement an auto-scaling mechanism to handle sudden traffic spikes while aiming to keep costs low.

Going live

As part of the migration from on-premise to the AWS Cloud, PCG also took over the hosting and operations management. In preparation for meeting the deadline of May 2022, it was decided that traffic was to be shifted in a responsible fashion. Multiple beta versions of the mobile apps were released long before the official launch. The website was published beforehand under a not publicly known URL in order to conduct preliminary testing. Shortly before the launch, web traffic was slowly shifted from the old t-online.de Wetter website to the new website.

With the help of PCG the high-traffic legacy website t-online Wetter was successfully modernized, making use of cloud native capabilities and enabling future innovation to come. The combination of AWS serverless services, a smart caching strategy in the backend and incremental site generation with Next.js in the frontend resulted in a cost-efficient, scalable and SEO optimized solution. On May 3rd of 2022, the official relaunch of t-online.de was announced; with a modernized cloud native Wetter website to fulfill the promise of “allowing us to work faster and you to load the website much faster”.

Public Cloud Group (PCG) supports companies in their digital transformation through the use of public cloud solutions.

With a product portfolio designed to accompany organisations of all sizes in their cloud journey and competence that is a synonym for highly qualified staff that clients and partners like to work with, PCG is positioned as a reliable and trustworthy partner for the hyperscalers, relevant and with repeatedly validated competence and credibility.

We have the highest partnership status with the three relevant hyperscalers: Amazon Web Services (AWS), Google, and Microsoft. As experienced providers, we advise our customers independently with cloud implementation, application development, and managed services.