Case Study

HDFC Bank

Throttling incoming traffic requests without changing the core banking system

Challenge

Digital banking services in India have become increasingly popular, but many core banking applications are still developed in-house, based on legacy technologies using monolithic architectures. This is exacerbated by the rise in popularity of the Unified Payments Interface (UPI) system in India, a payment transactions system that facilitates both peer-to-peer and peer-to-merchant transactions.

FYNDNA is an Indian software startup with the vision of enabling financial institutions to increase their customer engagement and profitability by developing cloud-native, digitally-enabled, versatile technologies. To improve customer experience and handle the increasing volumes of transaction, FYNDNA determined that banks needed a way to regulate, or throttle, the incoming traffic requests, without changing the core banking system.

Solution

Microservice based applications deployed on Kubernetes have become the de-facto standard for developing highly-resilient, continuously available, secure cloud native applications. And to bring about agility in adopting new and emerging technologies, and to address the technology obsolescence that large application architectures tend to suffer with time, FYNDA decided to explore alternative solutions, ultimately deciding on Dapr.

One of the applications leveraging the core FYNDNA foundational framework and Dapr is called Governor. Development began in early 2022 and the system went live after one year, becoming one of the fastest applications to be developed and deployed to a Tier One Bank – HDFC Bank – to overcome a myriad of challenges in handling UPI transactions. 

Impact

HDFC Bank handles close to 750 million transactions/month however the UPI transaction rate varies throughout the day. Dapr metrics along with KEDA have been leveraged to scale out horizontally based on the incoming HTTP traffic allowing the number of replicas of Governor to scale up and down without any issues.

With Governor running in production, the number of UPI transactions timing out has decreased significantly. As a result, this has provided a smoother user experience and established deeper customer trust in the bank.

Challenges:

Reliability, Scaling, Security

Industry:

Financial Services

Location:

India

Published:

August 31, 2023

Projects used

Throttling incoming traffic requests without changing the core banking system

FYNDNA is an Indian software startup with the vision of enabling financial institutions to increase their customer engagement and profitability by developing cloud-native, digitally-enabled, versatile technologies.

Rapid adoption of digital banking services in India and the rise of FinTechs has meant that banks have to match rising customer expectations. It is crucial to avoid service disruptions, as even the smallest downtime erodes customer trust and reduces usage.

While many non-critical applications have been outsourced, core banking applications are still developed in-house and are based on legacy technologies using monolithic architectures. This makes them difficult to change in order to scale to meet the needs of a growing business. 

This is exacerbated by the rise in popularity of the Unified Payments Interface (UPI) system in India, a payment transactions system that facilitates both peer-to-peer and peer-to-merchant transactions. It has seen a dramatic  increase in usage in the past few years and is expected to continue to grow linearly. UPI’s ever-growing customer base and fragile core legacy systems has made it increasingly difficult to come up with a solution.

This is a consistent pattern across banks and from their wealth of experience working in the financial industry, FYNDNA determined that banks needed a way to regulate, or throttle, the incoming traffic requests without changing the core banking system.

Customer Expectations

Increased customer expectations mean that any modern day banking application is expected to meet the following at the very minimum:

• Fast response time and continuous availability.

• Highly secure system with the ability to detect fraudulent activity.

• The ability to scale linearly with ever increasing HTTP traffic from various fintech applications and mobile front-ends.

Technical Challenges

While the above considerations are the key expectations from the business, they pose complex challenges in terms of creating a system architecture which meets all of the business needs while keeping the overall implementation cost under control. Some non-trivial technical challenges faced in building such applications are:

• Autoscaling applications based on HTTP traffic. 

• Application resiliency and graceful handling of disruptions.

• A secure, zero-trust-environment.

• A vendor-agnostic solution not including any 3rd-party technology stacks or managed services.

• Backpressure handled with asynchronous communication.

• Configurable policies to define service level access.

• The ability to connect to various data stores and security services.

• Secure communication between services, even within the same cluster.

• Support for the three pillars of observability: metrics, logs and traces. 

• A framework that addresses cross-cutting concerns across multiple services deployed in the cluster ( abstraction, security, observability to name a few )

Solution

Microservice based applications deployed on Kubernetes have become the de-facto standard for developing highly-resilient, continuously available, secure cloud native applications. While there are many frameworks available to develop such systems, FYNDNA was careful in choosing the framework as they wanted to have an abstraction of cross-cutting concerns from the core business logic but needed to consider operational and maintenance aspects as well. Addressing these cross cutting concerns at the network as well as application levels with the core foundation framework was the top priority while building the foundational services. A sidecar pattern is one of the prime architectural patterns used for this purpose, however just abstracting the network concerns with a sidecar did not solve our problem as there were many application level cross-cutting concerns to be addressed as well. For this reason FYNDNA looked for something which addressed both network level and application concerns and had a minimal footprint.

Traditionally FYNDNA’s approach had been to own and develop the foundational framework. However, to bring about agility in adopting new and emerging technologies, and to address the technology obsolescence that large application architectures tend to suffer with time, they decided to explore 3^rd party solutions, ultimately deciding on Dapr.

Dapr APIs to build microservices

Dapr runs as a sidecar alongside applications in Kubernetes and provides a set of APIs codifying industry best practices for microservices development. It takes away complex challenges pertaining to service discovery, transport security, authorization, message broker integration, encryption, observability, and secret management from the application developer allowing them to focus on the business domain. With support for reactive programming, Dapr also helps to handle back pressure in any application making it a very appealing addition to event-driven systems. Dapr provides a non-intrusive way to address cross cutting concerns unlike some other libraries available that also address microservices challenges.

The adoption of Dapr as a core piece of FYNDNA’s architecture helped in addressing some of the key challenges listed below:

1. Secure service-to-service communication with mTLS using the Dapr Sentry service removes the overhead of managing mTLS certificates.
   
2. Dapr support for both HTTP and gRPC calls removes the need for apps to understand multiple protocols.
   
3. Asynchronous messaging using the Dapr Pub/Sub API abstracts away the complexities of dealing with the Kafka API and provides a message broker-agnostic method for event-driven communication allowing an easy way to switch out Kafka for any other messaging system.
   
4. The Dapr secrets API provides a platform-agnostic abstraction on top of secret stores allowing implementation changes from, for instance, GCP Secret Manager to HashiCorp Vault with negligible effort.
   
5. Resiliency configuration using the Dapr resiliency spec makes the system resilient to cluster or service disruptions using only a simple declarative configuration file.
   
6. External service integration with Dapr HTTP Bindings provides security and resiliency to external communications.
   
7. Out-of-the-box observability with OpenTelemetry provides vendor-agnostic logs, metrics and traces and does not take any additional time investment.

Additional Dapr Considerations

As well as the benefits provided by the Dapr APIs, there were other determining factors in its choice. 

Cost and Time

The adoption of Dapr reduced the time to build the core foundational framework of the system and allowed for an earlier start on developing the business logic. Likewise, overall QA efforts were also reduced, reducing the overall cost of the project.

Latency

Every HTTP/gRPC request is intercepted by a Dapr sidecar adding minimal overhead latency. In extensive stress testing, this value was found to be between 1-2 milliseconds which is negligible in the system.

Governor Application for transaction rate limiting

One of the applications leveraging the core FYNDNA foundational framework and Dapr is called Governor. Development began in early 2022 and the system went live after one year, becoming one of the fastest applications to be developed and deployed to a Tier One Bank. Dapr played a crucial role in achieving this feat.

Transaction Rate Limiting

The issues described in the challenges are faced by banks across India and FYNDNA worked with one of the largest private sector banks HDFC Bank to overcome a myriad of challenges in handling UPI transactions. The Governor application regulates the UPI transaction traffic to the core banking system based on select health parameters. 

Governor leverages Dapr heavily, accelerating the overall development time to production and has allowed the production deployment to be a hassle-free experience. The system is currently running in production and scales up and down based on HTTP traffic. Dapr metrics, along with a KEDA ScaledObject,are used for scaling and the system has been deployed on Google Cloud using Google Kubernetes Engine (GKE). 

This is shown in the diagram below.

Kubernetes Architecture

The Governor application has three containers; the AppInterceptor container which connects to the core banking system, the Analyser container which is the heart of the application and the Dapr sidecar. Analyser calculates the rate at which HTTP traffic should be sent to the core banking system based on a number of parameters. It scrapes the metrics collected in AppInterceptor at 250 ms intervals and sends a signal to AppInterceptor if the request rate to the core banking system has to be modified. For cases where core banking requests need to be rate-limited, the messages are stored in the local queue of AppInterceptor.

Numbers and Scalability 

The bank handles close to 750 million transactions/month, however the UPI transaction rate varies throughout the day. Dapr metrics along with KEDA have been leveraged to scale out horizontally based on the incoming HTTP traffic allowing the number of replicas of Governor to scale up and down without any issues.

Business Impact

With Governor running in production, the number of UPI transactions timing out has decreased significantly. As a result, this has provided a smoother user experience and established deeper customer trust in the bank.