Azure Application Gateway with an Internal APIM – A Successful Proof of Concept!

The Lay of the Land

In an effort to keep up with the fierce competition in the customer's industry, the organization was eager to leverage certain Azure Platform as a Service (PaaS) offerings in order to provide end-users with new, interactive features through an existing web client.

However, due to the organization's strict regulations and security constraints, they needed an architecture that is compliant with those constraints while fulfilling the new set of functional requirements.

In this article, I will share my experience in designing the solution. I will also provide some tips while provisioning the solution's workloads in Azure, as I conducted an end-to-end proof of concept for it.

Solution Design

The solution is a hybrid integration model that spans across the organization's on-premises data centers and Azure over a site-to-site virtual private network (VPN) connection. This formed a secure boundary which extended the organization's network edge with Azure workloads through a virtual network (VNET) gateway service.

The App Service component was the new implementation which also leveraged Azure Cognitive Services to build the new interactive, intelligent application that will extend the existing client features.

To further enhance the solution's security posture, I segmented the solution's VNET into a public facing subnet for the application gateway instance, and internal subnets for the internal cloud-based PaaS workloads and the APIM instance for the back-end services. The internal network traffic was further restricted and controlled using network security group (NSG) rules for each subnet.

This way I minimized the attack surface area by isolating the different Azure resources and prevented any chance of having public Internet traffic directly accessing the downstream systems.

Now, I will be mainly focusing on the API Management (APIM) and Application Gateway. I will also briefly mention how message-level security, and monitoring aspects were tackled in this solution.

API Management

As mentioned earlier, in order to reduce the solution's attack surface area, I configured the APIM to be in the internal mode, this way APIM with all its endpoints: the gateway proxy, the APIM portals, and management endpoints were protected within the organization's internal VNET, and could not be directly accessed from any potential threat coming from the public Internet traffic.

The reason I used APIM in the solution was not only to handle API lifecycle management, but also to securely publish and control the required set of the organization's back-end services. Importantly, it allowed for applying the relevant compliance and governance rules for the organization's back-end requests in terms of authentication, throttling, and auditing.

Governance

Azure APIM API endpoints were secured using Azure Active Directory (AAD) as an identity management provider for application-level authentication using OAuth 2.0 authentication scheme.

All you need to do is to register the client and back-end as apps in AAD and grant permissions for client app to the back-end app in AAD client app settings. The AAD will be the authorization server, it has an OAuth 2.0 token endpoint to issue the access tokens for the authenticated clients.

Next, in APIM instance, you will configure APIM security setting to use OAuth 2.0, and finally apply the built-in policy (Validate JWT) to validate the JWT access token passed by client requests against the required back-end resource. You may also need to apply the Cross-Origin Resource Sharing (CORS) policy to allow cross-domain calls to the API from browser-based clients.

Check the following Microsoft Documentation for more details regarding OAuth 2.0 using Azure Active Directory and Azure APIM.

Rate Limit Policy

The back-end system will be protected against denial of service (DOS) and brute force attacks using APIM client-driven throttling inbound policy that was applied on the product level, using a custom client-based identifier. This is in addition to the VNET's native basic protection against distributed denial of service (DDOS).

Testing & Configurations

While I was working in the POC, I created a VM within the same VNET in order to be able to test and configure the APIM after it was configured using the internal mode. This simulated the customer's ability to access to APIM endpoints as part of its internal network.

Application Gateway

Application Gateway is a PaaS which provides Web Application Firewall (WAF) and Layer 7 load balancer capabilities. This acted as the DMZ, the first line defense, which guarded and securely integrated with the internal downstream systems.

In my scenario, it was a perfect fit against the customer's security requirements, as it offered a baseline security protection against exploits based on its built-in Web application firewall (WAF) rules which supported Open Web Application Security Project (OWASP) core rule sets 3.0 or 2.2.9.

I tried both Application Gateway SKUs: (WAF) and (WAF V2). At the time of writing this article, the main issue I faced with WAF tier was the slow performance in provisioning and applying configuration updates. In addition to the initial provisioning time, for every change I made, I had to wait for several minutes for it to be saved, in some cases, changing settings took around 10 minutes!

The (WAF V2) tier has a much better performance. It also includes some other features such as: autoscaling, header rewrite, zone redundancy; just to name a few.

APIM Internal Mode Configurations

In my solution, the Application Gateway was integrated with 2 back-end pools: APIM instance, and an App Service.

There is a built-in HTTP setting in Application Gateway for App Services, that handles everything for you, you just select the designated App Service in the back-end pool configurations and select the (Use for App service) checkbox in the Application Gateway HTTP settings.

However, configuring APIM was a bit tricky because it was configured using the internal mode. When APIM internal mode is used, Azure will no longer handle DNS routing for you.

Moreover, Azure APIM only responds to requests made to its host name, not the IP configured for its service endpoints. In that case, I had to use the IP for the back-end pool configurations while leveraging the (override host name) feature in the Application Gateway HTTP settings.

Another caveat, was to configure a custom health probe for APIM using its special built-in URL path: "/status-0123456789abcdef"

I highly recommend that you go through this Microsoft Documentation for integrating Application Gateway with APIM in an internal VNET. You should also check the following Microsoft Documentation for more details regarding the required DNS configurations.

APIM Whitelisting with Application Gateway

In order to implement an end-to-end SSL using Application Gateway, there are few additional steps you need to do.

Mainly, you need to whitelist the APIM with Application Gateway, otherwise you will get the following error message in your designated probe: "Back-end server certificate is not whitelisted for an application gateway". If the probe is indicating an issue, no traffic will actually be routed to the corresponding back-end, in that case, to the APIM.

To do the whitlisting, you will need to export APIM SSL certificate into a Base-64 encoded (CER) format, and apply the exported certificate in (Backend authentication certificates) under the Application Gateway's HTTP settings configured for the APIM.

The following Microsoft Documentation has all details regarding Application Gateway's end-to-end SSL. And, this documentation will walk you through the steps of exporting back-end certificates. Note that SKU V1 and V2 slightly differ in terms of the exported certificate. In my case, I was using V1.

However, I still got the same error message, even after following all the previous steps, this actually took some time to figure out.

The caveat was that the APIM certificate needed to have the common name (CN) matching the APIM proxy hostname. In my demo, I was using a self-signed certificate and I didn't realize that its CN was set to something else, and that actually caused the error message above to persist.

I changed my APIM hostname using the custom hostname configurations, and applied the self-signed certificate in the APIM settings, and whew!  I finally got a green check mark besides the SSL health probe, and end-to-end SSL finally worked.

Message-Level Security

Because of the organization's regulations, the requests and responses payloads needed to be encrypted during its journey in Azure. This ensured end-to-end encryption of the payloads containing sensitive data in transit and at rest, by avoiding the exposure of the sensitive data to any intermediate component, beyond the client and back-end system. This is in addition to applying end-to-end TLS/SSL across the system components.

Monitoring

Last but definitely not least, I had to take care of monitoring across the different solution components. Azure log analytics provided the operational monitoring features for the native Azure resources that comprise the solution.

In addition, for tracking and debugging purposes, I used Azure Application Insights to provide an application-level logging and monitoring of the client request's status, among other high-level request fields, without saving the actual payload.

Azure Application Insights allows for defining custom events for traced data and it also provides query-based search feature which can be utilized to search for and analyze the traced data gathered. In addition, configured alerts could be leveraged for specific cases or using pre-configured thresholds for the number of encountered failures, for instance.

Final Thoughts

In this article, I shared a hybrid solution design that extends the reach of the organization's network to Azure to take advantage of some of Azure's PaaS offerings in order to add new cool features to end-users. I also demonstrated some of the tips and caveats that I encountered in terms of the provisioning of the 2 main resources: Application Gateway, and APIM during working on the proof of concept.

Security constraints mandated that we also include end-to-end message encryption using the organization's managed keys, in addition to transport level security using end-to-end SSL. This assured the customer that message payloads containing sensitive data are properly secured.

Application-level monitoring provided a proper level of tracking and notification for the business critical requests initiated by the client.

The application gateway is a powerful service offering that can provide a baseline WAF protection and Layer 7 load balancer capabilities, it also gets automatically updated against new exploits.

Adding APIM to the solution allowed for applying all the necessary compliance and governance measures to the organization's on-premises assets.