IP Authentication: A Tale of Performance Pitfalls and Challenges in Prod

They cached 72 million IPs to fix a performance bug, but the system kept failing. The real root cause was far more subtle.

#1about 1 minute

The case against IP-based authentication for microservices

Migrating a PHP monolith to microservices revealed the fundamental problems with using IP-based authentication, a method you should avoid.

#2about 6 minutes

Understanding the mechanics and tradeoffs of IP authentication

IP authentication simplifies user access for large institutions by mapping an IP address to a user, but it creates issues with user tracking, licensing, and technical complexity.

#3about 3 minutes

Investigating initial deployment failures and performance issues

The first two production deployments failed due to massive error spikes, leading to an architectural change from a slow MySQL database lookup to a faster Redis cache.

#4about 2 minutes

Adopting a fail forward strategy to debug in production

After repeated failures, the team decided to stop rolling back and instead collaborated directly with customers in screen-sharing sessions to diagnose the issue live.

#5about 2 minutes

Discovering the root cause of the infinite redirect loop

The core problem was that legacy and new AWS accounts read the user's IP from different headers, causing a mismatch that sent cookieless bots into an infinite redirect loop.

#6about 2 minutes

How proxies and data entry errors break assumptions

The investigation revealed that academic and corporate proxies often rewrite URLs and strip cookies, and a simple data entry error in the CRM granted access to 1.5% of the internet.

#7about 1 minute

Lessons in collaborative problem-solving and architecture evolution

Using mob programming sessions during the crisis strengthened the team, and the final architecture evolved from Lambda to Fargate for cost and performance reasons.