Caching is a cop-out Cache stores such as Memcached and Redis are often used to speed up slow database queries or avoid recomputing expensive HTTP responses. Caching is critical for scalability, but it should not become a crutch. It should be introduced only after the underlying performance problem has been understood and other reasonable optimizations have been considered. I think running a cache for every encountered slow query is a cop-out. I believe understanding the cause of performance degradation is key; otherwise, the cache is just a duct-tape. If it is a database query, look at the plan. Do you need an index? Are there a lot of logical reads? Can you rewrite the query or include an additional filter to predict to minimize the search space? If it is an RPC, consider why the calls are chatty in the first place, and can you eliminate the calls at the client side? Oftentimes, libraries and frameworks can issue a fleet of queries when misused. Another reason black-boxes must be understood. After running out of tuning options, a cache might be needed. I emphasis in my fundamentals courses the importance of understanding the root cause, whether a database, backend end point or a networking protocol. If you want to elevate your skill set, check the courses on my website.

Connection pools help limit unbounded number of requests. But watch out for bad defaults. When a frontend connects to a reverse proxy (load balancer, api gateway, CDN) and sends a request, the reverse proxy turns around and establishes a connection to the origin backend (if one wasn't available) and forwards the request to the backend on that connection. The challenge is how do you handle a large fleet of requests from all the frontends? That easily can overwhelm the backend or results on connection resets error if a backend accept queue gets filled up. One common solution is to implement a connection pool, which is basically a queue with a limit. Requests get queued to the pool with a timeout and get processed by a free connection. But even with a connection pool if the defaults are set too high, you can still run into these issues without realizing. Finding the good number is balance based on workloads and protocols used.