In today's consumer-driven economy where customers require services delivered to them on-demand, slow or non-responsive, business-critical applications not only affect customer experience and decline in customer satisfaction but also cause a deterioration of business performance and can lead to a loss in revenue.

To increase customer satisfaction, improve end-user experience, reduce downtime, and increase revenue, engineering teams invest time and resources into designing a high performant software and the infrastructure on which the software run.

Application Performance Monitoring (APM) solutions like Amazon Cloud Watch can also be deployed to help engineers identify and fix stability and performance issues.

In this post, we will look at a challenge to improve the performance of a python application organized by David Thomas of A Cloud Guru.

Database

Let's begin by provisioning a database for our Python application. As per the challenge requirement, we will be using Amazon Relational Database Service (RDS) Postgresql. Amazon RDS makes it easy to set up, operate, and scale a relational databases in the cloud.

We will provision our database using AWS Cloudformation, a great tool used to model infrastructure using code. Download the scripts from GitHub: https://github.com/PeterOcansey/aws-iac-elastic-cache-app-performance

From the terminal, execute bash bash_scripts/create.sh 'your_stack_name_here' iac-rds-pgsql.yml iac-rds-params.json

The new stack will perform the following:

• Set up our database security group
• Set up a security group to allow our application server access our database instance
• Set up our database instance, create our database and database user
• Output our database endpoint, database port and application server security group id for us

Application

Next is to provision Amazon EC2 instance and deploy our Python application. The Python application code can be downloaded from Github: https://github.com/ACloudGuru/elastic-cache-challenge

Just like our database instance, we will provision our application server using Cloudformation. From our scripts repo above, search for iac-app-server.yml and iac-app-server-params.json.

From the terminal, execute bash bash_scripts/create.sh 'your_stack_name_here' iac-app-server.yml iac-app-server-params.json

The new stack will perform the following:

• Provision an EC2 instance, the Amazon Machine Image used is running Ubuntu
• Import our application server security group id created from our database script
• Install Nginx as our web server
• Configure Nginx as proxy server to our python instance running
• Clone the Python application from GitHub https://github.com/ACloudGuru/elastic-cache-challenge
• Install python3, python-configparser, python3-virtualenv, redis, flask, psycopg2-binary, postgresql-client-12 modules
• Update the application database.ini file with the HostAddress, DatabaseName, UserName, and Password
• Connect to our database instance and install the stored procedure
• Activate our Python virtual environment
• Start our Python application
• Output the public URL to our application server

Let's grab our URL and take application for a test ride:

Four spins and application still takes over 5 seconds to load. That is enough to put off any user and make them exit our application. Let's see how we can improve our application performance using Amazon ElastiCache so we can retain our users.

Cache

We need a Cache mechanism that our application will talk to for data while using our database as a fallback source for data. Amazon ElastiCache does a great job, it improves the performance of applications by retrieving information from managed in-memory caches instead of relying entirely on slower disk-based databases.

Just like our previous services, we will provision our ElastiCache Redis cluster using Cloudformation. From our script repo above, search for iac-elastic-cache.yml and iac-elastic-cache-params.json.

From the terminal, execute bash bash_scripts/create.sh 'your_stack_name_here' iac-elastic-cache.yml iac-elastic-cache-params.json

The new stack will perform the following:

• Set up our ElastiCache Security Group
• Spin up our ElastiCache cluster using redis as the engine
• Import and attach our Application Server Security Group to allow EC2 to communicate with ElastiCache
• Output our Redis ElastiCache endpoint and port

We need to update our app to use our ElastiCache now;

• Connect to the application server
• Update the database.ini file by appending [redis] and set redis_url to the Redis endpoint generated from the output console
• Our Python app always fetches from the database. With the introduction of elasticache, we need to update our Python app to first fetch from our elasticache and then fall back to our database if no data is returned. From our scripts repo above, I have an updated version of the app.py file as redisapp.py.
• Restart our python app

Let's take our updated app for a ride once more;

With the help of Amazon ElastiCache, we have a high performant app that responds quickly to users request.

Why the challenge

I took advantage of the challenge to learn more about the tech stacks outlined in the challenge and to get more hands on experience off my AWS Certified Solutions Architect - Associate (SAA-C02) course.

Some takeaways

There is so much I learnt from using Cloudformation to provision the services, sharing resources among stacks and working with UserData to make an EC2 instance readily available for use.

I must say, with my years of experience designing and developing software solutions, I have not used Python before, this challenge pushed me to get up to speed with the Python language and the Flask framework.

All aspects of the challenge is quite challenging, including writing this blog and I commend David Thomas and A Cloud Guru. for putting up hands-on monthly challenges like this one and I look forward to working on more challenges.

Keep being awesome gurus!