Containerizing My Webapp Project Using Docker Compose: Taiwan Covid19 Dashboard
In this article, I am going to show you how I transform my Webapp into movable containerized docker compose file.
Introduction
Before the containerization, I host my webapp on a VPS, but it’s a bit costly as data continued to grow. Hence I searched for a way to exhibit my webapp without having to host it in internet. As a result, docker containerization became a great solution.
Architecture
The full architecture of this project is stored within an independent Github repository. You may check here for more detail. The idea of Apache Airflow comes from official docker-compose file, this project is modified on the basis of this official docker-compose file.
Docker_Compose File
The docker compose file defines container images required for the application. It includes Mysql for database, Django for backend and Apache Airflow for workflow management and data processing. The Apache Airflow part should contain initiation for the first run and at least scheduler and webserver for the subsequent runs.
Detailed information and image template will be illustrated within specific DockerFile for each service mentioned in docker-compose yaml.
DockerFile
- Mysql
We use latest mysql image and copy several sql file that we may use later to the mounted volume and finally expose the application of Mysql 3306.
FROM mysql:latest
COPY source.sql ./source.sql
COPY cases.sql ./cases.sql
COPY vacc.sql ./vacc.sql
COPY suspects.sql ./suspects.sql
EXPOSE 3306- Django
Django container was created from Python container with required packages installed. The dockerfile also copies repository code to mounted volume and run migrate commands to migrate the backend to our present application.
FROM python:3.8.3-slim
RUN apt-get update && apt-get install -y default-libmysqlclient-dev && apt-get install -y gcc && apt-get install -y vim
WORKDIR /Covid19_Django_Webapp
COPY Covid19_Django_Webapp .
COPY django_requirements.txt requirements.txt
RUN pip install -r requirements.txt
EXPOSE 8000
CMD python manage.py migrate;python manage.py runserver 0.0.0.0:8000- Airflow-Init
This will be running only one time since it’s an initialization step. By running airflow db init, Airflow will initialize metadata database depending on the backend database designated in airflow.cfg configuration file or AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable.
- Airflow-Scheduler
This service runs Airflow scheduler. Scheduler in Aiflow handles triggering of dags and submission of task to executors to further run tasks in workers.
- Airflow-Webserver
This service runs webserver for Apache Airflow so that users can interact with Airflow through web interface GUI. Compared to command line interface, GUI provides more intuitive and convenient visualized interface to users. You can also manage connection, xcom, visualize tasks and dags as well as administration on Airflow web GUI.
The three Airflow service are all using same template image derived from Python-slim after installing required images.
Repository Code
There are also codes from previous project repository to be copied into the container volume so that the services can access and run as if we built the application from scatch within the container.
The first repository code is Covid19_Django_Webapp which is the backend system for the webapp to visualize our results. Django will migrate the backend as it starts the container. You may need to restart the container after Airflow database is loaded with correct data.
SQL Files
There are some SQL files in the repository. Most of them are for manual initialization for the database when you do not want to wait for time-consuming real initialization using dag. There is one specific mysql_setting file that is used to create and grant user privilege, set max packet and create initial database with correct character set.
Steps to Clone the Webapp
Detailed steps can be found on the Github README file in project repository. Here I only briefly introduce steps.
Git clone or download the project
Simply use download function in Github and unzip the file or run following command.
git clone https://github.com/BurgerWu/Covid19_Taiwan_Dashboard_Docker_Compose.gitInstall Docker as well as Docker-Compose.
You can simply install it from official site and make sure it covers docker-compose as well. Otherwise, you will need to install docker-compose independently to run docker-compose file.
Run docker-compose command
Direct to working directory or specify the full path to which this project was downloaded. Run the following command. This will create required images according to yaml file specified and up the containers.docker-compose -f path_to_docker_compose_yaml up
There are some additional tricks at this step. Please refer to Github Repo of this project for more detail.
Create mysql_conn connection in Apache airflow webpage
Run localhost:8080 on your browser because the webserver container expose this port according to configuration. You should be able to see Airflow web GUI at this moment. Create mysql connection in the Admin connections that will be used in later dag operation.
Initiate covid19 databases
Here I provide two ways to initiate the databases.
- Use Airflow Web GUI to run initiate database dag. However, this could take pretty a while.
2. Run source command in Mysql container to manual load data into Mysql database. Then you just have to run update dags in the future.
Restart Django Container
By the moment Django container was initially up, there is no data in the target database thus Django container will show error. After initializing database in Airflow, you can now restart Django container, and it should be able to visualize the data with our simple website.
Open Website by Using localhost:8000 in Browser
We have configured the Django exposed port to 8000. Thus by entering localhost:8000, you should be able to see the website visualizaing Covid19 data of Taiwan.
Conclusion
Here I successfully migrate my project from VPS in the cloud to docker-compose. Although this could work as expected, there are still things to get improved and resolved. I will keep on working on it and update as soon it gets new version.
