[FIXED] cloudbuild.yaml build pipeline for python flask app on gcloud compute engine

Issue

I have a Python Flask app which needs a library which is not preinstalled on Gcloud compute instances libsndfile1 — also why I can't run this through app engine. I have set up a cloud build pipeline so my cloudbuild.yaml triggers every time I push to master on my repository.

Here are my 2 questions:

1. Where does the cloudbuild.yaml execute it's instructions?
2. How do I make the cloudbuild.yaml execute apt-get install libsndfile1 python3 python3-pip and then also execute pip3 install requirements.txt where requirements.txt is a file in the repository that triggers the cloud build?

Disclaimer: I am an ultra-beginner with docker but if you think that docker is the only way to do it then please explain how something like this can be done.

Solution

If you are interested in App Engine, my recommendation would be to migrate your application to the Google App Engine Flexible environment, where you could choose a Custom runtime for your Flask application. Simply include a step on your Dockerfile to install the required library. It should look similar to this:

FROM python:3.7
WORKDIR /app
COPY . /app
RUN apt-get update &&\
    apt-get install -y libsndfile1
RUN pip install -r requirements.txt
EXPOSE 8080
CMD ["gunicorn", "main:app", "-b", ":8080", "--timeout", "300"]

And within this context you could use Cloud Build to automate your deployments in a fairly easy manner by following the relevant section of the documentation.

Now regarding your questions.

(1) Where does the cloudbuild.yaml execute it's instructions?

Here you can find all the relevant information of how Cloud Build works here and to answer your first question Cloud Build is a separate service and runs each step of the build in a Docker container within Google's Infrastructure.

(2) How do I make the cloudbuild.yaml execute apt-get install libsndfile1 python3 python3-pip and then also execute pip3 install requirements.txt where requirements.txt is a file in the repo that triggers the cloud build.

Please notice that Cloud Build is more oriented to be used with other products like Cloud Run, GKE, Cloud Functions, etc. And the only fairly straightforward that I've seen Cloud Build being used in the context of Compute Engine would be to build VM images using Packer. In which case you'll need to create a custom image with all the dependencies already installed along within your application.

The specifics of the solution will vary depending on the OS of your Compute Engine instance. But you'll basically need to do the following:

1. Upload a startup script that installs libsndfile1 as well as all the dependencies specified on the requirements.txt file to a bucket in Cloud Storage.
2. Take advantage that you can use gcloud to apply a startup script to running instances in a similar fashion to.

gcloud compute instances add-metadata example-instance \
    --metadata startup-script-url=gs://bucket/file

and there is cloud builder for gcloud 3. Build a cloudbuild.yaml file that uses 2. and restart the instance (not recommended if your app is in production since it will cause downtime) in a similar fashion to:

cloudbuild.yaml

steps:
- name: gcr.io/cloud-builders/gcloud
  args: ['compute', 'instances', 'add-metadata', 'example-instance','--metadata startup-script-url=gs://bucket/file']
- name: gcr.io/cloud-builders/gcloud
  args: ['compute', 'instances', 'reset', 'example-instance']

4. Give the cloud build service account enough permissions to execute that task (Compute Admin and Storage Admin).
5. Run the build.

I shared the previous steps in aims of answering your concrete question but I would insist that my recommendation would be to migrate the app to App Engine Flexible environment with custom runtime as explained before.

Answered By - Daniel Ocando