blog.johanv.org (Berichten over ci)

Running apache and php-fpm as services in a gitlab-ci job

Johan Vervloet — Thu, 16 Jun 2022 13:30:00 GMT

I've been using gitlab-ci to automate the end-2-end tests for my PHP-applications for several years now, but I wasn't really happy about the way I got it to work: it involved injecting IP-addresses into configuration files, and starting a web server as a part of the test job.

Today I can run apache and php-fpm as services, so that the job's script doesn't have to care about the web server, and can fully concentrate on running the actual tests.

Lees verder… (2 min resterende leestijd )

Selenium web tests on gitlab-ci for an app using Mercure

Johan Vervloet — Fri, 08 May 2020 20:15:00 GMT

I recently learned how to use Mercure so that the php/Symfony back-end of my application can send out events that can be picked up by the javascript on my front end. This is really cool technology, you should definitely try this out if you haven't already done so.

Now if you write this kind of software, of course you also want to write some web tests, so that you can automatically check if existing features of your program keep working after you add new things.

For these kind of tests, I use gitlab-ci and codeception, as I wrote in previous posts. Today I got these test working with a Mercure hub.

You need to use a mercure service in the job running your web tests, which you can configure in .gitlab-ci.yml:

Update 2020-12-03: I use the 0.10 version of the mercure container, because the acceptance test job started failing with 0.11, with these error messages:

Message: "Couldn't connect to server for "http://mercure:3000/.well-known/mercure"." In HandleMessageMiddleware.php line 80:

Couldn't connect to server for "http://mercure:3000/.well-known/mercure".)

So here's the adapted .gitlab-ci.yml-file:

variables:
  JWT_KEY: YourSecretKey
  DEMO: 1
  ALLOW_ANONYMOUS: 1
  HEARTBEAT_INTERVAL: 30s
  ADDR: ':3000'
  CORS_ALLOWED_ORIGINS: '*'
  PUBLISH_ALLOWED_ORIGINS: '*'

# ...

acceptance tests:
  # I would prefer to use a public php-7.3 image with mysqli, but
  # I don't think such a thing exists.
  # So let's reuse the image I created for the dikdikdik web tests.
  image: registry.gitlab.com/rva-vzw/dikdikdik.php
  services:
    - name: mariadb:10.4
      alias: db
    - name: selenium/standalone-chrome
      alias: chrome
    - name: dunglas/mercure
      alias: mercure:v0.10
  # (the rest of the job follows here)

The JWT_KEY should work with the JWT your backend uses to publish events. The port number in ADDR as well. If you use Symfony, as I do, the corresponding configuration of your backend should be in the .env file, e.g.

MERCURE_PUBLISH_URL=http://mercure:3000/.well-known/mercure
MERCURE_JWT_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJtZXJjdXJlIjp7InB1Ymxpc2giOltdfX0.FFSjymJCGRDWrmAmPJDoVGuYwnx5FRTjRFkkYfvLkUg

(Needless to say: don't use the production JWT and token in the source code that is used by gitlab! If you're unsure how to create a JWT, see the section ‘Creating your JWT’ of this excellent blog post.)

Update 2020-12-03: That blog post doesn't exist anymore, but I found another page that covers mercure and the JWT token.

Now the problem I ran into, is similar to the general problem I had with web tests: the chrome container doesn't know where the mercure hub is; the host name ‘mercure’ is not resolved to an ip address.

Update 2022-06-20: The solution described below will still work, but I found a better one. You should just set the FF_NETWORK_PER_BUILD variable to 1 in your .gitlab-ci.yml file, as described in this post. If you want to see how this works for a real life project, you can check out the .gitlab-ci.yml file of dikdikdik, an app to keep track of the scores when playing solo whist.

To fix this, I figure out the IP address of the Mercure hub when I start the job, and I replace the host name of the MERCURE_PUBLISH_URL in the .env file by this IP-address. This works, since my frontend retrieves the publish url from the backend by a rest-call.

This is how that looks like:

  script:
    - MERCURE_IP=$(getent hosts mercure | awk '{ print $1 }')
    - echo Mercure IP $MERCURE_IP
    - sed -i "s/mercure:3000/$MERCURE_IP:3000/" .env

You can review the complete job definition in the .gitlab-ci.yml file in the wdebelek repository. The file acceptance/TrickCest.php contains an example test involving server sent events. And hopefully you can see in the pipelines that all tests passed without problems 🤓.

gitlab-ci and yarn

Johan Vervloet — Sat, 29 Jun 2019 19:02:12 GMT

For my previous post, on gitlab-ci and codeception, I created a small project on gitlab that runs selenium webtests during gitlab-ci. The project being tested is just a small vue.js-application.

Recently I rewrote that small application; now it uses a single file vue component. I'm not completely sure what I've done, but it seems to work, and it involves webpack encore and yarn.

Now the challenge was to run yarn from gitlab-ci. I was looking for a docker container with yarn (even tried to build one myself), but it took me some time before I discovered that yarn is included in the node-container.

So I now use this node container in an extra task of the build stage in my .gitlab-ci.yaml. I create an artifact with the generated javascript, which is used in the test stage. That seems to work.

As I said before, I am new to vue, and it's the first time I use yarn and webpack. So maybe I am making some obvious mistakes. Please let me know if there are things that can be done in a better way.

gitlab-ci, codeception and selenium web tests

Johan Vervloet — Thu, 16 May 2019 05:27:33 GMT

All the php projects I work on, have tests. For some projects, we use PHPUnit as testing framework, but for more recent projects, I prefer to use Codeception, because it has an elegant way to write acceptance tests. We also use gitlab for our projects, and by adding a nifty .gitlab-ci.yml file to our source code, we made gitlab run our test suite every time new commits are pushed to the repository.

Recently, we started using Selenium WebDriver with codeception, so that we could run browser tests, to make sure that the javascripts we use keep working. Of course we wanted to run these browser tests with gitlab-ci as well, but that was not as easy as we thought it would be. The idea was to run selenium with a browser via a docker container, but the challenging part was setting up a webserver the browser could connect to.

Update 2022-06-17: In the mean time, I found out how to run an apache server as a gitlab job service, so the rest of this post has become kind of obsolete. I updated the demo project, you should check the .gitlab-ci.yml file, the Docker configuration for the apache container, and (of course 😉) the README to find out how it all works.

In a first attempt, I tried to start all the containers needed for our application, running docker-compose up from the ci like we do it for local development. This worked sometimes, but after a couple of builds the docker networks used by our gitlab runners got completely messed up, which was hard to fix.

In a second attempt, I tried to use Gitlab CI Services for all the containers in our docker-compose.yml file. I gave up before I completely configured that, because I noticed the networking problems wouldn't go away.

After days of swearing, I finally got it running. I think the networking problems were caused by the nginx-container I tried to use for the web tests. But actually I didn't need a container for nginx. It turned out I could just juse the php built-in development server to run the tests.

So I ended up by adding a service for selenium with chrome and a service for mysql (the project I was trying this on, didn't really need other services), and I just added a php -S statement to the script that ran the tests, I had a running server.

I still needed a small hack, to make the browser in the selenium container actually find the webserver. I ended up using a sed-command on the configuration file of my test suite in codeception.

For reference, here is a part of my gitlab-ci.yml file:

    - LOCAL_IP=$(ip route get 8.8.8.8 | awk -F"src " 'NR==1{split($2,a," ");print a[1]}')
    - echo Local IP $LOCAL_IP
    # Run the php built in server. Don't try to run nginx as a service, that only causes troubles. ;-)
    - php -S 0.0.0.0:8080 -t public/ 2> /dev/null &
    - sed -i "s/localhost:8080/$LOCAL_IP:8080/" tests/acceptance.suite.yml

Note that for this to work, I had to fiddle with the php container to have iproute2 installed.

If you want to see how it actutally works; I created a small demo project on gitlab that uses codeception in gitlab-ci to run a couple of browser tests. You can find the source code on gitlab: https://gitlab.com/johanv/codeception-ci-demo.

Van github naar gitlab

Johan Vervloet — Sat, 11 May 2019 08:16:15 GMT

Mijn website is een statische website. Dat wil zeggen dat de hele site automatisch wordt gegenereerd op basis van een aantal tekstbestanden. En die tekstbestanden beheer ik met git.

Gisteren verhuisde ik de git-repository met die teksten van github naar gitlab. Niet omdat ik problemen heb met het bedrijf achter github (want Microsoft is tegenwoordig érg goed bezig), maar wel omdat ik gitlab-ci wou gebruiken om mijn site te bouwen. Concreet komt het erop neer dat, telkens ik een nieuwe versie van de source push, gitlab voor mij mijn site automatisch bouwt en deployt.

Misschien heeft github ook wel zo'n service. Maar die van gitlab heb ik nu al vaak gebruikt, en ik ben er erg tevreden over. Dus vanaf nu wordt mijn blog gepowerd door nikola en gitlab. Waarvoor dank.