Who We Are

We are Optimum BH - A cutting edge software development agency specializing in Full-stack development with a focus on web and mobile applications built on top of PETAL stack.

What We Do

At Optimum BH, we are dedicated to pushing the boundaries of software development, delivering solutions that empower businesses to thrive in the digital landscape.

Web app development

We create dynamic and user-friendly web applications tailored to meet your specific needs and objectives.

Mobile app development

We design and develop mobile applications that captivate users, delivering an unparalleled experience across iOS and Android platforms.

Maintenance and support

Our commitment doesn't end with deployment. We provide ongoing maintenance and support to ensure your applications remain up-to-date, secure, and optimized for peak performance.

Blog Articles

Post Image

Zero downtime deployments with Fly.io

If you were wondering why you saw the topbar loading for ~5 seconds every time you deployed to Fly.io, you’re at the right place. We need to talk about deployment strategies. Typically, there are several, but Fly.io supports these: immediate rolling bluegreen canary   The complexity and cost go from low to high as we go down the list. The default option is rolling. That means, your machines will be replaced by new ones one by one. In case you only have one machine, it will be destroyed before there’s a new one that can handle requests. That’s why you’re waiting to be reconnected whenever you deploy. You can read more about these deployment strategies at https://fly.io/docs/apps/deploy/#deployment-strategy.   We’re using the blue-green deployment strategy as it strikes a balance between the benefits, cost, and ease of setup.   If you’re using volumes, I have to disappoint you as the blue-green strategy doesn’t work with them yet, but Fly.io plans to support that in the future.   Setup   You need to configure at least one health check to use the bluegreen strategy. I won’t go into details. You can find more at https://fly.io/docs/reference/configuration/#http_service-checks.   Here’s a configuration we use:   [[http_service.checks]] grace_period = &quot10s&quot interval = &quot30s&quot method = &quotGET&quot path = &quot/health&quot timeout = &quot5s&quot   Then, add strategy = “bluegreen” under [deploy] in your fly.toml file: [deploy] strategy = &quotbluegreen&quot   and run fly deploy.   That’s it! You probably expected the setup to be more complex than this. So did I!   Conclusion   While Fly.io is moving you from a blue to a green machine, your websocket connection will be dropped, but it will quickly reestablish. You shouldn’t even notice it unless you have your browser console open or you’re navigating through pages during the deployment.   One thing you should keep in mind, though, is that your client-side state (form data) might be lost if you don’t address that explicitly.   Another thing to think about is the way you run Ecto migrations. In case you’re dropping tables or columns, you might want to do that in multiple stages. For example, you might introduce changes in the code so you stop depending on specific columns or tables and deploy that change. After that, you can have subsequent deployment for the structural changes of the database. That way, both blue and green machines will have the same expectations regarding the database structure.   The future will bring us more options for deployment. Recently, Chris McCord teased us with hot deploys.   https://x.com/chris_mccord/status/1785678249424461897   Can’t wait for this!   This was a post from our Elixir DevOps series.
Almir Sarajčić
Post Image

Feature preview (PR review) apps on Fly.io

In this blog post, I explain how we approach manual testing of new features at Optimum.   Collaborating on new features with non-developers often requires sharing our progress with them. We can do quick demos in our dev environment, but if we want to let them play around on their own, we need to provide them with an environment facilitating that. Setting up a dev machine is easy thanks to phx.tools: Complete Development Environment for Elixir and Phoenix, but pulling updates in our projects still requires basic git knowledge.   We could solve this by deploying in-progress stuff to the staging server, but that becomes messy in larger teams, so we stay away from that. Instead, we replicate the production environment for each feature we are working on and we only deploy the main branch with finished features to the staging. With an environment created specifically for the feature we are working on, we can be sure nothing will surprise us after shipping it. Automated tests help with that too, but we still like doing manual checks just before deploying to production.   Heroku review apps Back when I was working on Ruby on Rails apps and websites, as many, I chose Heroku as my PaaS (platform-as-a-service). It had (and still has) a great feature called Review apps. App deployment pipeline on Heroku   It enables you to create new Heroku environments in your pipeline for the PRs in your GitHub repo, either manually through their UI or automatically using a configuration file. You can configure the dynos, environment variables, addons… any prerequisite for running your application. This was a great experience when I worked with Ruby, but when I moved to Elixir Heroku didn’t fit me anymore, so I moved to Fly.io.   Fly.io PR review apps Fly.io introduced something similar using GitHub Actions: https://github.com/superfly/fly-pr-review-apps. It’s not as powerful and is not as user-friendly, but it’s a good starting point for building your workflows. Here’s the official guide: https://fly.io/docs/blueprints/review-apps-guide/.   The current version forces you to share databases and volumes between different PR review apps. We didn’t want that, so last year my colleague Amos introduced a fork that solves this, accompanied by the blog post How to Automate Creating and Destroying Pull Request Review Phoenix Applications on Fly.io. We’ve also added some minor changes there. Some of them were implemented upstream since then, yet the setup for the database and volume is still missing. Here’s the diff: https://github.com/superfly/fly-pr-review-apps/compare/6f79ec3a7d017082ed11e7c464dae298ca75b21b...optimumBA:fly-preview-apps:b03f97a38e6a6189d683fad73b0249c321f3ef4a.   Examples We use preview apps for our phx.tools website. Although it doesn’t use DB and volumes, it’s still a good example of setting preview apps up on Fly.io: https://github.com/optimumBA/phx.tools/blob/main/.github/github_workflows.ex.   Here’s the code responsible for preview apps: @app_name &quotphx-tools&quot @environment_name &quotpr-${{ github.event.number }}&quot @preview_app_name &quot#{@app_name}-#{@environment_name}&quot @preview_app_host &quot#{@preview_app_name}.fly.dev&quot @repo_name &quotphx_tools&quot defp pr_workflow do [ [ name: &quotPR&quot, on: [ pull_request: [ branches: [&quotmain&quot], types: [&quotopened&quot, &quotreopened&quot, &quotsynchronize&quot] ] ], jobs: elixir_ci_jobs() ++ [ deploy_preview_app: deploy_preview_app_job() ] ] ] end defp pr_closure_workflow do [ [ name: &quotPR closure&quot, on: [ pull_request: [ branches: [&quotmain&quot], types: [&quotclosed&quot] ] ], jobs: [ delete_preview_app: delete_preview_app_job() ] ] ] end defp delete_preview_app_job do [ name: &quotDelete preview app&quot, &quotruns-on&quot: &quotubuntu-latest&quot, concurrency: [group: &quotpr-${{ github.event.number }}&quot], steps: [ checkout_step(), [ name: &quotDelete preview app&quot, uses: &quotoptimumBA/fly-preview-apps@main&quot, env: [ FLY_API_TOKEN: &quot${{ secrets.FLY_API_TOKEN }}&quot, REPO_NAME: @repo_name ], with: [ name: @preview_app_name ] ], [ name: &quotGenerate token&quot, uses: &quotnavikt/github-app-token-generator@v1.1.1&quot, id: &quotgenerate_token&quot, with: [ &quotapp-id&quot: &quot${{ secrets.GH_APP_ID }}&quot, &quotprivate-key&quot: &quot${{ secrets.GH_APP_PRIVATE_KEY }}&quot ] ], [ name: &quotDelete GitHub environment&quot, uses: &quotstrumwolf/delete-deployment-environment@v2.2.3&quot, with: [ token: &quot${{ steps.generate_token.outputs.token }}&quot, environment: @environment_name, ref: &quot${{ github.head_ref }}&quot ] ] ] ] end defp deploy_job(env, opts) do [ name: &quotDeploy #{env} app&quot, needs: [ :compile, :credo, :deps_audit, :dialyzer, :format, :hex_audit, :prettier, :sobelow, :test, :test_linux_script_job, :test_macos_script_job, :unused_deps ], &quotruns-on&quot: &quotubuntu-latest&quot ] ++ opts end defp deploy_preview_app_job do deploy_job(&quotpreview&quot, permissions: &quotwrite-all&quot, concurrency: [group: @environment_name], environment: preview_app_environment(), steps: [ checkout_step(), delete_previous_deployments_step(), [ name: &quotDeploy preview app&quot, uses: &quotoptimumBA/fly-preview-apps@main&quot, env: fly_env(), with: [ name: @preview_app_name, secrets: &quotAPPSIGNAL_APP_ENV=preview APPSIGNAL_PUSH_API_KEY=${{ secrets.APPSIGNAL_PUSH_API_KEY }} PHX_HOST=${{ env.PHX_HOST }} SECRET_KEY_BASE=${{ secrets.SECRET_KEY_BASE }}&quot ] ] ] ) end defp delete_previous_deployments_step do [ name: &quotDelete previous deployments&quot, uses: &quotstrumwolf/delete-deployment-environment@v2.2.3&quot, with: [ token: &quot${{ secrets.GITHUB_TOKEN }}&quot, environment: @environment_name, ref: &quot${{ github.head_ref }}&quot, onlyRemoveDeployments: true ] ] end defp fly_env do [ FLY_API_TOKEN: &quot${{ secrets.FLY_API_TOKEN }}&quot, FLY_ORG: &quotoptimum-bh&quot, FLY_REGION: &quotfra&quot, PHX_HOST: &quot#{@preview_app_name}.fly.dev&quot, REPO_NAME: @repo_name ] end defp preview_app_environment do [ name: @environment_name, url: &quothttps://#{@preview_app_host}&quot ] end   If you’re wondering why you’re seeing Elixir while working with GitHub Actions you should read our blog post on the subject: Maintaining GitHub Actions workflows.   Let’s explain what we’re doing above. We are running the pr_workflow when PR is (re)opened or when any new changes are pushed to it. It runs our code checks and tests, and, if everything passes, runs the deploy_preview_app_job.   GitHub Actions workflow for PRs   The deploy_preview_app_job uses action for deploying preview apps to Fly.io which checks if the server is already set up. If it isn’t, it creates the server, sets environment variables, etc. Then it deploys to it.   Preview app creation job that includes a DB and/or a volume doesn’t differ from the one above at all. That’s because our action optimumBA/fly-preview-apps internally checks whether an app contains Ecto migrations and if it does, it creates a DB if it doesn’t exist yet. The same goes for the volume: it checks whether the fly.toml configuration contains any mounts and if it does, it creates a volume, then attaches it to the app.   GitHub workflow for the website you’re on   Preview app for one of the PRs   We set environment to let GitHub show the preview app in the list of environments. It will show the status of the latest deployment in the PR. We don’t want too much noise in our PR from the deployment messages, so whenever we deploy a new version, we remove previous messages in the delete_previous_deployments_step.   List of deployments on GitHub   Deployment status message in the PR   Setting concurrency makes sure that two deployment jobs can’t run simultaneously for the same value passed to it. That prevents hypothetical race condition with multiple pushes, where for some reason deployment job for the latest commit could finish more quickly than the one for the previous commit, which would leave us with an older version of the app running.   Don’t forget to set GitHub secrets like FLY_API_TOKEN. You might want to do that on the organization level so you don’t have to do that for every repo. The token we’ve set in our GitHub organization is for a Fly.io user we’ve created specifically for deployments to staging and preview apps. We have a separate Fly.io organization it is a part of, so even if the token gets leaked, our production apps are safe as it doesn’t have access to them.   When we’re done working on a feature, we want to clean up our environment. It might seem strange that we use the same action to delete our app, but the action handles it by checking the event that triggered the workflow and acts accordingly. It destroys any associated volume and/or database, then the server. The next two steps of the delete_preview_app_job delete a GitHub environment. For some reason known to GitHub, the process is more complicated than it should be, but Amos explains it well in his blog post.   Getting back to the part about databases. Recently, the upstream version of the action was updated with an option to attach an existing PostgreSQL instance from Fly.io, but that still doesn’t solve potential issues with migrations. Let’s say you remove a table in one PR, while another PR depends on the same table. It will be deleted while deploying the first PR which will in turn cause errors for the second PR’s review app. Our solution avoids that by creating a completely isolated environment for each PR.   Additionally, Fly.io recently introduced (or we’ve just discovered) the ability to stop servers after some time of inactivity. That proved useful for us in lowering the cost when having many PRs open. In your fly.toml you probably want to set [http_service] auto_start_machines = true auto_stop_machines = true min_machines_running = 0   so your machines stop if you don’t access them for some period. We haven’t found a way to stop DBs for inactive apps yet. We weren’t eager to do so, though, because we’ve always had the smallest instances for the preview apps DBs. Only our apps sometimes have larger instances which incur greater costs, so we see a benefit in stopping them when we don’t use them.   More customization Some applications might require setting up additional resources. In the StoryDeck app, one of the services we use is Mux.   When a user uploads a video, we upload it to Mux, which sends us events to our webhooks. Whenever we create a new preview app, we need to let Mux know the URL of our new webhook. In theory, this could be solved by a simple proxy. In reality, it’s more complicated than that. We don’t want all our preview apps to receive an event when a video is uploaded from any of them. To know which preview app to proxy an event to, the proxy app would need to store associations between specific videos and preview apps they were uploaded from, but we don’t want to store that kind of data in the proxy app. Mux enables having many different environments in one account, which is perfect for us as each environment is a container for videos uploaded from one preview app. What is not perfect is the fact that currently there’s no API for managing Mux environments, so we have to do it through the Mux dashboard. We’ve built the proxy app using Phoenix. It has a simple API on which we receive requests sent from GitHub Actions using curl. When a new preview app is created, a request is received, then the app goes through the Mux dashboard using Wallaby, creates a new Mux environment, sets up the webhook URL, gets Mux secrets, and returns it so that the GitHub Actions workflow can set them in our new Fly.io environment. When deleting the preview app, our workflow sends a request to our proxy app which then deletes videos from Mux and deletes the Mux environment.   Creating Mux environment and saving credentials in GitHub Actions cache   That is just one example of what it might take to enable preview apps in your organization. It could seem like unnecessary work, but think of it as an investment into higher productivity and quality of work down the line.   This was a post from our Elixir DevOps series.
Almir Sarajčić
Post Image

Testing Elixir releases in CI

Have you ever deployed your app and called it a day only to find out later that in production some NIF was missing or a 3rd party application wasn’t started? No, that never happens to you because you always run your app locally with MIX_ENV=prod before deploying, right? Right?   Last year I worked on a project with a really conscientious team. We were working on an umbrella project consisting of 6 apps, and a colleague of mine always made sure to build a release for each of them in the prod environment and run it manually on his machine, then deploy it worry-free. It took some time to do that and it was a boring process, so to help him out, I automated the process by building a release and running it for each app in CI. Then, if everything passed, we’d proceed with the deployment.   At Optimum, we usually strive to deploy preview apps assuring us that the app is successfully built and running on a Fly.io server. Sometimes we have a different setup, for which we may build a release as part of our CI.   I’m going to show you how it works in a sample Phoenix app that will execute some code from ExUnit which will be missing in production. The code is available in the repo: https://github.com/almirsarajcic/testing_release.   Failing example Let’s generate a new Phoenix app that consists of only an API endpoint. mix phx.new testing_release --adapter bandit --no-assets --no-ecto --no-esbuild --no-gettext --no-html --no-live --no-mailer --no-tailwind   Create a controller in a new file lib/testing_release_web/controllers/home_controller.ex: defmodule TestingReleaseWeb.HomeController do use TestingReleaseWeb, :controller def index(conn, _params) do ExUnit.__info__(:functions) |&gt IO.inspect(label: &quotExUnit functions&quot) json(conn, %{status: &quotok&quot}) end end   add the route scope &quot/api&quot, TestingReleaseWeb do pipe_through :api get &quot/&quot, HomeController, :index end   You can see the full commit here: https://github.com/almirsarajcic/testing_release/commit/378f30c124ca7814e43b8685e57cf350caceb20d.   After setting that up, I can launch a Fly.io server: fly launch --generate-name --vm-memory 256   after it’s been deployed I can visit URL https://old-wave-7774.fly.dev/api and get the following response: {&quoterrors&quot:{&quotdetail&quot:&quotInternal Server Error&quot}}   Executing fly logs shows: [error] ** (UndefinedFunctionError) function ExUnit.__info__/1 is undefined (module ExUnit is not available) ExUnit.__info__(:functions) (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:5: TestingReleaseWeb.HomeController.index/2 (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.action/2 (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.phoenix_controller_pipeline/2 (phoenix 1.7.12) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5 (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.plug_builder_call/2 (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.call/2 (bandit 1.5.0) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2   but we won’t fix the error yet. Let’s reproduce it in CI.   Automating release process We can use the same Dockerfile generated for us while running fly launch to start a container in GitHub Actions and then send an HTTP request to verify it works as expected. We’ll be using a script github_workflows_generator we introduced in the blog post Maintaining GitHub Actions workflows to write the workflow in Elixir.   Here we’ll focus only on the steps of the workflow, but you can see the full commit on the following link: https://github.com/almirsarajcic/testing_release/commit/65def71031669cf7fcde77918f9b073e95bca4d3.   steps: [ [ name: &quotCheckout&quot, uses: &quotactions/checkout@v3&quot ], [ name: &quotSet up Docker Buildx&quot, uses: &quotdocker/setup-buildx-action@v1&quot ], [ name: &quotCache Docker layers&quot, uses: &quotactions/cache@v3&quot, with: [ path: &quot/tmp/.buildx-cache&quot, key: &quot${{ runner.os }}-buildx-${{ github.sha }}&quot, &quotrestore-keys&quot: &quot${{ runner.os }}-buildx&quot ] ], [ name: &quotBuild image&quot, uses: &quotdocker/build-push-action@v2&quot, with: [ context: &quot.&quot, builder: &quot${{ steps.buildx.outputs.name }}&quot, tags: &quottesting_release:latest&quot, load: true, &quotbuild-args&quot: &quottarget=testing_release&quot, &quotcache-from&quot: &quottype=local,src=/tmp/.buildx-cache&quot, &quotcache-to&quot: &quottype=local,dest=/tmp/.buildx-cache-new,mode=max&quot ] ], [ # Temp fix # https://github.com/docker/build-push-action/issues/252 # https://github.com/moby/buildkit/issues/1896 name: &quotMove cache&quot, run: &quotrm -rf /tmp/.buildx-cache\nmv /tmp/.buildx-cache-new /tmp/.buildx-cache&quot ], [ name: &quotCreate the container&quot, id: &quotcreate_container&quot, run: &quotecho ::set-output name=container_id::$(docker create -p 4000:4000 -e FLY_APP_NAME=${{ env.FLY_APP_NAME }} -e FLY_PRIVATE_IP=${{ env.FLY_PRIVATE_IP }} -e PHX_HOST=${{ env.PHX_HOST }} -e SECRET_KEY_BASE=${{ env.SECRET_KEY_BASE }} testing_release | tail -1)&quot ], [ name: &quotStart the container&quot, run: &quotdocker start ${{ steps.create_container.outputs.container_id }}&quot ], [ name: &quotCheck HTTP status code&quot, uses: &quotnick-fields/retry@v2&quot, with: [ command: &quotINPUT_SITES='[\&quothttp://localhost:4000/api\&quot]' INPUT_EXPECTED='[200]' ./scripts/check_status_code.sh&quot, max_attempts: 3, retry_wait_seconds: 5, timeout_seconds: 1 ] ], [ name: &quotWrite Docker logs to a file&quot, if: &quotfailure() &amp&amp steps.create_container.outcome == 'success'&quot, run: &quotdocker logs ${{ steps.create_container.outputs.container_id }} &gt&gt docker.log&quot ], [ name: &quotUpload Docker log file&quot, if: &quotfailure()&quot, uses: &quotactions/upload-artifact@v3&quot, with: [ name: &quotdocker.log&quot, path: &quotdocker.log&quot ] ] ]   Most of the steps are related to setting up Docker for caching intermediary images so that subsequent runs are quicker, but these steps are the most important: [ name: &quotStart the container&quot, run: &quotdocker start ${{ steps.create_container.outputs.container_id }}&quot ], [ name: &quotCheck HTTP status code&quot, uses: &quotnick-fields/retry@v2&quot, with: [ command: &quotINPUT_SITES='[\&quothttp://localhost:4000/api\&quot]' INPUT_EXPECTED='[200]' ./scripts/check_status_code.sh&quot, max_attempts: 3, retry_wait_seconds: 5, timeout_seconds: 1 ] ]   After building the image and creating the container, we start it, and then send an HTTP request to it. We’re not sure when the server is ready, so we use nick-fields/retry action to retry sending the request with a configurable number of maximum attempts. To send a request we use a convenient script scripts/check_status_code.sh I copied from https://github.com/lakuapik/gh-actions-http-status.   In the end, we upload a log as an artifact so we can inspect it in case the request fails.   There’s an additional change we have to make to enable running the release outside of Fly.io environment. In the file rel/env.sh.eex replace the line export ERL_AFLAGS=&quot-proto_dist inet6_tcp&quot   with if [[ -z &quot${FLY_PRIVATE_IP}&quot ]]; then export ERL_AFLAGS=&quot-proto_dist inet6_tcp&quot fi   After pushing the code to GitHub, you can see requests failing. Run nick-fields/retry@v2 step   Checking the log file saved as an artifact docker.log artifact   shows the following error messages: 15:31:02.527 [info] Running TestingReleaseWeb.Endpoint with Bandit 1.5.0 at :::4000 (http) 15:31:02.528 [info] Access TestingReleaseWeb.Endpoint at https://localhost 15:31:07.848 request_id=F8rJ9kxkVangNu4AAAAE [info] GET /api 15:31:07.848 request_id=F8rJ9kxkVangNu4AAAAE [info] Sent 500 in 251µs 15:31:07.849 request_id=F8rJ9kxkVangNu4AAAAE [error] ** (UndefinedFunctionError) function ExUnit.__info__/1 is undefined (module ExUnit is not available) ExUnit.__info__(:functions) (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:5: TestingReleaseWeb.HomeController.index/2 (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.action/2 (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.phoenix_controller_pipeline/2 (phoenix 1.7.12) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5 (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.plug_builder_call/2 (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.call/2 (bandit 1.5.0) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2 15:31:13.856 request_id=F8rJ97J1J9nc0xoAAAAB [info] GET /api 15:31:13.856 request_id=F8rJ97J1J9nc0xoAAAAB [info] Sent 500 in 243µs 15:31:13.856 request_id=F8rJ97J1J9nc0xoAAAAB [error] ** (UndefinedFunctionError) function ExUnit.__info__/1 is undefined (module ExUnit is not available) ExUnit.__info__(:functions) (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:5: TestingReleaseWeb.HomeController.index/2 (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.action/2 (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.phoenix_controller_pipeline/2 (phoenix 1.7.12) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5 (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.plug_builder_call/2 (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.call/2 (bandit 1.5.0) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2   The fix for this is simple: change the :extra_applications in the mix.exs file from [:logger, :runtime_tools]   to [:ex_unit, :logger, :runtime_tools]   (https://github.com/almirsarajcic/testing_release/commit/534615eb5dcf0cc19c166971cdeeb23f4ad49708)   After pushing the code, we verify it works. main.yml workflow result   You can also use a database by setting up the Docker container to use the network from the DB service, Redis, and whatnot. Possibilities are vast.   This was a post from our Elixir DevOps series.
Almir Sarajčić
Post Image

Optimum Elixir CI with GitHub Actions

Here’s yet another “ultimate Elixir CI” blog post. We haven’t had one for quite some time. But on a more serious note we have some unique ideas, so continue reading and I’m sure you’ll get some inspiration for your development workflows.   When a post like this comes out, I check it out to see if I can learn about a new tool to use in pursuit of higher code quality, but the thing I get most excited about is reducing the time it takes to get that CI checkmark for my or someone else’s PR. Unfortunately, mostly I realize it’s a twist to an older approach with everything else pretty much the same. I have yet to see one offering a different caching solution. Usually, it’s the same approach presented in the GitHub Actions docs. I saw some downside in their workflows which I’ll explain below, but ours won’t be spared of criticism either. As with anything, the goal is to find the balance, and as our name suggests, we strive to create optimum solutions, so here’s one on us.   A quick reminder: even though here we use GitHub Actions, the principles are also applicable to other types of CIs.   But first, what’s a CI? This article is about a software development practice. For other uses, see Informant. (🙄 I rewatched The Wire recently)   During the development of new features, there comes a time when the developer submits the code for review. At one stage of the code review process, project maintainers want to make sure that the newly written code doesn’t introduce any regressions to existing features. That is, unless they blindly trust the phrase “it works on my machine”. Then, if they are satisfied with the code quality they can merge the pull request and potentially proceed with a release process if there is a continuous delivery (CD) system in place.   CI (continuous integration) system automates the testing process, enabling everyone involved to see which commit introduced a regression early in the workflow, before the reviewer even starts the review process. It frees the project maintainer from having to run the tests (either manually or using an automated system) on their machine, conserving their energy to focus on other aspects of code quality and the business domain. Machines are better at those boring, repetitive tasks, anyway. Let them have it, so they don’t start the uprise.   Crosses and checkmarks show whether the CI passed for the particular commit   Now, if you don’t write and run tests in your Elixir applications, you probably have bigger issues to worry about. So make sure to handle that before going further.   Old approach If you’re just starting to build your CI, you might not be interested in this part and can jump straight to the New approach.   The old approach consists of having all the checks as steps of one job of GitHub Actions workflow. That means commands for the code checks are running one after the other. For example, you might be running the formatter, then dialyzer, and finally, tests.   The good thing about this approach is that the code gets compiled once and then used for each of these steps. You have to make sure, though, that the commands are running in the test environment, either by prefixing the command with MIX_ENV=test or by setting the :preferred_cli_env option to ensure compilation is done only in one environment, otherwise you’d unnecessarily compile in both dev and test environments.   The bad thing is that if one of the commands fails, at that moment you don’t know yet whether the subsequent commands will fail also. So, you might fix the formatting and push the code only to find out minutes later that the tests failed too. Then you have to fix them and repeat the process.   The other bad thing is the caching of dependencies. To understand why, you need to know how the caching works in GitHub Actions. You can learn about that in the official documentation, but here’s the gist of it.   When setting up caching, you provide a key to be used for saving and restoring it. Once saved, cache with specified key cannot be updated. It gets purged after 7 days if it’s not used, or if the total cache size goes above the limit. But you shouldn’t rely on that. The key doesn’t have to match exactly, though. You have an option of using multiple restore keys for partial matching.   Here’s an example from the documentation: - name: Cache node modules uses: actions/cache@v3 with: path: ~/.npm key: npm-${{ hashFiles('**/package-lock.json') }} restore-keys: | npm-   The thing is, that might work for JS community where each run of npm install command causes the lock file to change, making frequent cache updates.   More importantly, when using Elixir, we don’t only want to cache dependencies (deps directory), but also the compiled code (_build). When our application code changes, the cache key isn’t updated, meaning, as time goes by, there will be more changed files that will need to be compiled, making the CI slower and slower. For an active repo, the cache will never get purged, so the only way to reduce the number of files to be compiled each time is to update the lock file, or manually change the cache key, none of which is ideal. Theoretically, the cache might never be refreshed, but in practice, you would probably do an update of dependencies every few months. But still, you need to unnecessarily wait for all the files that were changed since the cache was created to (re)compile.   The issue is multiplied if you extract each command into its own job to enable running them in parallel, but without improving to the caching strategy. That will cause each command to compile all the files in the app that were changed since the cache was created, which for big codebases can be too much, unnecessarily increasing the cost of CI. Not only that, it’s hard to maintain those workflows because GitHub Actions doesn’t have a good mechanism for the reuse of jobs and steps. You can learn how to deal with that in [Maintaining GitHub Actions workflows]().   It’s important to mention that workflows can only access the cache that was created in that branch or in the parent branch. So, if you have a PR open and update the cache there, don’t expect that other PRs will be able to access that cache until that one gets merged. And even then, if you don’t create a cache in the main branch, it won’t be available to other branches. So even if you don’t want to run code checks in the main branch, you should at least cache the dependencies as part of the CI workflow. I saw some examples of CIs that didn’t cache dependencies on the main branch which means caching didn’t exist the first time PRs were created - only when it was synced.   Another example of inadequate setup is not using restore keys and matching on the complete cache key. That forces the whole app including all the dependencies to be recompiled every time the lock file changes.   Workflow running the old way   Run and billable time of the old approach New approach I won’t go too much into explaining what we do. One Look is Worth A Thousand Words. Workflow running the new way   The work is parallelized so the time waiting for the CI is shortened. Compiling is done only once in a job, and then cached for use by all the other jobs. Jobs that don’t depend on the cache run independently. Every job is running in the test environment to prevent triggering unnecessary compilation. It’s possible to see from the list of commits which check has failed.   Checks running separately   Those were the benefits. Now let’s talk about the detriments of this approach:   It’s using too much cache. There’s a 10 GB limit in GitHub Actions, and the old cache is automatically evicted. So, that doesn’t worry me much. Issues could arise from using cache instead of running a fresh build in CI. The old approach is susceptible to this as well, but I guess this one is more because it provides better caching 😁What we could do to improve this is to disable using cache on retries. Or we could manually delete the cache from the GitHub Actions UI. We didn’t need either of those yet.   Cache management under the Actions tab   It’s more expensive. The workflow running this way uses more runner minutes. You’d expect it’s because of the containers being set up, but GitHub doesn’t bill us for the time it takes to set up their environment. Thanks, GitHub! They get us the other way, though: when rounding minutes, they are ceiling, and that’s what makes all the difference. Even if the job finishes in 10 seconds, it’s billed as a whole minute, so if you have 10 steps that are each running in 10 to 30 seconds, you’ll be billed 10 minutes even though the whole workflow might have been completed in one job running under 5 minutes. You can see that most of our jobs are running for less than half a minute, but we get billed for the whole minute. In our projects, we still go under the quota, so it wasn’t a concert for us, but it’s something to be aware of. If you use a macOS runner and/or have a pretty active codebase, you will notice the greater cost.   Run and billable time of the new approach   Now that we have cleared that, let’s see some code.   We solved the caching part by using git commit hash as the key and using a restore key that enables restoring cache, while still creating a new one every time the workflow runs.   [ uses: &quotactions/cache@v3&quot, with: [ key: &quotmix-${{ github.sha }}&quot, path: ~S&quot&quot&quot _build deps &quot&quot&quot, &quotrestore-keys&quot: ~S&quot&quot&quot mix- &quot&quot&quot ] ]   You can verify by looking at the logs.   For the caching step, this would show something like this: Cache restored successfully Cache restored from key: mix-4c9ce406f9b55bdfa535dac34c1a9dbb347dd803   but it would still show this in the post-job cache step: Cache saved successfully Cache saved with key: mix-83cb8d66280ccf99207c202da7c6f51dfc43fa38   Our solution for the jobs parallelization is harder to show: defp pr_workflow do [ [ name: &quotPR&quot, on: [ pull_request: [ branches: [&quotmain&quot], ] ], jobs: [ compile: compile_job(), credo: credo_job(), deps_audit: deps_audit_job(), dialyzer: dialyzer_job(), format: format_job(), hex_audit: hex_audit_job(), migrations: migrations_job(), prettier: prettier_job(), sobelow: sobelow_job(), test: test_job(), unused_deps: unused_deps_job() ] ] ] end defp compile_job do elixir_job(&quotInstall deps and compile&quot, steps: [ [ name: &quotInstall Elixir dependencies&quot, env: [MIX_ENV: &quottest&quot], run: &quotmix deps.get&quot ], [ name: &quotCompile&quot, env: [MIX_ENV: &quottest&quot], run: &quotmix compile&quot ] ] ) end defp credo_job do elixir_job(&quotCredo&quot, needs: :compile, steps: [ [ name: &quotCheck code style&quot, env: [MIX_ENV: &quottest&quot], run: &quotmix credo --strict&quot ] ] ) end   Another benefit of splitting the workflow into multiple jobs is that the cache is still written even if some of the checks fail. Before, everything would still need to be recompiled (and PLT files for dialyzer created - I know, I know, I’ve been there) every time the workflow runs after failing. It could be solved another way by saving the cache immediately after compiling the code and then running the checks in the same job. Just saying.   But hold on a minute. Are we writing our GitHub Actions workflows in Elixir?! That can’t be right… It’s not magic, it’s a script we wrote to maintain GitHub Actions more easily.   Full example of a complex workflow we made for our phx.tools project is available here: https://github.com/optimumBA/phx.tools/blob/main/.github/github_workflows.ex, and here you can see it in action(s): https://github.com/optimumBA/phx.tools/actions.   Running the checks locally We don’t rely only on GitHub Actions for the code checks. Usually, just before committing the code, we run the checks locally. That way we find errors more quickly and don’t unnecessarily waste our GitHub Actions minutes.   To execute them all one after the other, we run a convenient mix ci command. It’s an alias we add to our apps that locally runs the same commands that are run in GitHub Actions.   defp aliases do [ ... ci: [ &quotdeps.unlock --check-unused&quot, &quotdeps.audit&quot, &quothex.audit&quot, &quotsobelow --config .sobelow-conf&quot, &quotformat --check-formatted&quot, &quotcmd --cd assets npx prettier -c ..&quot, &quotcredo --strict&quot, &quotdialyzer&quot, &quotecto.create --quiet&quot, &quotecto.migrate --quiet&quot, &quottest --cover --warnings-as-errors&quot, ] ... ]   When one of these commands fails, we run it again in isolation and try fixing it while rerunning the command until the issue is fixed. Then we run mix ci again until every command passes.   To run each of these commands without having to prefix with MIX_ENV=test, you can pass the :preferred_cli_env option to the project/0:   def project do [ ... preferred_cli_env: [ ci: :test, coveralls: :test, &quotcoveralls.detail&quot: :test, &quotcoveralls.html&quot: :test, credo: :test, dialyzer: :test, sobelow: :test ], ... ] end   Again, the reason why I run these commands in the test environment is that the app is already compiled in that environment and if I need to run it in the dev environment, it will start the compilation. Locally, it doesn’t matter much, but in GitHub Actions, as you’d expect, it makes a huge difference.   Usually in our projects, we also like to check whether all migrations can be rolled back. To achieve that, we run the command mix ecto.rollback --all --quiet after these. Unfortunately, it doesn’t work if it’s added to the end of this list because when the command is run the app is still connected to the DB causing it to fail. Don’t worry, there’s a tool that can help us, and it’s available in any Unix system. Yes, it’s Make. Create a Makefile in the root of your project with the following content:   ci: mix ci MIX_ENV=test mix ecto.rollback --all --quiet   and run make ci. We could put all the commands there instead of creating a mix alias, something like:   ci.slow: mix deps.unlock --check-unused mix deps.audit mix hex.audit mix sobelow --config .sobelow-conf mix format --check-formatted mix cmd --cd assets npx prettier -c .. mix credo --strict mix dialyzer MIX_ENV=test ecto.create --quiet MIX_ENV=test ecto.migrate --quiet test --cover --warnings-as-errors MIX_ENV=test mix ecto.rollback --all --quiet   but I prefer doing it as a mix alias as it performs more quickly.   See for yourself: $ time make ci … make ci 18.88s user 3.39s system 177% cpu 12.509 total $ time make ci.slow make ci.slow 22.08s user 4.92s system 157% cpu 17.180 total   Almost 5 seconds difference. I suspect it’s because the app is booted only once, unlike when running make ci.slow with each mix ... command booting the app again. Now it makes sense why the rollback step didn’t work when it was a part of the ci alias.   Need help? You’re probably reading this because you’re just starting to build your CI pipeline, or maybe you’re looking for ways to make the existing one better. In any case, we’re confident we can find ways to improve your overall development experience.   We’ve done more complex pipelines for our clients and in our internal projects. These include creating additional resources during the preview apps setup, running production Docker container build as a CI step, using self-hosted runners, etc. We can create a custom solution suitable to your needs.   Whether you’re just testing out your idea with a proof-of-concept (PoC), building a minimum viable product (MVP), or want us to extend and refactor your app that’s already serving your customers, I’m sure we can help you out. You can reach us at projects@optimum.ba.   This was a post from our Elixir DevOps series.
Almir Sarajčić
Post Image

Elixir DevOps series

Everyone who’s dipped their toes in Elixir knows its ecosystem has the best-in-class documentation, and learning resources for beginners are vast. There are so many blog posts about various domains Elixir is used in, including, but not limited to machine learning, embedded systems, web applications, etc.   One area we felt didn’t receive much love is DevOps. Specifically, complex continuous integration (CI) and continuous delivery (CD) systems. So here we are, coming up with a remedy. We’re trying to shine some light on some non-trivial problems to share knowledge with the community we learned a lot from, while, simultaneously increasing the visibility of our company in the field.   Blog posts So here they are, in the order they were published - not necessarily in the order they should be read.   Maintaining GitHub Actions workflows Optimum Elixir CI with GitHub Actions Testing Elixir releases in CI Feature preview (PR review) apps on Fly.io Zero downtime deployments with Fly.io   The list is not final and we might add more posts in the future.   Is there some topic that you’d like us to cover in this series? Feel free to reach us at blog@optimum.ba.
Almir Sarajčić
Post Image

Maintaining GitHub Actions workflows

Whenever we have a choice to make for a CI system to use on a project, we pick GitHub Actions, mainly for convenience. Our code is already hosted on GitHub, and it doesn’t make sense to introduce other tools unnecessarily, so some time ago we started using GitHub Actions as our CI provider.   Over the years we’ve been constantly improving our development workflows thereby adding more complexity to our CI. There were many steps in our pipeline for various code checks, Elixir tests, etc., each increasing the time we needed to wait to make sure our code was good to go. So we’d wait 5 to 10 minutes or so just to find out the code wasn’t formatted properly, there was some compiler warning or something trivial as that. We knew there were better ways to set up our CI, but we felt separating the workflows into separate jobs was going to make for a harder-to-maintain code because GitHub Actions does not support full YAML syntax.   I came to Elixir from the Ruby on Rails community where YAML is a default for any kind of configuration, so I was excited to see GitHub Actions using YAML for the workflow definitions. I quickly came to realize it’s not the same YAML I was used to (You’ve changed, bro). Specifically, I couldn’t use anchors which provide the ability to write reusable code in .yml files.   Script Our way of working around this is writing workflow definitions in Elixir and translating them to YAML, letting us benefit from the beautiful Elixir syntax in sharing variables, steps, and jobs between workflows while still, as a result, having workflow files in the YAML format GitHub Actions supports.   To convert the workflow definitions from Elixir to YAML, we wrote a CLI script that uses fast_yaml library with a small amount of code wrapping it up in an easy-to-use package. We used this script internally for years, but now we’ve decided to share it with the community.   I’ve had some trouble distributing the script. Usually, we’d execute .exs script to convert the workflow, so I wanted to build an escript, but the fast_yaml library contains NIFs that don’t work with it. I liked the way Phoenix is installed so I tried adopting that approach, creating a mix project containing a task, then archiving the project into a .ez file, only to find out that when it gets installed, it doesn’t contain any dependencies. This can be alleviated using burrito or bakeware, but they introduce more complexity, and I didn’t like the way error messages were displayed in the Terminal, so I ended up with a hex package that’s added to an Elixir project in a usual way. Ultimately, I didn’t plan to use the script outside of Elixir projects, so that’s a compromise I was willing to make. If at a later point I feel the need, I’ll distribute it some other way, which will deserve another blog post.   Usage Anyway, here’s how you can use this mix task. Add the github_workflows_generator package as a dependency to your mix.exs file: defp deps do [ {:github_workflows_generator, &quot~&gt 0.1&quot} ] end   You most likely don’t want to use it in runtime and environments other than dev, so you might find this more appropriate: defp deps do [ {:github_workflows_generator, &quot~&gt 0.1&quot, only: :dev, runtime: false} ] end   That will let you execute mix github_workflows.generate   command that given a .github/github_workflows.ex file like this one: defmodule GithubWorkflows do def get do %{ &quotmain.yml&quot =&gt main_workflow(), &quotpr.yml&quot =&gt pr_workflow() } end defp main_workflow do [ [ name: &quotMain&quot, on: [ push: [ branches: [&quotmain&quot] ] ], jobs: [ test: test_job(), deploy: [ name: &quotDeploy&quot, needs: :test, steps: [ checkout_step(), [ name: &quotDeploy&quot, run: &quotmake deploy&quot ] ] ] ] ] ] end defp pr_workflow do [ [ name: &quotPR&quot, on: [ pull_request: [ branches: [&quotmain&quot] ] ], jobs: [ test: test_job() ] ] ] end defp test_job do [ name: &quotTest&quot, steps: [ checkout_step(), [ name: &quotRun tests&quot, run: &quotmake test&quot ] ] ] end defp checkout_step do [ name: &quotCheckout&quot, uses: &quotactions/checkout@v4&quot ] end end   creates multiple files in the .github/workflows directory.   main.yml name: Main on: push: branches: - main jobs: test: name: Test steps: - name: Checkout uses: actions/checkout@v4 - name: Run tests run: make test deploy: name: Deploy needs: test steps: - name: Checkout uses: actions/checkout@v4 - name: Deploy run: make deploy   pr.yml name: PR on: pull_request: branches: - main jobs: test: name: Test steps: - name: Checkout uses: actions/checkout@v4 - name: Run tests run: make test   Path to the source file and the output directory can be customized. To see available options, run mix help github_workflows.generate   You might also want to read the documentation or check out the source code.   The generator’s repo contains its own CI workflows (something something-ception) that show how useful the command is in complex scenarios.   defmodule GithubWorkflows do @moduledoc false def get do %{ &quotci.yml&quot =&gt ci_workflow() } end defp ci_workflow do [ [ name: &quotCI&quot, on: [ pull_request: [], push: [ branches: [&quotmain&quot] ] ], jobs: [ compile: compile_job(), credo: credo_job(), deps_audit: deps_audit_job(), dialyzer: dialyzer_job(), format: format_job(), hex_audit: hex_audit_job(), prettier: prettier_job(), test: test_job(), unused_deps: unused_deps_job() ] ] ] end defp compile_job do elixir_job(&quotInstall deps and compile&quot, steps: [ [ name: &quotInstall Elixir dependencies&quot, env: [MIX_ENV: &quottest&quot], run: &quotmix deps.get&quot ], [ name: &quotCompile&quot, env: [MIX_ENV: &quottest&quot], run: &quotmix compile&quot ] ] ) end defp credo_job do elixir_job(&quotCredo&quot, needs: :compile, steps: [ [ name: &quotCheck code style&quot, env: [MIX_ENV: &quottest&quot], run: &quotmix credo --strict&quot ] ] ) end # Removed for brevity # ... defp elixir_job(name, opts) do needs = Keyword.get(opts, :needs) steps = Keyword.get(opts, :steps, []) job = [ name: name, &quotruns-on&quot: &quot${{ matrix.versions.runner-image }}&quot, strategy: [ &quotfail-fast&quot: false, matrix: [ versions: [ %{ elixir: &quot1.11&quot, otp: &quot21.3&quot, &quotrunner-image&quot: &quotubuntu-20.04&quot }, %{ elixir: &quot1.16&quot, otp: &quot26.2&quot, &quotrunner-image&quot: &quotubuntu-latest&quot } ] ] ], steps: [ checkout_step(), [ name: &quotSet up Elixir&quot, uses: &quoterlef/setup-beam@v1&quot, with: [ &quotelixir-version&quot: &quot${{ matrix.versions.elixir }}&quot, &quototp-version&quot: &quot${{ matrix.versions.otp }}&quot ] ], [ uses: &quotactions/cache@v3&quot, with: [ path: ~S&quot&quot&quot _build deps &quot&quot&quot ] ++ cache_opts(prefix: &quotmix-${{ matrix.versions.runner-image }}&quot) ] ] ++ steps ] if needs do Keyword.put(job, :needs, needs) else job end end # Removed for brevity # ... end   That creates a YAML file I wouldn’t want to look at, much less maintain it, but enables us to have this CI pipeline CI pipeline with jobs running in parallel   Our phx.tools project has an even better example with 3 different workflows. Workflow executed on push to the main branch   Workflow executed when PR gets created and synchronized   Cleanup workflow when PR gets merged or closed   Let’s step back to see how the script works.   The only rule that we enforce is that the source file must contain a GithubWorkflows module with a get/0 function that returns a map of workflows in which keys are filenames and values are workflow definitions.   defmodule GithubWorkflows do def get do %{ &quotci.yml&quot =&gt [[ name: &quotMain&quot, on: [ push: [] ], jobs: [] ]] } end end   Everything else is up to you.   When you look at the generated .yml files, they might not look exactly the same as if you wrote them by hand.   For example, if you were to add caching for Elixir dependencies as in actions/cache code samples, you’d want to have this YAML code:   - uses: actions/cache@v3 with: path: | deps _build key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }} restore-keys: | ${{ runner.os }}-mix-   with two paths passed without quotes.   I haven’t found a way to tell the YAML encoder to format it that way, so my workaround is to use a sigil that preserves the newline, so that [ uses: &quotactions/cache@v3&quot, with: [ path: ~S&quot&quot&quot _build deps &quot&quot&quot ], key: &quot${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}&quot restore-keys: ~S&quot&quot&quot ${{ runner.os }}-mix- &quot&quot&quot ]   gets converted to   - uses: actions/cache@v3 with: path: &quot_build\ndeps&quot key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }} restore-keys: ${{ runner.os }}-mix-   Elixir DevOps series In our workflows, you may notice some new ideas not seen elsewhere, so be sure to look out for more posts on our blog in a new series where we’ll unpack our unique DevOps practices. If you have any questions, you can contact us at blog@optimum.ba and we’ll try to answer them in our future posts.   If our approach to software development resonates with you and you're ready to kickstart your project, drop us an email at projects@optimum.ba. Share your project requirements and budget, and we'll promptly conduct a review. We'll then schedule a call to dive deeper into your needs. Let's bring your vision to life!   This was the first post from our Elixir DevOps series.
Almir Sarajčić


  • Phx.tools

    Powerful shell script designed for Linux and macOS that simplifies the process of setting up a development environment for Phoenix applications using the Elixir programming language. It configures the environment in just a few easy steps, allowing users to start the database server, create a new Phoenix application, and launch the server seamlessly. The script is particularly useful for new developers who may find the setup process challenging. With Phoenix Tools, the Elixir ecosystem becomes more approachable and accessible, allowing developers to unlock the full potential of the Phoenix and Elixir stack.

  • Prati.ba

    Bosnian news aggregator website that collects and curates news articles from various sources, including local news outlets and international media. The website provides news on a variety of topics, including politics, sports, business, culture, and entertainment, among others.

  • StoryDeck

    StoryDeck is a cloud-based video production tool that offers a range of features for content creators. It allows users to store and archive all their content in one location, track tasks and collaborate easily with team members, and use a multi-use text editor to manage multiple contributors. The platform also offers a timecode video review feature, allowing users to provide precise feedback on video files and a publishing tool with SEO optimization capabilities for traffic-driving content.