Testing Elixir releases in CI

By Almir Sarajčić , Software Developer

10 min read

Have you ever deployed your app and called it a day only to find out later that in production some NIF was missing or a 3rd party application wasn't started? No, that never happens to you because you always run your app locally with MIX_ENV=prod before deploying, right? Right?

Last year I worked on a project with a really conscientious team. We were working on an umbrella project consisting of 6 apps, and a colleague of mine always made sure to build a release for each of them in the prod environment and run it manually on his machine, then deploy it worry-free. It took some time to do that and it was a boring process, so to help him out, I automated the process by building a release and running it for each app in CI. Then, if everything passed, we'd proceed with the deployment.

At Optimum, we usually strive to deploy preview apps assuring us that the app is successfully built and running on a Fly.io server. Sometimes we have a different setup, for which we may build a release as part of our CI.

I'm going to show you how it works in a sample Phoenix app that will execute some code from ExUnit which will be missing in production. The code is available in the repo: https://github.com/almirsarajcic/testing_release.

Failing example

Let's generate a new Phoenix app that consists of only an API endpoint.

mix phx.new testing_release --adapter bandit --no-assets --no-ecto --no-esbuild --no-gettext --no-html --no-live --no-mailer --no-tailwind

Create a controller in a new file lib/testing_release_web/controllers/home_controller.ex:

defmodule TestingReleaseWeb.HomeController do
  use TestingReleaseWeb, :controller

  def index(conn, _params) do
    ExUnit.__info__(:functions) |> IO.inspect(label: "ExUnit functions")

    json(conn, %{status: "ok"})
  end
end

Add the route:

scope "/api", TestingReleaseWeb do
  pipe_through :api

  get "/", HomeController, :index
end

You can see the full commit here: https://github.com/almirsarajcic/testing_release/commit/378f30c124ca7814e43b8685e57cf350caceb20d.

After setting that up, I can launch a Fly.io server:

fly launch --generate-name --vm-memory 256

After it's been deployed I can visit URL https://old-wave-7774.fly.dev/api and get the following response:

{"errors":{"detail":"Internal Server Error"}}

Executing fly logs shows:

[error] ** (UndefinedFunctionError) function ExUnit.__info__/1 is undefined (module ExUnit is not available)
ExUnit.__info__(:functions)
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:5: TestingReleaseWeb.HomeController.index/2
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.action/2
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.phoenix_controller_pipeline/2
    (phoenix 1.7.12) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5
    (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.plug_builder_call/2
    (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.call/2
    (bandit 1.5.0) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2

But we won't fix the error yet. Let's reproduce it in CI.

Automating release process

We can use the same Dockerfile generated for us while running fly launch to start a container in GitHub Actions and then send an HTTP request to verify it works as expected. We'll be using a script github_workflows_generator we introduced in the blog post Maintaining GitHub Actions workflows to write the workflow in Elixir.

Here we'll focus only on the steps of the workflow, but you can see the full commit on the following link: https://github.com/almirsarajcic/testing_release/commit/65def71031669cf7fcde77918f9b073e95bca4d3.

steps: [
  [
    name: "Checkout",
    uses: "actions/checkout@v3"
  ],
  [
    name: "Set up Docker Buildx",
    uses: "docker/setup-buildx-action@v1"
  ],
  [
    name: "Cache Docker layers",
    uses: "actions/cache@v3",
    with: [
      path: "/tmp/.buildx-cache",
      key: "${{ runner.os }}-buildx-${{ github.sha }}",
      "restore-keys": "${{ runner.os }}-buildx"
    ]
  ],
  [
    name: "Build image",
    uses: "docker/build-push-action@v2",
    with: [
      context: ".",
      builder: "${{ steps.buildx.outputs.name }}",
      tags: "testing_release:latest",
      load: true,
      "build-args": "target=testing_release",
      "cache-from": "type=local,src=/tmp/.buildx-cache",
      "cache-to": "type=local,dest=/tmp/.buildx-cache-new,mode=max"
    ]
  ],
  [
    # Temp fix
    # https://github.com/docker/build-push-action/issues/252
    # https://github.com/moby/buildkit/issues/1896
    name: "Move cache",
    run: "rm -rf /tmp/.buildx-cache\nmv /tmp/.buildx-cache-new /tmp/.buildx-cache"
  ],
  [
    name: "Create the container",
    id: "create_container",
    run:
      "echo ::set-output name=container_id::$(docker create -p 4000:4000 -e FLY_APP_NAME=${{ env.FLY_APP_NAME }} -e FLY_PRIVATE_IP=${{ env.FLY_PRIVATE_IP }} -e PHX_HOST=${{ env.PHX_HOST }} -e SECRET_KEY_BASE=${{ env.SECRET_KEY_BASE }} testing_release | tail -1)"
  ],
  [
    name: "Start the container",
    run: "docker start ${{ steps.create_container.outputs.container_id }}"
  ],
  [
    name: "Check HTTP status code",
    uses: "nick-fields/retry@v2",
    with: [
      command:
        "INPUT_SITES='[\"http://localhost:4000/api\"]' INPUT_EXPECTED='[200]' ./scripts/check_status_code.sh",
      max_attempts: 3,
      retry_wait_seconds: 5,
      timeout_seconds: 1
    ]
  ],
  [
    name: "Write Docker logs to a file",
    if: "failure() && steps.create_container.outcome == 'success'",
    run:
      "docker logs ${{ steps.create_container.outputs.container_id }} >> docker.log"
  ],
  [
    name: "Upload Docker log file",
    if: "failure()",
    uses: "actions/upload-artifact@v3",
    with: [
      name: "docker.log",
      path: "docker.log"
    ]
  ]
]

Most of the steps are related to setting up Docker for caching intermediary images so that subsequent runs are quicker, but these steps are the most important:

[
  name: "Start the container",
  run: "docker start ${{ steps.create_container.outputs.container_id }}"
],
[
  name: "Check HTTP status code",
  uses: "nick-fields/retry@v2",
  with: [
    command:
      "INPUT_SITES='[\"http://localhost:4000/api\"]' INPUT_EXPECTED='[200]' ./scripts/check_status_code.sh",
    max_attempts: 3,
    retry_wait_seconds: 5,
    timeout_seconds: 1
  ]
]

After building the image and creating the container, we start it, and then send an HTTP request to it. We're not sure when the server is ready, so we use nick-fields/retry action to retry sending the request with a configurable number of maximum attempts. To send a request we use a convenient script scripts/check_status_code.sh I copied from https://github.com/lakuapik/gh-actions-http-status.

In the end, we upload a log as an artifact so we can inspect it in case the request fails.

There's an additional change we have to make to enable running the release outside of Fly.io environment. In the file rel/env.sh.eex replace the line

export ERL_AFLAGS="-proto_dist inet6_tcp"

with

if [[ -z "${FLY_PRIVATE_IP}" ]]; then
    export ERL_AFLAGS="-proto_dist inet6_tcp"
fi

After pushing the code to GitHub, you can see requests failing.

Run nick-fields/retry@v2 step

Checking the log file saved as an artifact

docker.log artifact

shows the following error messages:

15:31:02.527 [info] Running TestingReleaseWeb.Endpoint with Bandit 1.5.0 at :::4000 (http)
15:31:02.528 [info] Access TestingReleaseWeb.Endpoint at https://localhost
15:31:07.848 request_id=F8rJ9kxkVangNu4AAAAE [info] GET /api
15:31:07.848 request_id=F8rJ9kxkVangNu4AAAAE [info] Sent 500 in 251µs
15:31:07.849 request_id=F8rJ9kxkVangNu4AAAAE [error] ** (UndefinedFunctionError) function ExUnit.__info__/1 is undefined (module ExUnit is not available)
    ExUnit.__info__(:functions)
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:5: TestingReleaseWeb.HomeController.index/2
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.action/2
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.phoenix_controller_pipeline/2
    (phoenix 1.7.12) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5
    (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.plug_builder_call/2
    (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.call/2
    (bandit 1.5.0) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2

15:31:13.856 request_id=F8rJ97J1J9nc0xoAAAAB [info] GET /api
15:31:13.856 request_id=F8rJ97J1J9nc0xoAAAAB [info] Sent 500 in 243µs
15:31:13.856 request_id=F8rJ97J1J9nc0xoAAAAB [error] ** (UndefinedFunctionError) function ExUnit.__info__/1 is undefined (module ExUnit is not available)
    ExUnit.__info__(:functions)
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:5: TestingReleaseWeb.HomeController.index/2
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.action/2
    (testing_release 0.1.0) lib/testing_release_web/controllers/home_controller.ex:1: TestingReleaseWeb.HomeController.phoenix_controller_pipeline/2
    (phoenix 1.7.12) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5
    (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.plug_builder_call/2
    (testing_release 0.1.0) lib/testing_release_web/endpoint.ex:1: TestingReleaseWeb.Endpoint.call/2
    (bandit 1.5.0) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2

The fix for this is simple: change the :extra_applications in the mix.exs file from

[:logger, :runtime_tools]

to

[:ex_unit, :logger, :runtime_tools]

(https://github.com/almirsarajcic/testing_release/commit/534615eb5dcf0cc19c166971cdeeb23f4ad49708)

After pushing the code, we verify it works.

main.yml workflow result

You can also use a database by setting up the Docker container to use the network from the DB service, Redis, and whatnot. Possibilities are vast.

This was a post from our Elixir DevOps series.

More articles

Post Image

Leveraging FLAME for Efficient Screenshot Generation

A successful knowledge-sharing platform, like Elixir Drops, depends on making its content both discoverable and easily shareable. Metatags are essential for achieving this by providing search engines, social media platforms, and browsers with critical metadata to understand, display, and rank content appropriately. Among these, metatag images—visuals that appear when a link is shared on platforms like Facebook, Twitter, or Slack—play a key role in engaging users. These images, specified via Open Graph (og:image) and Twitter Card (twitter:image) tags, significantly enhance the appearance and click-through potential of shared links. For platforms centered on code sharing, developers typically use tools to style and format code snippets into visually appealing screenshots for social media. However, manually creating such images for a growing platform like Elixir Drops is not scalable. To address this, we automated the creation of custom metatag images using Elixir tools and the FLAME library for dynamic scaling. Automating Metatag Image Creation 1.Triggering a Background Job Every time a post is created or updated, a background job is queued to generate a screenshot of the post content. This image is uploaded to an S3 bucket and referenced in the post's metatags. def handle_event("save", %{"drop" => drop_params}, socket) do case create_or_update_drop(socket, socket.assigns.live_action, drop_params) do {:ok, drop} -> enqueue_seo_screenshot_creation(drop.id) { :noreply, push_navigate( socket, to: ~p"/profile" ) } {:error, changeset} -> {:noreply, assign_form(socket, changeset)} end end defp enqueue_seo_screenshot_creation(drop_id) do %{"drop_id" => drop_id} |> ScreenshotGeneratorWorker.new() |> Oban.insert() end 2. Background Job for Screenshot Generation The background job ensures that only posts with code blocks trigger the screenshot generation process. The job scans the Markdown content of the post using a regex. If no code block is found, the job is canceled to avoid unnecessary retries. def perform(%Oban.Job{args: args}) do args |> maybe_create_screenshot() |> maybe_retry_job() end defp maybe_retry_job({:ok, _image_url}), do: :ok defp maybe_retry_job({:cancel, reason}), do: {:cancel, reason} defp maybe_retry_job(error), do: error defp maybe_create_screenshot(args) do with {:ok, drop} <- get_drop(args["drop_id"]), :ok <- check_for_code_block(drop.body) do drop_screenshot(drop) else _error -> {:cancel, "No code block found"} end end defp check_for_code_block(body) do case Regex.run(@markdown_regex, body, capture: :first) do nil -> {:error, "No code block found"} _code_block -> :ok end end 3. Screenshot Generation Screenshots are created using Wallaby, which relies on Chromedriver. The setup dynamically generates browser-based screenshots of the code blocks. defp generate_screenshot(drop) do {:ok, session} = Wallaby.start_session( capabilities: %{ chromeOptions: %{ args: [ "--headless", "--no-sandbox", "window-size=1280,800", "--fullscreen", "--disable-gpu", "--disable-dev-shm-usage" ] } } ) url = build_url_with_auth(drop) %Wallaby.Session{screenshots: [screenshot]} = session |> Browser.visit(url) |> Browser.take_screenshot() Wallaby.end_session(session) {:ok, screenshot} end 4. Uploading to S3 The generated screenshot is uploaded to S3 for persistent storage and later reference in the post's metatags. defp upload_screenshot(screenshot, drop) do timestamp = Timex.to_unix(drop.updated_at) image_name = "drop-meta-image-#{timestamp}-#{drop.id}.png" case Client.upload_image(screenshot, image_name, "image/png") do {:ok, image_url} -> {:ok, image_url} error -> error end end Handling tasks like screenshot generation and S3 uploads can lead to spikes in resource usage, especially with increasing demand. To manage this efficiently, we leveraged FLAME, a library designed for elastic workloads. FLAME allows resource-intensive operations to run on short-lived infrastructure, scaling dynamically to meet demand and scaling down during idle times. FLAME is a distributed, serverless-inspired library and paradigm in Elixir, designed to efficiently manage elastic workloads—tasks with highly variable resource demands. It enables developers to treat their entire application as a lambda, allowing modular components to execute on short-lived infrastructure without requiring rewrites or complex orchestration.- Docs Using FLAME to Handle Screenshot Generation FLAME provides a powerful way to manage elastic workloads. Here are the steps to integrate FLAME for dynamically generating screenshots for your posts. 1. Add FLAME dependency # mix.exs: {:flame, "~> 0.5.1"}, 2. Setting up FLAME Inspired by a great example. We configure FLAME to enable or disable services depending on whether a node is running as a FLAME child. You can learn more about this setup from the Deployment Considerations and Pools documentation. Here’s how we configured application.exs: # application.exs @impl Application def start(_type, _args) do children = children( always: ElixirDropsWeb.Telemetry, always: ElixirDropsWeb.Endpoint, parent: ElixirDrops.Repo, parent: {DNSCluster, query: Application.get_env(:elixir_drops, :dns_cluster_query) || :ignore}, parent: {Phoenix.PubSub, name: ElixirDrops.PubSub}, # Start the Finch HTTP client for sending emails parent: {Finch, name: ElixirDrops.Finch}, # Start a worker by calling: ElixirDrops.Worker.start_link(arg) # {ElixirDrops.Worker, arg}, # Start to serve requests, typically the last entry parent: {FLAME.Pool, name: ElixirDrops.ScreenshotGenerator, idle_shutdown_after: 30_000, log: :info, max_concurrency: 2, max: 4, min: 0}, parent: {Oban, Application.get_env(:elixir_drops, Oban)} ) # See https://hexdocs.pm/elixir/Supervisor.html # for other strategies and supported options opts = [strategy: :one_for_one, name: ElixirDrops.Supervisor] Supervisor.start_link(children, opts) end # Tell Phoenix to update the endpoint configuration # whenever the application is updated. @impl Application def config_change(changed, _new, removed) do ElixirDropsWeb.Endpoint.config_change(changed, removed) :ok end # Exclude children marked with `parent` in the FLAME environment defp children(child_specs) do is_parent? = is_nil(FLAME.Parent.get()) is_flame? = !is_parent? || FLAME.Backend.impl() == FLAME.LocalBackend Enum.flat_map(child_specs, fn {:always, spec} -> [spec] {:parent, spec} when is_parent? == true -> [spec] {:parent, _spec} when is_parent? == false -> [] {:flame, spec} when is_flame? == true -> [spec] {:flame, _spec} when is_flame? == false -> [] end) end 3. Configuring the Fly Backend Since we're using Fly.io machines, we need to configure the FlyBackend and set the environment variables for FLAME. Note that Fly.io machines running FLAME tasks do not inherit the parent’s environment variables. Add the following configuration in config/runtime.exs: # config/runtime.exs config :flame, backend: FLAME.FlyBackend, env: %{ "AWS_ACCESS_KEY_ID" => aws_access_key_id, "AWS_ENDPOINT_URL_S3" => aws_endpoint_url, "AWS_REGION" => aws_region, "AWS_SECRET_ACCESS_KEY" => aws_secret_access_key, "BUCKET_NAME" => aws_bucket }, token: fly_api_token 4. Wrapping Screenshot Generation in FLAME Once FLAME is set up, we can wrap the screenshot generation logic in a FLAME call to execute the task on short-lived infrastructure. Here’s how you can wrap the screenshot generation and S3 upload logic: defp drop_screenshot(drop) do FLAME.call(ScreenshotGenerator, fn -> with {:ok, screenshot} <- generate_screenshot(drop), {:ok, image} <- File.read(screenshot) do upload_screenshot(image, drop) end end) end By integrating FLAME into our workflow, we have streamlined the process of generating and uploading screenshots for our posts, allowing us to handle elastic workloads efficiently without introducing unnecessary complexity. FLAME’s ability to dynamically scale resources on-demand ensures that we can handle spikes in demand—such as when processing multiple screenshot requests—while maintaining cost-effectiveness and simplicity. This approach not only simplifies our infrastructure but also enables us to focus on providing a better experience for users without worrying about the complexities of serverless architectures. Whether you're handling resource-intensive tasks like screenshot generation or other elastic workloads, FLAME offers a robust and scalable solution for modern web applications. You can learn more about FLAME: Official Docs Rethinking Serverless with FLAME Scaling Your Phoenix App in Elixir with FLAME Serverless With Servers? FLAME is...weird
Nyakio Muriuki
Post Image

Dripping Elixir knowledge

I started the first draft of this blog post by describing my Elixir journey and how I always felt a good place for sharing bits of Elixir knowledge was missing. But who cares? I’m on a mission here, and my goal is to show you what my team and I have been working on recently, so I’ll keep it concise. We’re the hard data types anyway, not the poetic ones. Those are the Ruby guys. In his blog post, Zach Daniel mentioned there were too many silos in the Elixir community. Well, I’m here to say that we might have added to the problem by creating another platform. Or maybe crafted something useful, time will tell. I'm talking about a website that’s a hybrid between Twitter and a blog in that it can host shorter articles and code snippets that are quick to write and even quicker to read. https://elixirdrops.net https://elixirdrops.net/d/VrY6GLZA There are no comments. There are no tags. There is no search. There are no likes. The list goes on. Still, there are some benefits to dropping some knowledge on our website: pages are indexable we have proper syntax highlighting sharing links on other websites looks great https://x.com/phx_tools/status/1854445026710843483 Okay, it could be bigger. But we're making improvements as we go. These are still the early days of the platform, so we don’t have many features yet. Some on purpose. For example, I’m on the fence about whether or not we should have comments. This website was created to share knowledge in the form of tips for one-way communication, so for chat and longer discussions, there are better channels. We're not trying to be all for everyone. We’re building software in a lean manner, so the first thing we’ll probably be adding are tags for discoverability. Currently, the site is empty, so tags at this point would be a form of premature optimization. Hopefully, it won’t stay that way and this becomes a problem, a nice one to have. Until then, you can discover new tips on the homepage, write new posts, and link to them on other platforms. Go on and share your knowledge with the world. https://elixirdrops.net
Almir Sarajčić
Post Image

Usability Heuristics for UI/UX design

What are usability heuristics? Developed by Jakob Nielsen, a prominent usability expert, usability heuristics are rules of thumb that can be used to evaluate user interfaces. These heuristics are based on principles of human-computer interaction and cognitive psychology. They serve as a checklist for designers, enabling them to create interfaces that enhance user satisfaction and reduce frustration, ultimately leading to a better user experience. The 10 Usability Heuristics by Jakob Nielsen 1. Visibility of System Status Keep users informed about the system's status through appropriate feedback in a reasonable time. This includes visual indicators, loading animations, or status messages so users know their actions have been recognized. 2. Match Between System and the Real World Use familiar language, concepts, and visual elements that align with users’ real-world experiences. Avoid technical jargon, and instead use terms users understand. For example, a magnifying glass icon indicates a search function. 3. User Control and Freedom Provide undo and redo options to empower users, allowing them control over their actions, especially when they select system functions by mistake. 4. Consistency and Standards Design should be consistent across screens and platforms. Users shouldn’t wonder whether different words, situations, or actions mean the same thing. 5. Error Prevention Prevent errors by using constraints, clear instructions, and confirmations for critical actions. For example, display password criteria to avoid mistakes in login forms. 6. Help Users Recognize, Diagnose, and Recover from Errors Provide clear, constructive error messages that use plain language and offer solutions, making it easy for users to understand and fix issues. 7. Recognition Rather Than Recall Reduce users' memory load by making objects, actions, and options visible, rather than forcing them to recall information from one part of the interface to another. 8. Aesthetic and Minimalist Design Avoid information overload. Strive for a clean, minimalist design that highlights key elements and avoids unnecessary clutter. 9. Help and Documentation If necessary, provide accessible help and documentation that is focused on the user's task and context. 10. Flexibility and Efficiency of Use Allow users to customize frequent actions. Provide shortcuts, accelerators, and customizations to enhance efficiency for experienced users while keeping the interface accessible for beginners. Applying These Rules While Designing Usability heuristics are invaluable tools for creating user-friendly interfaces. Conduct a heuristic evaluation by having a small group check your UI design against these principles. Follow up with user testing to observe real interactions, and make improvements based on feedback. Stay updated with industry trends to continuously improve your design skills.
Oghogho Igbinomwanhia