Bitbucket Pipelines costs slowly boiled me alive

When we started Design and Test Lab (software outsourcing business) back in 2011, git was our default choice to work with source codes. Initially, we put a server box under a work desk in our office and spent some efforts to set-up it and maintain it. It was some pain to administer it, uptime was not so good. And I was out looking for cloud-based alternative.

It was 2011, GitLab was launched only three years later (in 2014). At that time I've found two competitors: GitHub and Bitbucket.

I don't remember the exact numbers, but I've chosen Bitbucket because of the price per user. As far I remember, GitHub had pretty big pricing (something like $4 vs $20 per user). That's how we ended up with Bitbucket.

Later with adoption of automatic tests, we started using Bitbucket Pipelines — additional CI/CD product, which is integrated with Bitbucket git repositories. Every time we push our codes — automatic tests are running and telling if everything is OK or not. It helps us a lot — if a developer breaks some functions, Bitbucket Pipelines will run tests and mark code commits as green or red.

And this is at no additional cost! Bitbucket includes 2,500 minutes per month per organization for building, testing, and deploying with Bitbucket Pipelines. What is 2,500 minutes? If my tests are running for 10 minutes, I can run 250 tests per month, for free! (Ok, not for free, because we still pay $3/user/month.)

It seems a lot from the first sight, but what happened when our team grew to 20+ people:

• the bigger code base → more tests → tests run longer;
• more people on the project → more pull-requests → more test runs;
• more complex test infrastructure setup (for example, database servers, headless web browsers) → 2X/3X build minutes consumed by a single test run;
• business is growing → more projects → more and more and more test runs.

And it happened in 2017: we ran out of free 2,500 minutes per month. And my Bitbucket Pipelines bill was not so big — additional $10-$20 per month, that is not a big deal for me. Again, the Pipelines' value is still bigger than the bill — we have a point of control to test our software, and find out functional issues in minutes!

But eventually it was a trap. I was boiled slowly alive. Since 2017 my monthly bills for additional build minutes raised to $200+ per month, or over $2500+ per year. Now 75% of a monthly bill for Bitbucket — it is a bill from running tests in the cloud.

From Atlassian Annual Report 2023 I see that the business is going well: $2B of $3.5B revenue comes from cloud products (and mine contribution, too!).

(24,640 + 2,500 (given for free)) / 60 = 452 hours, or 18 days of computing time. Where all these tens of thousand build minutes come from? Here is an example of a pipeline run for one backend written in Python. It has two test suites consisting of 3915 tests each, and additionally we spin Postgres, Elasticsearch and Minio servers for integration tests. Wall time is just 16 minutes, but due to needed additional parallel processes for services, the spent build minutes is 69. Given, Bitbucket sells 1,000 build minutes for $10, such one 69-minutes test run costs us $0.69.

Everything is worse for UI tests in Selenium. UI tests are slower that unit and API tests. Here is an example that one UI test run consumes 217 build minutes (3 hours 37 minutes), which costs us $2.17!

So how can I assess and compare the computing costs? Are they big or small? If I will run tests 24/7 in a single process, it will cost me 31 days × 24 hours × 60 minutes × $0.01/min = $446. And here is what I can rent for $42 per month from Hetzner:

• 14-core Intel® Core™ i5-13500
• 64 GB DDR4 RAM
• 2 x 512 GB NVMe SSD (Gen4)
• 1 GBit/s bandwidth

So if we will just consider costs (I'm not taking into account bigger computing power), then Hetzner $446 / $42 = 10 times cheaper than Bitbucket Pipelines! I feel myself being used.

Quick FAQ:

— What you are going to do? 🤔🤔🤔

— I'm moving to Hetzner.

— But it will require SYSTEM ADMINISTRATION! 😱😱😱

— Yes, it will require system administration.

— But system administration will cost you fortune! 👻👻👻

— I don't think it will require too much administration. For example, Kamal simplifies deploying own services to bare servers to minutes.

What is your CI/CD bill? How do you reduce it?