Most teams start with a simple rule: if a test helps us catch problems early, put it in CI. That is a good instinct, but it does not scale forever. As a product and test suite grow, the question shifts from “should this test run?” to “where should this test run, and what decision should it influence?”

That is the real challenge behind the decision to keep tests in CI vs dedicated QA pipeline. CI is excellent for fast feedback and merge protection, but it is not a universal home for every automated check. Some tests are too slow, too flaky, too environment-dependent, or too expensive to run on every commit. Others are critical enough that they must stay close to the code change, even if they are imperfect.

This guide breaks down a practical way to split your checks, with concrete criteria for release gating, asynchronous validation, and QA workflow design. The goal is not to create a “perfect” pipeline, but a pipeline that gives developers confidence without turning every merge into a queue.

The basic distinction: CI is for fast decisions, QA pipelines are for broader confidence

Continuous integration is a software engineering practice where code changes are merged frequently and validated automatically, usually by running a set of tests and checks on each commit or pull request. For background, see continuous integration and test automation.

In practice, CI is strongest when it answers one question quickly:

Is this change safe enough to merge, or does it clearly break something important?

A dedicated QA pipeline answers a different question:

Does this build behave correctly across the broader set of scenarios, environments, and integrations we care about before release?

Those are related questions, but not identical. Mixing them blindly creates two common failure modes:

  1. CI becomes too slow, so people ignore it, rerun failed jobs, or batch changes unnecessarily.
  2. QA becomes too detached from the code path, so defects are discovered late and become expensive to fix.

A good test pipeline design balances both.

What belongs in CI by default

If a test is fast, deterministic, and directly tied to the code change, it should usually live in CI. That includes checks that can block merges with a high signal-to-noise ratio.

1. Unit tests with clear boundaries

Unit tests are usually the first line of defense because they are fast, easy to parallelize, and closely coupled to the code under change. They should be kept in CI if they:

  • run in seconds or a few minutes,
  • do not require external services,
  • fail for a clear, reproducible reason,
  • provide useful diagnostics when they fail.

If a unit test starts depending on network calls, time, the filesystem in a brittle way, or shared mutable state, it is no longer behaving like a true unit test. That is often a sign the test itself needs refactoring, not relocation.

2. Static analysis and linting

Linting, type checks, formatting checks, and basic security scans are ideal CI candidates because they are deterministic and cheap. They are not “tests” in the classical sense, but they belong in the same gate because they reduce obvious defects before merge.

3. Fast API checks on isolated dependencies

A small set of API tests can work well in CI if the environment is controlled. For example, if the service runs in a container with stubbed downstream dependencies, a smoke-level check can validate routing, authentication, schema shape, and a few core flows.

These tests are especially useful when they cover code paths that unit tests miss, such as serialization, middleware, or integration glue inside a service boundary.

4. Contract tests for important interfaces

Contract tests can stay in CI when they protect a team-owned boundary, such as a service-to-service API or a message schema. They are valuable because they detect breaking changes early without needing a full end-to-end environment.

If the contract is stable and the tests are narrow, they are a strong candidate for merge blocking.

What usually does not belong in CI

The fact that a test is automated does not make it a good fit for the main PR gate. Once a test becomes slow, expensive, or sensitive to unrelated failures, it starts competing with developer flow.

1. Broad end-to-end regression suites

Full-browser end-to-end suites are often the first thing teams try to run in CI, and often the first thing that causes pain. These tests are useful, but they are usually better suited to a later pipeline stage or a separate QA workflow because they tend to be:

  • slower than unit or API tests,
  • more likely to fail due to UI timing or environment issues,
  • more expensive to debug,
  • more sensitive to data setup and cleanup.

A few thin smoke tests can stay in CI, but the broad regression set usually belongs elsewhere.

2. Long-running cross-system validations

Any check that depends on multiple external systems, real third-party APIs, email delivery, payment gateways, or asynchronous backend jobs often has too many failure modes for the main CI gate.

These tests still matter, but they are often better as post-merge validation, nightly runs, or release-candidate verification.

3. Tests with known flakiness that you have not isolated yet

Flaky tests are one of the main reasons teams move checks out of CI. But moving them should not be the first move. If a flaky test guards a critical risk, you should first try to isolate the root cause:

  • remove shared state,
  • stop depending on timing assumptions,
  • replace arbitrary sleeps with explicit waits,
  • stabilize test data,
  • run parallel-safe setup and teardown,
  • eliminate environment coupling.

If a test is flaky because it is asserting something real in an unstable way, relocating it can hide the problem. If it is flaky because the environment is noisy and the assertion is still valuable, moving it can be the right short-term choice.

4. Exploratory automation and low-signal checks

Tests that are mostly informational, duplicate coverage, or catch rare defects with high setup cost are usually not merge blockers. They can still be useful in a QA pipeline or scheduled suite, especially if they help detect regressions in a high-risk area.

A practical decision framework for keep tests in CI vs dedicated QA pipeline

The decision is easier if you evaluate each test using a few dimensions instead of asking whether it is “important.” Most tests are important. The real question is whether they are appropriate for immediate merge gating.

1. Feedback time

Ask how long the test adds to the critical path for every developer.

Good CI candidates usually have one or more of these characteristics:

  • they finish quickly,
  • they can run in parallel,
  • they fail early,
  • they do not need expensive environment provisioning.

If a suite turns a pull request into a 30-minute wait, the team will feel that pain daily. At that point, even reliable tests can become counterproductive as blockers.

2. Determinism

A test should fail because the product changed, not because the environment was inconvenient. Deterministic tests are strong CI candidates. Tests that depend on race conditions, third-party uptime, or unstable shared resources are weaker candidates.

3. Diagnostic quality

A test is more valuable in CI if its failure tells you something actionable. A unit test that points to a specific function or assertion is easy to act on. A UI test that fails somewhere in a multi-step flow with no clear step boundary is harder to trust as a merge gate.

4. Coverage gap

Some tests should stay in CI because nothing else catches that defect class cheaply. For example, a tiny end-to-end smoke test might be the only automated validation that a deployable build actually boots, logs in, and reaches a key page.

5. Cost of delay versus cost of escape

A CI gate should block only when the risk of merging a bad change outweighs the cost of slowing developers. A QA pipeline should absorb broader validation when the cost of delay would be too high on every commit.

This is where teams often get stuck. They overestimate how much confidence a huge CI suite really adds, and underestimate the damage of slowing every merge.

A useful split: blocking, asynchronous, and release-stage checks

Instead of thinking in binary terms, treat your pipeline as three layers.

Layer 1: blocking CI checks

These run on each pull request and should block merge if they fail.

Typical examples:

  • unit tests,
  • lint and type checks,
  • a small number of smoke tests,
  • contract tests for critical interfaces,
  • a tiny set of fast API checks.

Keep this layer small enough that people trust it and wait for it.

Layer 2: asynchronous validation

These checks run after merge, after deploy to an integration environment, or on a schedule. They do not block the developer from merging, but they do affect release confidence.

Typical examples:

  • broader API regression,
  • cross-service integration tests,
  • browser regression suites,
  • migration checks,
  • data integrity checks.

This layer is where many teams should place their most expensive automated coverage.

Layer 3: release gating in a dedicated QA pipeline

This is the last stage before production release, or before promoting a build to staging, UAT, or a production-like environment. It is the right place for tests that require a more stable environment and more complete validation.

Typical examples:

  • full end-to-end journey tests,
  • business-critical workflow validation,
  • compatibility checks across browsers or devices,
  • tests requiring seeded data and dedicated environments,
  • final smoke checks after deployment.

A dedicated QA pipeline is especially valuable when release timing is explicit and controlled, such as weekly releases or promoted build workflows.

How flaky test isolation changes the answer

Flaky tests deserve special treatment because they distort your signal. A flaky test in CI is worse than no test in many cases, because it creates uncertainty and slows the whole team.

The right response depends on why it is flaky.

When to fix it and keep it in CI

Keep the test in CI if:

  • it protects a critical code path,
  • it is only flaky in a narrow, diagnosable way,
  • the failure mode is due to test design, not environmental noise,
  • the team can afford to harden it quickly.

Common fixes include:

  • replacing fixed sleeps with waits tied to observable conditions,
  • using isolated test data per run,
  • removing order dependence,
  • avoiding shared browser sessions or shared mutable fixtures,
  • mocking only unstable external boundaries.

For example, in Playwright, an explicit wait for a visible element is better than assuming a page is ready after an arbitrary pause:

typescript

await page.goto('/checkout');
await page.getByRole('button', { name: 'Place order' }).waitFor({ state: 'visible' });
await page.getByRole('button', { name: 'Place order' }).click();

When to move it out temporarily

Move the test into a QA pipeline if:

  • the failure is environment-driven and not quickly fixable,
  • the suite is too heavy to keep blocking merges,
  • the test still provides value, but the value is better realized after merge,
  • you need time to redesign the test or the environment.

That should be a deliberate move, not a permanent excuse to keep unstable coverage somewhere hidden. Track the test and set a plan to either stabilize it or retire it.

How to think about test pipeline design

A strong test pipeline is not just a list of jobs. It reflects how your organization makes decisions.

Start with the risk model, not the tool

Ask which defects are most expensive if they reach users. Examples include:

  • broken login,
  • failed payment submission,
  • data loss,
  • broken deployment artifacts,
  • bad permissions behavior,
  • API contract regressions.

Then map each risk to the cheapest reliable test level that can catch it early.

  • Unit test if the logic is local.
  • API or contract test if the issue is around service boundaries.
  • UI or end-to-end test if the user-facing flow needs validation.
  • Dedicated QA pipeline if the test is broad, slow, or environment-heavy.

Use multiple suites with explicit intent

One of the biggest anti-patterns is a single “test” job that tries to do everything. A better structure is to give each suite a purpose:

  • smoke: can we build and boot the product?
  • PR gate: can this change safely merge?
  • integration: do key services and APIs work together?
  • regression: did anything else break?
  • release: is this build acceptable to ship?

That naming matters because it keeps the team honest about why a suite exists.

Keep the CI gate short and stable

A practical rule is that the merge gate should feel predictable. If people can usually get a result quickly, they will respect it. If it frequently fails for reasons unrelated to the diff, it becomes noise.

Good CI design often includes:

  • test shard parallelization,
  • targeted test selection for changed areas,
  • containerized dependencies,
  • disposable test databases,
  • deterministic seeds and fixtures,
  • clear ownership of failures.

Example pipeline split for a typical web product

Here is one reasonable shape for a team building a web app with backend services.

In CI on every pull request

  • linting,
  • type checking,
  • unit tests,
  • component tests,
  • 1 to 3 smoke-level API tests,
  • 1 or 2 critical browser checks.

After merge to main

  • broader API regression,
  • service integration tests,
  • database migration checks,
  • more browser coverage in a clean environment.

In a dedicated QA pipeline before release

  • end-to-end business journeys,
  • payment or checkout flows,
  • role-based access and permission verification,
  • cross-browser validation,
  • final deployment smoke tests.

This split reduces merge friction without sacrificing release confidence.

A minimal GitHub Actions example for separating fast and slow checks

The mechanism does not matter as much as the policy, but it helps to make the split concrete.

name: ci

on: pull_request: push: branches: [main]

jobs: fast-checks: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 - run: npm ci - run: npm run lint - run: npm test – –runInBand

qa-suite: if: github.ref == ‘refs/heads/main’ runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 - run: npm ci - run: npm run test:integration - run: npm run test:e2e

In a real setup, the QA suite often runs against a deployed environment rather than local process execution. The important part is that the merge gate and the broader QA workflow are not the same thing.

Edge cases where the answer is not obvious

Highly regulated or safety-sensitive systems

If you are in a domain where evidence, traceability, or pre-release verification matters more than developer speed, you may keep more tests in the release gate. That is common in finance, healthcare, or systems with strict change control.

Even then, the principle remains the same, fast checks in CI, broader validation in QA. The difference is that your QA gate may be mandatory and heavily audited.

Monorepos with many services

Large monorepos often cannot afford to run everything in CI on every change. In those cases, selective execution becomes crucial.

You may keep only impacted unit tests, a few shared contract tests, and a smoke subset in CI, while pushing service-level regression into a dedicated pipeline keyed off changed packages or deployment candidates.

Teams with frequent UI churn

If the UI changes frequently, broad browser regression in CI can become a maintenance burden. That does not mean browser coverage is bad. It means you should be selective.

Stable user journeys, critical conversion paths, and authentication flows are better candidates for automation than every decorative or transient UI state.

Legacy systems with slow test setup

When environment provisioning is slow, teams sometimes force too much into CI because they fear a separate environment. That usually backfires. If test setup is the bottleneck, improve environment management, use containers where possible, and separate fast feedback from heavy validation.

A decision checklist you can actually use

When deciding whether a test should stay in CI or move to a dedicated QA pipeline, ask:

  1. Does this test fail fast enough to be useful on every merge?
  2. Is the signal clear and reproducible?
  3. Does it depend on unstable external systems or shared state?
  4. Does it protect a defect class that nothing else catches?
  5. Would a failure block developers for too long relative to the risk it reduces?
  6. Could the same risk be covered with a smaller or lower-level test?
  7. Is the test currently flaky, and if so, why?
  8. Do we need this check to block merge, or is release gating enough?

If most answers point to speed, determinism, and strong signal, keep it in CI. If most answers point to breadth, cost, environment dependence, or release-specific validation, move it into the QA pipeline.

Common mistakes teams make

Putting too much into CI because “it is important”

Importance alone is not enough. A test can be critical and still belong in a later stage if it is too slow or brittle for every commit.

Moving flaky tests out without fixing root causes

That can be a temporary containment strategy, but it is not a solution. Flaky tests are a quality problem, not just a scheduling problem.

Treating QA as a dumping ground

A dedicated QA pipeline should not become a place where low-value tests go to disappear. It should be a deliberate stage with ownership, timing, and reporting.

Using one environment for every kind of validation

A single shared environment often causes coupling between PR checks, integration validation, and release approval. Separate environments or ephemeral instances can dramatically improve test pipeline design.

A simple rule of thumb

If a test is fast, deterministic, and directly answers whether a change is safe to merge, keep it in CI.

If a test is broader, slower, environment-heavy, or primarily validates release readiness, move it into a dedicated QA pipeline.

If a test is flaky, first decide whether it is flaky because the product or the test is unstable. That distinction matters more than where the test runs.

Final thoughts

The best pipeline is not the one with the most automation, it is the one that gives the team the right signal at the right time. CI should protect developer flow and catch obvious regressions early. A dedicated QA pipeline should absorb the slower, broader, and more expensive checks that are still essential before release.

That separation makes release gating more trustworthy, reduces rerun culture, and gives QA a clearer operational role. Instead of asking whether every test belongs in CI, ask what decision each test is supposed to support. Once you frame it that way, the boundary between CI and QA becomes much easier to draw.