Dependabot PRs: Semver Is Not a Risk Model

A version number says what the library author promises. It does not say what the update will change in your project. In this guide Jev rates Dependabot pull requests from the release notes and the project policy. A Buddy pipeline merges the safe updates on its own and leaves the rest for review.

Where semver gets it wrong

Semver (semantic versioning) is a convention for numbering releases as major.minor.patch:

• patch - a fix without API changes, for example 3.9.8 → 3.9.9,
• minor - a new feature that stays backward compatible, for example 0.26 → 0.27,
• major - a change that may break something, for example 9 → 10.

The simplest auto-merge rule is built on this convention: merge patch and minor, wait on major. Three cases where this rule gets it wrong:

• A minor in 0.x in production code. @floating-ui/react 0.26 → 0.27 is a minor for the rule, so merge. In 0.x, however, a minor may change the API.
• A major of a development tool. ESLint 9 → 10 is a major, so wait. ESLint never reaches the browser. A major may change the lint result, but then the checks fail and the pipeline merges nothing. If lint, tests and build pass, users will not notice anything.
• A major of a build tool. Vite 7 → 8 is also in devDependencies, but it is the tool that builds the bundle. Unit tests do not check whether the built page works in the browser.

GitHub has the dependabot/fetch-metadata action for this. It returns the bump type: semver-patch, semver-minor or semver-major. It is still the same version number. The context is missing: what the package does in the project and what the release notes say.

Project

FrogePC is a small store built with Vite and React. The cart, discount and currency logic lives in src/ and has tests. All dependencies are pinned to exact versions, so Dependabot proposes every new release.

.github/dependabot.yml checks npm once a week and groups the ESLint packages into one PR:

yaml
version: 2
updates:
  - package-ecosystem: npm
    directory: /
    schedule:
      interval: weekly
    open-pull-requests-limit: 10
    groups:
      eslint:
        patterns:
          - eslint
          - "@eslint/*"

Policy in the repository

Jev judges only what it is given. That is why the rules are written down in .buddy/dependency-policy.md and not only agreed within the team:

• everything in dependencies ends up in the browser,
• any change to a 0.x package in runtime is high risk, a minor included,
• a Vite major is high risk, because the tests do not run the built page,
• a major of a tool that does not reach the bundle (ESLint, Prettier) is low risk if the checks pass,
• breaking changes in release notes raise the risk regardless of the version number.

The policy also says what the tests do not cover: the React components in web/.

Pipeline

The pipeline starts on a PR to main. A condition on BUDDY_RUN_PR_HEAD_BRANCH lets through only dependabot/ branches. The actions in order: release notes, checks, Jev's rating and then one of three branches: merge, a request for review or Fail the run.

Image loading...

Release notes and checks

Dependabot pastes the release notes and the changelog into the PR description, so this is the most important context for Jev. A GitHub CLI action fetches them. It signs in through the GitHub integration in Buddy, with no separate token in variables:

yaml
- action: Fetch release notes
  type: GIT_HUB_CLI
  integration: GitHub
  commands: gh pr view "$BUDDY_RUN_PR_NO" --json title,body --template '# {{.title}}{{"\n\n"}}{{.body}}' > pr-notes.md

BUDDY_RUN_PR_NO is the PR number. The GH_REPO pipeline variable points to the repository. The file lands in the pipeline filesystem, so the next actions can see it. Then the checks:

yaml
- action: Run checks
  type: BUILD
  docker_image_name: node
  docker_image_tag: "22"
  commands: |-
    npm ci --ignore-scripts --no-audit --no-fund
    if { npm run lint && npm run format:check && npm test && npm run build; } > checks.log 2>&1; then export OUTPUT_CHECKS=passed; else export OUTPUT_CHECKS=failed; fi

The action stays green and the result of the checks goes to OUTPUT_CHECKS.

Jev rates the risk

Image loading...

The type is Score, because risk levels have an order. The criteria, from the lowest:

• none: only development tools change, or a stable runtime package gets a patch that does not change behaviour,
• low: a major of a development tool, or a minor of a stable runtime package with no breaking changes and no changed defaults,
• high: any change to a 0.x runtime package, a major of a runtime package or of Vite, or breaking changes in the release notes.

As context Jev gets the PR title, the policy, package.json and the release notes. It runs only after green checks. There is no point in rating an update that does not pass the build.

Merge or review

Merge pull request runs when the checks passed, the result is not high and confidence is at least 0.7. Merge first, comment second:

bash
RATING="Jev rated this update \`$BUDDY_ACTION_JEV_RESULT\` risk (confidence $BUDDY_ACTION_JEV_CONFIDENCE). Checks passed"
if gh pr merge "$BUDDY_RUN_PR_NO" --squash --match-head-commit "$BUDDY_RUN_COMMIT"; then
  gh pr comment "$BUDDY_RUN_PR_NO" --body "$RATING, merged."
  exit 0
fi
HEAD_SHA=$(gh pr view "$BUDDY_RUN_PR_NO" --json headRefOid -q .headRefOid)
MERGEABLE=$(gh pr view "$BUDDY_RUN_PR_NO" --json mergeable -q .mergeable)
if [ "$HEAD_SHA" != "$BUDDY_RUN_COMMIT" ]; then
  echo "The branch moved to $HEAD_SHA during the run, the next run reviews it."
  exit 0
fi
if [ "$MERGEABLE" = "CONFLICTING" ]; then
  gh pr comment "$BUDDY_RUN_PR_NO" --body "@dependabot rebase"
  exit 0
fi
gh pr comment "$BUDDY_RUN_PR_NO" --body "$RATING, but the merge failed ($MERGEABLE). Merge it by hand."
exit 1
$$$$$$$$$$$$$$$$$

• --match-head-commit merges only the commit that passed the checks. If Dependabot pushes a new one during the run, the next run rates it.
• On a conflict with main the action asks Dependabot for a rebase, and the synchronize event starts the pipeline again.
• Any other error leaves a comment in the PR and ends the run with the Failed status.

On the PR with eslint 10.12.0 Jev gives none with confidence 0.95. The action log shows the order: first the merge of the exact commit that passed the checks, then the comment.

Image loading...

Ask for review runs on high or on confidence below 0.7 and leaves Jev's rating in the PR. Fail the run runs after failed checks and prints the end of checks.log.

Five PRs, one pipeline

The semver rule and Jev agree in only three cases out of five. The pipeline merged two PRs on its own:

Image loading...

Three wait for a human, each with a comment from the pipeline:

Image loading...

Prettier: a patch that changes nothing

The 3.9.9 release notes have one fix in the Markdown parser. Prettier only formats code, so Jev gives none with confidence 1.00. The pipeline leaves a comment and merges. This run used an earlier version of the action that commented before the merge. That is why the comment says "merging" and not "merged":

Image loading...

ESLint 10: a major that can be merged

The ESLint 10 release notes are a long list of breaking changes: the end of .eslintrc support, removed SourceCode and context methods, a stricter RuleTester. For a plugin author that is a lot of work. This project uses flat config and has no custom rules, and ESLint never reaches the bundle. Lint, tests and build pass, so Jev gives low with confidence 0.78 and the pipeline merges despite the major.

The first run did not merge the PR. After the Prettier merge the ESLint branch had a conflict with main. After the @dependabot rebase comment Dependabot rebuilt the branch, synchronize started the pipeline again and that run merged the PR. Today the Merge pull request action sends that comment itself.

Image loading...

Vite 8: a major of the tool that builds the page

The Vite 8.3.1 release notes are mostly fixes, but it is still a major of the tool that builds the bundle. The policy says that the tests do not run the built page, so Jev gives high. Confidence is the lowest here (0.64-0.67), because the checks pass and there is nothing alarming in the notes. After every Dependabot rebase the pipeline rates the PR again and asks for a review each time:

Image loading...

Floating UI 0.27: a minor the rule would not have stopped

This is the strongest example. For the semver rule 0.26 → 0.27 is a minor, so an automatic merge. The 0.27 release notes list only "Patch Changes", but they change how useRole, useDismiss and FloatingFocusManager behave. The store uses useRole in the discount tooltip in the cart. The tests in src/ will not catch this, because the components in web/ have no tests. The package is in 0.x and in dependencies, so Jev gives high with confidence 0.97:

Image loading...

Things to watch

• The branch name does not confirm the author. The condition on dependabot/ only limits which PRs start the pipeline. Anyone with push access can create a branch with that name. Before you turn this on in a shared repository, add a check at the start of Fetch release notes:

  bash
  AUTHOR=$(gh pr view "$BUDDY_RUN_PR_NO" --json author -q .author.login)
  [ "$AUTHOR" = "app/dependabot" ] || exit 1
  $$

• mergeable can be UNKNOWN right after a push. GitHub computes it in the background. If the merge fails and the answer is UNKNOWN, the action leaves a comment to merge by hand. Ask again after a few seconds before you treat the answer as final.
• Code from the PR runs in your project. npm ci installs packages from the Dependabot branch, which is why it uses --ignore-scripts. Only the GitHub CLI action can merge, not the container that runs the checks.
• Jev will not detect a malicious version. A hijacked package has clean release notes and passes the tests, so it gets none. Malicious versions are usually removed from npm within hours. With auto-merge, extend the cooldown in dependabot.yml, for example default-days: 7. By default Dependabot waits 3 days after a release. Security updates skip the cooldown.
• Branch protection can block the merge. If main requires a status from Buddy, the run that is still in progress has no green status yet, so GitHub may reject the merge.
• The policy is half of the result. Without the sentence about 0.x Jev rates the Floating UI minor the same way the semver rule does.
• Jev does not replace tests. It rates the risk of what the checks do not verify. The weaker the tests, the more should end up in high.

What next

• TypeSafe integration - how to connect Jev to your workspace,
• Conditional executions - all types of action conditions,
• Pull request testing - pipelines triggered by PRs,
• Flaky Test or Bug? Let the Pipeline Decide - the same model in failed test triage.