Not a log line. Not a Slack message that scrolls past. A first-class work item in an ops queue that fans out to GitHub issue and Slack and is eligible for auto-fix.

The problem — failures had no queue

Builds broke, webhooks failed, checks flaked. The signal existed, but it had nowhere to go that an agent could pick up. Someone had to notice it in a console, copy the logs, write a bug, tag it, and hope the right person saw it hours later.

The cost wasn't the failure. It was the queue that didn't exist.

The fix — one queue for everything that can fail

Every external event hits one shape: a ticket. Same fields, same routing, same agent lifecycle.

A Cloud Build failure. A flaky E2E run. A customer report from the widget. A cron that missed its window. All of them land as type: "bug" via the admin API — title, repro steps, actual vs expected, priority, tags — and then the platform does the same thing every time: opens a GitHub issue, pings the right Slack channel, marks it eligible for an AI agent to investigate and open a PR.

Build on main breaks → ticket → agent PR is just one instance of the pattern. The pattern is what matters.

Design — boring on purpose

• Source → already on a bus (Pub/Sub, webhook, schedule) — no new infra

• Push or POST to /webhooks/<source> — verify token, decode, dedupe by delivery ID

• Client → POST /api/admin/ops with type: "bug" — title, stepsToReproduce, actualResult, expectedResult, priority, tags

• Shipeasy ops queue → fans out to GitHub issue + Slack → agent investigates → PR

What makes this cheap is what we didn't build: we use the delivery mechanism the cloud already provides and the service we already run. The glue is forty lines.

Implementation — one thin client

Same shape every time:

ShipeasyOps::Client.new.file_bug(
  title:               "Cloud Build FAILURE on #{branch} — #{sha}",
  steps_to_reproduce:  "Trigger \"#{trigger}\" reported FAILURE.",
  actual_result:       "Logs: #{log_url}\n\n#{failure_detail}",
  expected_result:     "Build completes and deploys.",
  priority:            "high",
  tags:                %w[cloud-build ci]
)

The method is a typed wrapper over one HTTP call:

def file_bug(title:, steps_to_reproduce:, actual_result:, expected_result:, priority:, tags:)
  post("/api/admin/ops", {
    type:             "bug",
    title:            title,
    stepsToReproduce: steps_to_reproduce,
    actualResult:     actual_result,
    expectedResult:   expected_result,
    priority:         priority,
    tags:             tags,
  })
end

def post(path, body)
  req = Net::HTTP::Post.new(URI("#{BASE_URL}#{path}"))
  req["Authorization"] = "Bearer #{@admin_key}"
  req["X-Project-Id"]  = @project_id
  req["Content-Type"]  = "application/json"
  req.body = body.compact.to_json
  res = Net::HTTP.start(req.uri.host, req.uri.port, use_ssl: true) { |h| h.request(req) }
  JSON.parse(res.body)
end

Dedupe at the edge (write-if-not-exists on the delivery ID) so at-least-once delivery doesn't create duplicate tickets.

Ways this could have gone

• Log the error via standard flag evaluation: good for exceptions, not a tracked work item with priority and repro.

• Public ticket path for in-app feedback: zero-auth, built for "report a problem" widgets — too heavy for an internal signal.

• File a proper bug via admin API (chosen): real work item, prioritized, tagged, routed, agent-ready.

Payoff — from red to green without paging anyone

A typical failure now travels from a bus to a PR without waking anyone. The team sees a PR that already exists and merges it. The person who merges it doesn't need to know the failure mode.

The rule we use before building new plumbing: if the event is already on a bus you can subscribe to, and something you already run can catch it, file a ticket and let the ops queue do the rest.

Originally published on dev.to at https://dev.to/shipeasy/ops-queue-for-agents-turning-every-failure-into-a-fixable-ticket-1jno

Built on Shipeasy — ops queue that delegates to agents. Supporting infra: flags, kill switches, dynamic configs. Core loop lives in the queue, not the flag.

Free tier, no card required; Team at $49/seat/mo removes limits.

→ shipeasy.ai · → docs.shipeasy.ai · → github.com/shipeasy-ai/shipeasy

PROBLEM — A failed build told no one

Our CI would fail, and then… nothing would happen. The failure sat quietly in a build console that nobody keeps open. Eventually someone would notice a change hadn't gone out, go digging, and realize the build had been red for hours.

And noticing was the easy part. Actually resolving it meant a whole code session: pull up the logs, find the failing step, reproduce it, and have an engineer sit down and personally shepherd the fix from broken to green. Every red build cost real human hours — plus the invisible tax of the delay before anyone even knew there was a problem.

The true cost of a broken build was never the build. It was a person having to find it, understand it, and hand-fix it.

SOLUTION — The failure files its own ticket — and an agent takes it from there

Now nobody watches a console and nobody triages. The moment a build fails, it automatically files a bug in Shipeasy — our ops platform — as a real, prioritized ticket with the failing step, the branch, and a link to the logs already attached.

From there it leaves human hands entirely. Shipeasy hands the bug to an AI agent, which investigates the failure, writes the patch, and opens a pull request against it. The loop that used to be "human notices → human reads logs → human fixes" is now "build fails → bug appears → agent fixes." The engineer's job shrank to reviewing a PR that already exists.

DESIGN — How the whole thing hangs together

The pipeline is deliberately boring — every hop is either something the cloud already does for free, or a service we already run:

• Cloud Build — build fails: a red deploy on main publishes automatically

• Pub/Sub topic → push subscription: filters to FAILURE · TIMEOUT · INTERNAL_ERROR

• HTTPS POST /webhooks/cloud_build

• Webhooks::CloudBuildController: verify token · decode · dedupe by build id

• ShipeasyOps::Client#file_bug: POST /api/admin/ops (type: "bug")

• Shipeasy ops queue — bug filed: fans out to GitHub issue + Slack

• ✦ AI agent investigates → opens a PR

What makes this cheap is the shape of it: we added no new infrastructure. The event was already on a bus (Pub/Sub). We already ran a service that could receive it. All we wrote was the glue in the middle.

IMPLEMENTATION — How it worked out in Rails

The build side needed no changes at all — Cloud Build publishes to the cloud-builds topic on its own. So the work was three small pieces: one gcloud command, one route, and one controller.

1. Point a filtered push subscription at the app. Pub/Sub does the delivery; the filter means the endpoint only ever wakes for a real failure.

gcloud pubsub subscriptions create cloud-build-failures \
  --topic=cloud-builds \
  --push-endpoint="https://our-app/webhooks/cloud_build?token=$SECRET" \
  --message-filter='attributes.status = "FAILURE"
    OR attributes.status = "INTERNAL_ERROR"
    OR attributes.status = "TIMEOUT"'

1. Add the route — it slots into the same webhook surface as our other providers.

scope "/webhooks", module: :webhooks do
  post "cloud_build", to: "cloud_build#create"
end

1. The controller. It authenticates the push, decodes the build payload (it arrives base64-encoded in message.data), throws away anything that isn't a real failure, dedupes — Pub/Sub delivers at-least-once, so a redelivery must not file a second bug — and files the ticket.

class Webhooks::CloudBuildController < Webhooks::ApplicationController
  before_action :verify_token

  FAILURE_STATUSES = %w[FAILURE INTERNAL_ERROR TIMEOUT]

  # POST /webhooks/cloud_build
  def create
    message = params[:message]
    return head(:bad_request) if message.blank?

    build  = decode_build_json(message[:data])   # base64 JSON → Hash
    status = (message.dig(:attributes, :status) || build["status"]).to_s
    return head(:ok) unless FAILURE_STATUSES.include?(status)

    return head(:ok) unless first_delivery?(build["id"])   # dedupe

    file_bug_for(build, status)
    head :ok
  end

  private

  # Compare-and-set on the build id: the first delivery wins, redeliveries no-op.
  def first_delivery?(build_id)
    Rails.cache.write("cloud_build:filed:#{build_id}", true,
                      unless_exist: true, expires_in: 7.days)
  end

  def verify_token
    expected = App::Secrets.cloud_build_webhook_secret
    provided = params[:token] || request.headers["X-CloudBuild-Token"]
    head :unauthorized unless
      ActiveSupport::SecurityUtils.secure_compare(provided.to_s, expected.to_s)
  end
end

The one line that files the ticket. The controller hands the build context to a thin Shipeasy client. This is the exact call — it turns a red build into a first-class bug that opens a GitHub issue, pings Slack, and becomes eligible for the auto-fix agent:

def file_bug_for(build, status)
    subs = build["substitutions"] || {}

    ShipeasyOps::Client.new.file_bug(
      title:              "Cloud Build #{status.downcase} on #{subs["BRANCH_NAME"]} — #{subs["SHORT_SHA"]}",
      steps_to_reproduce: "Cloud Build trigger \"#{subs["TRIGGER_NAME"]}\" reported #{status}.",
      actual_result:      "Logs: #{build["logUrl"]}\n\n#{build.dig("failureInfo", "detail")}",
      expected_result:    "The build completes and deploys.",
      priority:           "high",
      tags:               %w[cloud-build ci],
    )
  end

And the client itself is just a typed wrapper over one HTTP call — no new gems, no framework. This is all it takes to create a bug on the platform:

def file_bug(title:, steps_to_reproduce:, actual_result:, expected_result:, priority:, tags:)
    post("/api/admin/ops", {
      type:             "bug",
      title:            title,
      stepsToReproduce: steps_to_reproduce,
      actualResult:     actual_result,
      expectedResult:   expected_result,
      priority:         priority,
      tags:             tags,
    })
  end

  def post(path, body)
    req = Net::HTTP::Post.new(URI("#{BASE_URL}#{path}"))
    req["Authorization"] = "Bearer #{@admin_key}"   # sdk_admin_… key
    req["X-Project-Id"]  = @project_id
    req["Content-Type"]  = "application/json"
    req.body = body.compact.to_json

    res = Net::HTTP.start(req.uri.host, req.uri.port, use_ssl: true) { |h| h.request(req) }
    JSON.parse(res.body)
  end

That's the whole integration. The failure travels from Cloud Build to a filed, prioritized bug through two managed hops and about forty lines of Ruby — and the moment it lands, it's the platform's problem, not a person's.

SHIPEASY — One problem, several ways to solve it

• Report it as an error via see(): Dead simple and already everywhere in our code — but it produces an auto-filed error, not a first-class bug with repro steps and priority. Great for exceptions; not quite the tracked work item we wanted.

• Public feedback ticket: The zero-auth path meant for in-app "report a problem" widgets. Perfect for user feedback, heavier than we needed for an internal signal.

• File a real bug via the admin API (chosen): A proper bug — title, repro, priority, tags — that immediately opens a GitHub issue, pings Slack, and is eligible for the auto-fix agent. Exactly the lifecycle a broken build deserves.

THE PAYOFF — The last red build nobody had to fix

A build died with FATAL ERROR: Reached heap limit — JavaScript heap out of memory. The bundle outgrew its memory ceiling. The agent recognized the OOM pattern and opened a PR with the entire fix:

- NODE_OPTIONS="--max-old-space-size=4096"
+ NODE_OPTIONS="--max-old-space-size=8192"

A teammate who doesn't write backend merged it. The person who "fixed" it never had to know what a heap limit is.

The failure notices itself, files itself, fixes itself, and asks a human for nothing more than a nod.

Before standing up new infrastructure to react to an event, check whether the event is already on a bus you can subscribe to — and whether something you already run can catch it.

Originally published on dev.to at https://dev.to/shipeasy/when-a-build-breaks-the-bug-fixes-itself-3nmm

The ops side of this runs on Shipeasy — the ticket, the agent, the rollout. Core product is the ops queue that delegates to agents. Free tier, no card required; Team at $49/seat/mo removes limits.

→ shipeasy.ai · → docs.shipeasy.ai/sdks/ruby · → github.com/shipeasy-ai/shipeasy