Write tests in plain English. Let AI run them. See pass/fail in Jira.

1 Why VibeTest

Every team knows it should test more. But automated tests are expensive to write and maintain, and manual testing doesn't scale. VibeTest takes a different path: describe the test in plain English on the Jira issue, and an AI agent runs it for real — on your own machine.

How it compares

VibeTest sits between the worlds of manual QA and heavyweight test automation — and sidesteps the usual trade-offs of each.

Who it's for

• QA & test engineers who want coverage without maintaining a fragile automation suite.
• Product managers & analysts who can finally write executable acceptance tests in the language they already use.
• Developers who want a fast, AI-run check attached to the issue — using the agent and machine they already trust.
• Security-conscious teams who can't send their code or data to a third-party testing cloud (see Security & privacy).

2 How VibeTest works

VibeTest has two halves that work together: a Jira app where you write and review tests, and a small desktop app on your Mac that actually runs them with an AI agent.

The end-to-end flow

When you press Run test in Jira, here is what happens behind the scenes:

What you need

• A Jira Cloud site with the VibeTest by AppDash app installed (ask your Jira admin if you don't see the panel on issues).
• A Mac to install the desktop app on (macOS).
• An AI coding agent installed on that Mac. VibeTest uses Claude Code by default; you can point it at others (see Use the desktop app).

The rest of this guide walks through it in order: install the desktop app, pair it with Jira, then create, run, and review your first test.

3 Install the desktop app

Everything runs through AppDash AI Link, the free desktop companion. Install it first — you'll pair it with Jira in the next step.

1. Download the app. Open appdash.ai/download and download the latest macOS build (a .dmg file). You'll also find this link inside Jira — in the VibeTest panel's empty state (Download →) and on the pairing page (Download now →).
2. Install it. Open the .dmg and drag AppDash AI Link into your Applications folder, then launch it. On first launch macOS may ask you to confirm you want to open an app downloaded from the internet — choose Open.
3. Find it in your menu bar. AppDash AI Link lives in your Mac's menu bar and opens a single window with four tabs: Connections, Activity, Console, and Settings. It opens on the Connections tab.

4 Pair your Mac with Jira

Pairing links one Jira site to your desktop app, so runs you launch in Jira reach your Mac. You do it once per site, across two windows — the VibeTest settings page in Jira and the desktop app — confirming a short code on both so nobody can pair without your approval. Jira walks you through four stages: Generate → Send to desktop → Confirm → Connected.

1. Open the Connections page in Jira. In Jira, go to Project settings → Apps → VibeTest by AppDash. The Connections page lists any desktops already paired to this project, and is where you pair a new one.

   

2. Generate the pairing link in Jira. Click Pair AppDash AI Link. Jira advances to Send to desktop and shows a short user code (note it — you'll check it in step 4) and a Copy pair link button. Click Copy pair link.

   

3. Paste the link into the desktop app. Switch to AppDash AI Link. On the Connections tab, click + Add Connection, paste the link into the box, and click Pair this device.

   

4. Check that the codes match. The desktop now shows "Pairing in progress" with the same user code plus an app fingerprint. Confirm the user code matches the one Jira showed in step 2 — this is the security check that proves you're pairing the right machine.

   

5. Confirm in Jira — you're connected. Back on the Jira page, with the codes matching, click Confirm pairing. The tracker advances to Connected, and the desktop's Connections tab shows the new site under Sites with a green Polling dot. Your Mac is now listening for test runs from that site.

5 Create a test in Jira

Tests live on Jira issues. Every issue gets a VibeTest panel where you write, run, and review tests.

Open the VibeTest panel on an issue

Once your Jira admin has installed the app, each issue gains a Tests section — that's the VibeTest panel. Open any Jira issue and scroll down, past the description and fields, to find it.

On an issue with no tests yet, the panel shows a welcome screen inviting you to create your first one.

Three ways to write a test

Writing a BDD scenario

Click BDD Scenario. The editor opens with your test as a list of steps — each a coloured keyword plus a plain-English sentence. The title you gave the test becomes the scenario name.

Anatomy of a step

Reading a step row from left to right:

• A status dot shows that step's result after a run, and a grip handle beside it lets you drag the step to reorder it (or focus it and use the arrow keys).
• The keyword — a coloured dropdown (Given / When / Then / And / But). Click it to change the keyword.
• The sentence — a plain-English text box describing the step; it grows as you type.
• Row actions on the right — add a data table, duplicate the step, or delete it.

Given, When, Then — what each step means

BDD describes behaviour as a short story: set the scene, take an action, then check the result. Each keyword has a job (and a colour), and every step is checked individually — so a run shows exactly which one failed.

Adding steps

The blue + Add … step button adds the next step and suggests the natural keyword — after a Given it offers When, then Then, then And. To add a specific keyword instead, click the ▾ caret on the right of the button and pick one.

Reordering steps

Steps run top to bottom, so order matters. To move one, hover the step to reveal its drag handle (the ⠿ grip on the left), then drag it up or down — a teal line shows where it will land. Prefer the keyboard? Tab to the handle and press ↑ / ↓.

Adding a data table to a step

A data table attaches a small grid of structured data to a step — handy for example inputs such as credentials, products, or parameters. Click the table icon in a step's row actions to add one.

• Edit any header or cell by clicking it.
• + Add row / + Add column grow the table; hover a row or column and click its ✕ to remove it.
• Remove table detaches the table from the step.

Save it

Click Save in the top bar. A "Saved" confirmation flashes and the test joins the list on the issue. Your edits are held locally until you save, so you won't clash with someone editing the same test at the same time.

Prefer free-form? Click Free-Form Task instead and describe the test in two boxes: the Task (what to do) and the optional Success criteria (how to tell it passed).

Generate tests with AI

Click Generate with AI (or choose it from the New Test menu) to let the AI draft a BDD test for you. Describe what to cover in the Requirements box and click Generate. The AI drafts a BDD scenario you can review, edit, and save. With a paired Mac your local agent produces the draft; with no Mac paired a confirm dialog offers Generate with Rovo AI to draft it inside Jira instead. Note the current MVP limit: one test per generation.

6 Manage your tests

Once an issue has tests, the panel shows them as a list. The top of the panel gives you an at-a-glance summary and a button to add more.

Reading the list

• Status tiles count tests by their latest run. Click a tile (e.g. failed) to filter the list to just those tests; a banner appears with a Clear filter link.
• Each row shows a status icon, the title, the type (BDD or Free Form), a status pill (passed failed running queued), and the time of the last run. A failed test also shows a one-line summary of what went wrong.
• A FLAKY chip appears when a test has been giving mixed results across its recent runs.
• Tests stay in the order you arrange them. Drag a row by its grip to reorder it (or focus a row and press ↑/↓); the order is saved. Reordering is available when no status filter is active.

Adding more tests

Click New Test (top-right) and pick BDD test, Free-Form test, or Generate with AI — the same flows from section 5.

Opening and editing a test

Click any row to expand it. The row and its editor fuse into a single teal-outlined focus card so the test you're working on stands out from the rest of the list.

• Edit the title or any step, then click Save. A "Saved" flash confirms it. Your edits are kept locally until you save, so you won't lose work if someone else changes the test at the same time.
• Duplicate (in the ⋯ menu) makes an identical copy to tweak — handy for testing variations.
• Delete (in the ⋯ menu) permanently removes a test and its run history after a confirmation — this can't be undone.

7 Run a test

Running a test sends it to your paired Mac, where the AI agent carries it out for real. You start it right from the panel.

1. Open the test by clicking its row to expand the focus card.
2. Click ▶ Run test in the editor's top bar. (Run test appears on any saved test — BDD or Free-Form; it's disabled only while a run is already in progress.)
3. Confirm in the dialog. A short "Run this test with your AI agent?" panel names the device the run will go to; click Run test to dispatch (or Cancel to back out).
4. Watch it go. A run card appears and moves through these states:
   • Queued — sent to your Mac, waiting for it to pick up.
   • Running — the agent is executing. For BDD tests the step checklist lights up step by step in real time, with a spinner on the current step.
   • Passed / Failed — the final verdict (covered next).

8 Review the results

When a run finishes, its card fills in with everything you need to understand what happened — the per-step outcome, evidence, screenshots, and an audit trail.

A passing run

A green Passed run shows every step with a checkmark and how long each took. The editor's run pill reads, for example, "Passed · 3/3 steps · 2m 58s." The first time a test ever passes, you'll get a little "Test passed for the first time!" celebration.

A failing run

A red Failed run pinpoints the exact step that failed and shows the evidence the agent gathered:

• Step checklist — steps before the failure are ticked; the failing step is marked with a red ✗; later steps are skipped (they were never reached).
• Expected vs Actual — what the test was looking for versus what actually happened.
• What to do — a suggested fix written by the agent.
• Screenshots — when the agent captured the screen, a screenshot of the moment of failure appears; click it to enlarge and step through any others.
• Audit footer — the run ID, which Agent/Model ran it, the agent's CLI version, and a failure category (e.g. assertion_failed, timeout).

Run history

Recent runs are kept (the panel shows the latest 10; your Mac keeps the full history under Activity). The most recent is shown expanded; click Show N more runs to review older ones.

Recording a result by hand

You can Record a result manually (Pass or Fail, with optional screenshots) from a test's run history without running on a Mac — useful when you have no paired Mac or are reviewing evidence gathered elsewhere. Manual runs are marked Manual to distinguish them from agent-executed runs.

9 Use the desktop app

Most of the time AppDash AI Link just runs quietly in your menu bar. When you want to check on it, its four tabs cover connections, run history, low-level logs, and settings.

Connections — your paired sites

The Connections tab (shown in section 3) lists every paired Jira site under Sites, each as a card with a live status dot. Expand a card for its actions:

• Test — run a quick connectivity check against that site.
• Jobs — jump to the Activity tab filtered to this site.
• Network log — jump to the Console tab filtered to this site.
• Pause / Resume — temporarily stop (or restart) listening for that site.
• Remove — unpair the site and delete its locally stored tokens (this can't be undone — you'd re-pair to reconnect).

Activity — your run history

The Activity tab is a searchable table of every test run this Mac has executed: when it started, its status, type, who triggered it, the issue, the verdict, and how long it took (a Source column also appears once you have more than one connection). Click a row for the full detail, including the step list and the agent's transcript. While a run is still in progress, open it and click Stop to cancel it — the run is stopped and recorded as cancelled in Jira. Use Clear Log to empty the table.

Console — the technical log

The Console tab is for diagnosing problems. It has three sub-views in a left list: Network log (the app's HTTP calls to Jira, agent subprocess launches, and pairing messages — colour-coded by success or error; routine background "heartbeat" polls are hidden by default, use the Show menu to reveal them; click a row to inspect the full request and response, or a run's launch to read its conversation), Audit log (the server-side security ledger for your Atlassian connections — pairings, token rotations, scope checks), and Events (a local record of actions on this Mac — pausing/resuming connections, settings changes, sleep/wake).

Settings — agents and preferences

The Settings tab opens on Agents, where you choose and configure the agent that runs your tests.

• Default agent — the agent used for any run that doesn't pin its own (Claude Code out of the box). Click Set as default agent on a healthy agent to switch.
• Profiles — each agent ships with a built-in Default profile and can have any number of named profiles. A test in Jira can pin a named profile by name; the Default runs otherwise.
• Add custom agent — point AppDash AI Link at any coding-agent CLI (name, command, output format, launch template).
• Open a profile (the › arrow) to configure it: its binary (auto-detected, Re-detect, or Browse), an optional working directory (blank = a fresh sandbox each run; a real path lets the agent touch project files), the launch command template, per-profile environment variables (which can be marked secret), advanced timeouts, a live command preview, and a Test connection button.

Other items in the Settings sidebar: Relay clients (pair an iPhone for read-only status checks — off by default), Project secrets (store credentials for your tests — see section 10), and About (version, update check, and the Forget me data-erasure wizard at the bottom).

10 Use secrets in your tests

Real tests often need a password, an API token, or an internal URL — and you don't want those typed into the test, which is stored in Jira. Secrets solve this: store the value once on your Mac, give it a name, and reference it in a step as {{secrets.NAME}}. Your Mac swaps in the real value at run time, so Jira only ever sees the name.

How a secret flows

A secret's value never leaves your Mac — only its name travels to Jira, so the editor can offer it as a suggestion:

Step 1 · Store the secret on your Mac

Secrets live in the desktop app, grouped by Jira project. Open AppDash AI Link, go to the Settings tab, and choose Project secrets from the sidebar (under AI).

1. Open the Add-secret form. Click Add secret on a project's group, or Add secrets for another project… at the bottom to start a brand-new project.
2. Name it and paste the value. Give the secret a name — letters, digits, hyphens, and underscores, e.g. STD_PW — and paste its value into the masked field. Click Add. The value is written straight to your keychain.

   

3. Match the project key. A secret is scoped to one Jira project key — the prefix of your issue keys (the ENG in ENG-123). A test on an ENG issue can only use secrets stored under ENG, so when you add secrets for a new project, type that exact key in the Project key field.

Step 2 · Reference it in a Jira test

In the BDD editor, type {{ in any step body and an autocomplete list of your project's secret names appears. Pick one to insert a complete {{secrets.NAME}}. Secrets resolve in both test types — BDD steps and Free-Form Task fields (the Task and Success-criteria boxes).

A step that references a secret is tagged with a small secret badge (a shield), so it's easy to see which steps depend on a stored value. The suggested names are the union from every Mac paired to this project — so you can reference a teammate's secret too; it simply resolves on whichever Mac runs the test.

What happens when the test runs

The moment you press Run test, three things happen on your Mac — never in the cloud:

• Resolve. Just before launching the agent, your Mac looks up each {{secrets.NAME}} in the keychain (for the issue's project) and swaps in the real value. The agent runs against the real credentials.
• Fail closed. If a referenced secret isn't stored on the Mac running the test, the run doesn't proceed with a blank — it comes back Errored with a secret-resolution message naming what's missing. Add the secret and run again.
• Redact. The agent's output can echo a value back (say, printing a URL it fetched). Before any result leaves your Mac, every resolved value is replaced with its {{secrets.NAME}} placeholder — across step messages, the verdict, and failure detail alike — so it can't surface in a Jira comment, the panel, or the run history.

Manage your secrets

11 Security & privacy

VibeTest is built so that your most sensitive assets — source code, credentials, and customer data — never leave your control. This section summarises the architecture and controls a corporate security review typically asks about.

Where your data lives

There are only two places VibeTest keeps anything, and you own both of them:

Security controls at a glance

A reviewer-friendly summary of how VibeTest handles the questions that come up most often:

Jira permissions (scopes) the app requests

VibeTest asks for only what it needs to read issues, post results, and store its own data — five Jira scopes in total:

The Jira side runs on Atlassian's hosted runtime

The Jira side of VibeTest runs entirely inside Atlassian's hosted, sandboxed runtime, with per-tenant data isolation and Atlassian-managed encryption — there is no separate AppDash-hosted cloud to assess. (The test-execution agent runs on your own Mac, not on Atlassian — so the product as a whole is not an Atlassian "Runs on Atlassian" app; this section describes only the Jira-hosted half.) For Atlassian's platform certifications, sub-processors, and data-residency options, point your reviewers to the Atlassian Trust Center and the app's Marketplace listing.

Your responsibilities

Because tests run on your machine with your agent, a few things stay in your hands:

• Protect the Mac. Anyone with access to the paired Mac can run tests against the connected site. Use full-disk encryption (FileVault) and a screen lock.
• Mind what a test can reach. The agent runs with your local permissions and logins. Review scenarios that touch production systems, and prefer non-production accounts where you can.
• Use secrets, not plain text. Reference credentials with {{secrets.NAME}} (see Use secrets in your tests) so they aren't written into the test definition stored in Jira.

12 Troubleshooting

Most issues come down to a connection that needs attention or the desktop app not running. Start with the status dot on the Connections tab.

Connection statuses and what to do

Check the wiring: Test connection

To confirm your Mac can reach the AI agent, open Settings → Agents, select your agent environment, and click Test connection. It runs a tiny PONG prompt through the agent's full launch path. A "Ready ✓" result (with the round-trip time) means the desktop-to-agent link works; a "Failed" result points at an agent that isn't installed or configured — re-check the environment's command and binary path on the same screen.

Common problems

For a deeper look, the Console tab shows the exact requests and errors, and its Pair history sub-view records why past pairings succeeded or failed.

Privacy: erase everything

Everything VibeTest stores on your Mac — connections, tokens, activity, and logs — lives there, not in the cloud. To wipe it all, open Settings → About → Forget me and confirm. This permanently deletes the local data and unpairs your sites.