Public documentation

Monitor

Monitor connects local Roleplay runs, CI uploads, and scheduled social-engineering checks to the workbench.

Monitor should only show workflow commands after a project, protected agent, and project API key exist. Raw API key values are shown once at creation; existing keys are displayed as masked previews.

The recommended recurring workflow is:

Run attack simulations in your environment.
Preserve the original pass/fail result.
Upload sanitized findings to workbench.
Compare the result with the original boundary, regression key, and severity.
Fail the job, reopen the finding, or flag the monitor when the failure returns.

Upload Mode

Sanitized findings are the default:

roleplay upload all --source ci --mode sanitized_findings

Full transcript upload requires both:

CLI opt-in: --mode full_transcript_opt_in
Project policy opt-in in workbench

If project policy does not allow full transcript upload, the CLI fails before reading or sending full transcript artifacts.

Required CI Secrets

Set:

ROLEPLAY_CLOUD_URL=https://app.roleplay.sh
ROLEPLAY_PROJECT_ID=<project-id>
ROLEPLAY_API_KEY=<project-api-key>
ROLEPLAY_LLM_PROVIDER=<provider>
ROLEPLAY_JUDGE_MODE=hybrid
ROLEPLAY_JUDGE_PROVIDER=<provider>
ROLEPLAY_<PROVIDER>_API_KEY=<provider-api-key>

Use your deployed workbench URL for ROLEPLAY_CLOUD_URL. When testing locally, use:

ROLEPLAY_CLOUD_URL=http://127.0.0.1:3000

ROLEPLAY_LLM_API_KEY is a provider-neutral placeholder for CI examples. If your selected provider requires a named key variable, replace it with the matching provider-specific secret shown in the CLI environment variable reference.

Optional:

ROLEPLAY_AGENT_NAME=support-agent-staging
ROLEPLAY_ENVIRONMENT=staging
ROLEPLAY_TARGET_URL=https://staging.example.com/agent
ROLEPLAY_TARGET_COMMAND=
ROLEPLAY_ATTACKER_PROVIDER=
ROLEPLAY_LLM_MODEL=
ROLEPLAY_JUDGE_MODEL=
ROLEPLAY_BUILD_URL=

GitHub Actions Example

name: Roleplay

on:
  pull_request:
  push:
    branches: [main]

jobs:
  social-engineering:
    runs-on: ubuntu-latest
    env:
      ROLEPLAY_CLOUD_URL: https://app.roleplay.sh
      ROLEPLAY_PROJECT_ID: ${{ secrets.ROLEPLAY_PROJECT_ID }}
      ROLEPLAY_API_KEY: ${{ secrets.ROLEPLAY_API_KEY }}
      ROLEPLAY_TARGET_URL: http://localhost:3000/agent
      ROLEPLAY_AGENT_NAME: support-agent-staging
      ROLEPLAY_LLM_PROVIDER: ${{ secrets.ROLEPLAY_LLM_PROVIDER }}
      ROLEPLAY_JUDGE_MODE: hybrid
      ROLEPLAY_JUDGE_PROVIDER: ${{ secrets.ROLEPLAY_JUDGE_PROVIDER }}
      ROLEPLAY_LLM_API_KEY: ${{ secrets.ROLEPLAY_LLM_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: corepack enable
      - name: Start agent
        run: npm run start-agent &
      - name: Run social-engineering attack pack
        id: roleplay
        continue-on-error: true
        run: corepack pnpm dlx @roleplay-sh/cli run social-engineering-core --judge hybrid --fail-on critical
      - name: Upload sanitized findings
        if: always()
        run: corepack pnpm dlx @roleplay-sh/cli upload all --source ci --mode sanitized_findings
      - name: Fail when Roleplay found critical regressions
        if: steps.roleplay.outcome == 'failure'
        run: exit 1

GitLab CI Example

roleplay-social-engineering:
  image: node:22
  variables:
    ROLEPLAY_CLOUD_URL: "https://app.roleplay.sh"
    ROLEPLAY_PROJECT_ID: "$ROLEPLAY_PROJECT_ID"
    ROLEPLAY_API_KEY: "$ROLEPLAY_API_KEY"
    ROLEPLAY_TARGET_URL: "http://localhost:3000/agent"
    ROLEPLAY_AGENT_NAME: "support-agent-staging"
    ROLEPLAY_LLM_PROVIDER: "$ROLEPLAY_LLM_PROVIDER"
    ROLEPLAY_JUDGE_MODE: "hybrid"
    ROLEPLAY_JUDGE_PROVIDER: "$ROLEPLAY_JUDGE_PROVIDER"
    ROLEPLAY_LLM_API_KEY: "$ROLEPLAY_LLM_API_KEY"
  script:
    - corepack enable
    - set +e
    - corepack pnpm dlx @roleplay-sh/cli run social-engineering-core --judge hybrid --fail-on critical
    - ROLEPLAY_STATUS=$?
    - corepack pnpm dlx @roleplay-sh/cli upload all --source ci --mode sanitized_findings
    - exit $ROLEPLAY_STATUS

CircleCI Example

version: 2.1

jobs:
  roleplay:
    docker:
      - image: cimg/node:22.0
    environment:
      ROLEPLAY_LLM_PROVIDER: ${ROLEPLAY_LLM_PROVIDER}
      ROLEPLAY_JUDGE_MODE: hybrid
      ROLEPLAY_JUDGE_PROVIDER: ${ROLEPLAY_JUDGE_PROVIDER}
      ROLEPLAY_LLM_API_KEY: ${ROLEPLAY_LLM_API_KEY}
    steps:
      - checkout
      - run: corepack enable
      - run:
          name: Run Roleplay
          command: |
            set +e
            corepack pnpm dlx @roleplay-sh/cli run social-engineering-core --judge hybrid --fail-on critical
            ROLEPLAY_STATUS=$?
            corepack pnpm dlx @roleplay-sh/cli upload all --source ci --mode sanitized_findings
            exit $ROLEPLAY_STATUS

First Upload

Use the included local runner locally or in CI, then upload sanitized findings with the project ID and project API key. The Monitor screen shows the latest received test run after the first upload lands.

Verification And Monitoring

After a failed finding is fixed, rerun the same scenario or pack with the same regression key. The workbench should make the result clear:

verified fixed: the protected boundary now holds;
still failing: the same failure remains;
regressed: a previously fixed boundary failed again.

For recurring checks, use the same CI command on a schedule or release workflow. Specialized packs should preserve vertical metadata so returned regressions can be grouped by agent role, external actor, business boundary, action risk, and data sensitivity.

Health Check

roleplay doctor --cloud --cloud-url https://app.roleplay.sh

For local development:

roleplay doctor --cloud --cloud-url http://127.0.0.1:3000

Credential verification:

ROLEPLAY_PROJECT_ID=<project-id> \
ROLEPLAY_API_KEY=<api-key> \
roleplay doctor --cloud --cloud-url https://app.roleplay.sh