Test Event Integration

Report test execution events from your automation framework to see live results in the Dashboard.

Overview

Device Farm provides a REST API for reporting test execution events. When you integrate your test framework with this API, you get:

• Live Progress - See test results in real-time as they execute on the Dashboard via WebSocket
• Historical Analysis - All results are stored for reporting and trends
• Progress Tracking - Track test execution progress with estimated remaining time

Authentication

All requests must include the X-API-Key header with a valid API key. This is the same key used for /wd/hub Appium/WebDriver endpoints.

X-API-Key: YOUR_API_KEY

API Endpoints

Report Single Event

POST /df/api/automation/sessions/{sessionId}/test-events

Request Body:

{
  "eventId": "uuid-v4",       // Optional: Client-generated UUID for idempotency
  "type": "TEST_STARTED",     // Required: Event type
  "testName": "testLogin",    // Required: Test method/function name
  "suiteName": "LoginTests",  // Optional: Test class/suite name
  "duration": 1234,           // Optional: Duration in milliseconds (for completed tests)
  "timestamp": "2025-01-01T12:00:00Z", // Optional: Event timestamp (defaults to server time)
  "error": {                  // Optional: Error details (for TEST_FAILED)
    "message": "Expected true but got false",
    "stackTrace": "at LoginTest.testLogin(LoginTest.java:42)..."
  }
}

Response (201 Created):

{
  "eventId": "generated-or-provided-uuid"
}

Response (200 OK - Duplicate):

{
  "eventId": "provided-uuid",
  "duplicate": true
}

Report Batch Events

For efficiency, you can report multiple events in a single request:

POST /df/api/automation/sessions/{sessionId}/test-events/batch

Request Body:

{
  "events": [
    { "type": "TEST_STARTED", "testName": "test1", "suiteName": "Suite1" },
    { "type": "TEST_PASSED", "testName": "test1", "suiteName": "Suite1", "duration": 500 },
    { "type": "TEST_STARTED", "testName": "test2", "suiteName": "Suite1" }
  ]
}

Response:

{
  "processed": 3,
  "duplicates": 0,
  "eventIds": ["uuid1", "uuid2", "uuid3"]
}

Report Session Result

After all tests complete, report the final session result:

PUT /df/api/automation/sessions/{sessionId}/test-events/result

Request Body:

{
  "status": "passed",
  "reason": "All tests passed"
}

Field	Type	Description
status	string	passed or failed
reason	string	Summary message (e.g. "3 test(s) failed")

This marks the session with a final pass/fail status in the Dashboard and Reports.

Get Events

Retrieve all events for a session:

GET /df/api/automation/sessions/{sessionId}/test-events

Get Progress

Get current execution progress summary:

GET /df/api/automation/sessions/{sessionId}/test-events/progress

Response:

{
  "sessionId": "session-uuid",
  "total": 10,
  "completed": 5,
  "passed": 4,
  "failed": 1,
  "skipped": 0,
  "currentTest": "testPayment",
  "progress": 0.5,
  "estimatedRemainingSeconds": 120
}

Event Types

Test Events

Type	When to Send	Required Fields
TEST_STARTED	When a test begins	testName
TEST_PASSED	When a test passes	testName, duration
TEST_FAILED	When a test fails	testName, duration, error
TEST_SKIPPED	When a test is skipped	testName

Suite Events

Type	When to Send	Required Fields
SUITE_STARTED	When a test suite begins	suiteName
SUITE_FINISHED	When a test suite completes	suiteName

Suite events are optional but help the Dashboard group and display test results by suite. Send SUITE_STARTED before the first test in a suite and SUITE_FINISHED after the last test completes.

Integration Examples

Java (TestNG)

public class DeviceFarmReporter implements ITestListener {
    private final String baseUrl;
    private final String sessionId;
    private final String apiKey;
    private final HttpClient client = HttpClient.newHttpClient();

    public DeviceFarmReporter(String baseUrl, String sessionId, String apiKey) {
        this.baseUrl = baseUrl;
        this.sessionId = sessionId;
        this.apiKey = apiKey;
    }

    @Override
    public void onTestStart(ITestResult result) {
        sendEvent(new TestEvent("TEST_STARTED", getTestName(result), getSuiteName(result)));
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        sendEvent(new TestEvent("TEST_PASSED", getTestName(result), getSuiteName(result),
            result.getEndMillis() - result.getStartMillis()));
    }

    @Override
    public void onTestFailure(ITestResult result) {
        TestEvent event = new TestEvent("TEST_FAILED", getTestName(result), getSuiteName(result),
            result.getEndMillis() - result.getStartMillis());
        event.setError(result.getThrowable());
        sendEvent(event);
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        sendEvent(new TestEvent("TEST_SKIPPED", getTestName(result), getSuiteName(result)));
    }

    private void sendEvent(TestEvent event) {
        try {
            String json = new ObjectMapper().writeValueAsString(event);
            HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(baseUrl + "/api/automation/sessions/" + sessionId + "/test-events"))
                .header("X-API-Key", apiKey)
                .header("Content-Type", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(json))
                .build();
            client.sendAsync(request, HttpResponse.BodyHandlers.ofString());
        } catch (Exception e) {
            // Log but don't fail test
            System.err.println("Failed to send test event: " + e.getMessage());
        }
    }

    private String getTestName(ITestResult result) {
        return result.getMethod().getMethodName();
    }

    private String getSuiteName(ITestResult result) {
        return result.getTestClass().getName();
    }
}

Python (pytest)

import requests
import pytest
from datetime import datetime

class DeviceFarmReporter:
    def __init__(self, base_url: str, session_id: str, api_key: str):
        self.base_url = base_url
        self.session_id = session_id
        self.api_key = api_key
        self.test_start_times = {}

    def _send_event(self, event: dict):
        try:
            requests.post(
                f"{self.base_url}/api/automation/sessions/{self.session_id}/test-events",
                headers={
                    "X-API-Key": self.api_key,
                    "Content-Type": "application/json"
                },
                json=event,
                timeout=5
            )
        except Exception as e:
            print(f"Failed to send test event: {e}")

    def pytest_runtest_setup(self, item):
        self.test_start_times[item.nodeid] = datetime.now()
        self._send_event({
            "type": "TEST_STARTED",
            "testName": item.name,
            "suiteName": item.module.__name__ if item.module else None
        })

    def pytest_runtest_makereport(self, item, call):
        if call.when != "call":
            return

        start_time = self.test_start_times.get(item.nodeid)
        duration = int((datetime.now() - start_time).total_seconds() * 1000) if start_time else None

        if call.excinfo is None:
            event = {
                "type": "TEST_PASSED",
                "testName": item.name,
                "suiteName": item.module.__name__ if item.module else None,
                "duration": duration
            }
        elif call.excinfo.typename == "Skipped":
            event = {
                "type": "TEST_SKIPPED",
                "testName": item.name,
                "suiteName": item.module.__name__ if item.module else None
            }
        else:
            event = {
                "type": "TEST_FAILED",
                "testName": item.name,
                "suiteName": item.module.__name__ if item.module else None,
                "duration": duration,
                "error": {
                    "message": str(call.excinfo.value),
                    "stackTrace": str(call.excinfo.getrepr())
                }
            }

        self._send_event(event)


# Usage in conftest.py:
def pytest_configure(config):
    base_url = config.getoption("--device-farm-url")
    session_id = config.getoption("--session-id")
    api_key = config.getoption("--api-key")

    if all([base_url, session_id, api_key]):
        reporter = DeviceFarmReporter(base_url, session_id, api_key)
        config.pluginmanager.register(reporter, "device_farm_reporter")

JavaScript (Mocha)

const axios = require('axios');

class DeviceFarmReporter {
  constructor(runner, options) {
    const { baseUrl, sessionId, apiKey } = options.reporterOptions;
    this.baseUrl = baseUrl;
    this.sessionId = sessionId;
    this.apiKey = apiKey;
    this.testStartTimes = new Map();

    runner.on('test', (test) => this.onTestStart(test));
    runner.on('pass', (test) => this.onTestPass(test));
    runner.on('fail', (test, err) => this.onTestFail(test, err));
    runner.on('pending', (test) => this.onTestSkipped(test));
  }

  async sendEvent(event) {
    try {
      await axios.post(
        `${this.baseUrl}/api/automation/sessions/${this.sessionId}/test-events`,
        event,
        {
          headers: {
            'X-API-Key': this.apiKey,
            'Content-Type': 'application/json',
          },
          timeout: 5000,
        },
      );
    } catch (error) {
      console.error('Failed to send test event:', error.message);
    }
  }

  onTestStart(test) {
    this.testStartTimes.set(test.fullTitle(), Date.now());
    this.sendEvent({
      type: 'TEST_STARTED',
      testName: test.title,
      suiteName: test.parent?.title || null,
    });
  }

  onTestPass(test) {
    const startTime = this.testStartTimes.get(test.fullTitle());
    const duration = startTime ? Date.now() - startTime : test.duration;
    this.sendEvent({
      type: 'TEST_PASSED',
      testName: test.title,
      suiteName: test.parent?.title || null,
      duration,
    });
  }

  onTestFail(test, err) {
    const startTime = this.testStartTimes.get(test.fullTitle());
    const duration = startTime ? Date.now() - startTime : test.duration;
    this.sendEvent({
      type: 'TEST_FAILED',
      testName: test.title,
      suiteName: test.parent?.title || null,
      duration,
      error: {
        message: err.message,
        stackTrace: err.stack,
      },
    });
  }

  onTestSkipped(test) {
    this.sendEvent({
      type: 'TEST_SKIPPED',
      testName: test.title,
      suiteName: test.parent?.title || null,
    });
  }
}

module.exports = DeviceFarmReporter;

WebDriverIO (WDIO)

WebDriverIO integrates with Device Farm through built-in hooks. The beforeTest and afterTest hooks report events automatically, and after reports the final session result.

// wdio.conf.js
const API_KEY = process.env.DF_API_KEY;
const BASE_URL = process.env.DF_URL || 'https://api.rabbitqa.com/df';

async function sendEvent(sessionId, body) {
  try {
    await fetch(`${BASE_URL}/api/automation/sessions/${sessionId}/test-events`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'X-API-Key': API_KEY },
      body: JSON.stringify(body),
    });
  } catch (e) {
    console.warn('Event send failed:', e.message);
  }
}

async function reportResult(sessionId, status, reason) {
  try {
    await fetch(`${BASE_URL}/api/automation/sessions/${sessionId}/test-events/result`, {
      method: 'PUT',
      headers: { 'Content-Type': 'application/json', 'X-API-Key': API_KEY },
      body: JSON.stringify({ status, reason }),
    });
  } catch (e) {
    console.warn('Result report failed:', e.message);
  }
}

exports.config = {
  runner: 'local',
  hostname: 'api.rabbitqa.com',
  port: 443,
  protocol: 'https',
  path: '/df/wd/hub',

  specs: ['./test/specs/**/*.spec.js'],
  maxInstances: 1,

  capabilities: [{
    platformName: 'android',
    'appium:automationName': 'UiAutomator2',
    'df:options': {
      buildName: `build-${process.env.BUILD_NUMBER || 'local'}`,
      projectName: 'Regression Tests',
      sessionName: 'smoke-android',
      tags: ['regression', 'android'],
    },
  }],

  headers: { 'X-API-Key': API_KEY },

  framework: 'mocha',
  reporters: ['spec'],
  mochaOpts: { ui: 'bdd', timeout: 60000 },

  async beforeTest(test) {
    const sid = browser.sessionId;
    if (!sid) return;
    await sendEvent(sid, {
      type: 'TEST_STARTED',
      testName: test.title,
      suiteName: test.parent,
    });
  },

  async afterTest(test, context, { error, duration, passed }) {
    const sid = browser.sessionId;
    if (!sid) return;
    const type = passed ? 'TEST_PASSED' : error ? 'TEST_FAILED' : 'TEST_SKIPPED';
    await sendEvent(sid, {
      type,
      testName: test.title,
      suiteName: test.parent,
      duration,
      ...(error ? { error: { message: error.message, stackTrace: error.stack } } : {}),
    });
  },

  async after(result) {
    const sid = browser.sessionId;
    if (!sid) return;
    const failed = result > 0;
    await reportResult(
      sid,
      failed ? 'failed' : 'passed',
      failed ? `${result} test(s) failed` : 'All tests passed'
    );
  },
};

df:options

Use df:options in your capabilities to organize test runs by build, project, and tags. See Appium Config for all available options.

cURL Examples

# Report TEST_STARTED event
curl -X POST "https://api.rabbitqa.com/api/automation/sessions/{sessionId}/test-events" \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "TEST_STARTED",
    "testName": "testLogin",
    "suiteName": "LoginTests"
  }'

# Report TEST_PASSED event
curl -X POST "https://api.rabbitqa.com/api/automation/sessions/{sessionId}/test-events" \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "TEST_PASSED",
    "testName": "testLogin",
    "suiteName": "LoginTests",
    "duration": 1250
  }'

# Report TEST_FAILED event with error
curl -X POST "https://api.rabbitqa.com/api/automation/sessions/{sessionId}/test-events" \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "TEST_FAILED",
    "testName": "testPayment",
    "suiteName": "PaymentTests",
    "duration": 3500,
    "error": {
      "message": "Expected status 200 but got 500",
      "stackTrace": "at PaymentTest.testPayment(PaymentTest.java:42)..."
    }
  }'

Best Practices

Use eventId for Idempotency

If your test framework might retry requests (e.g., network issues), always include a client-generated UUID as eventId. Duplicate events with the same ID are ignored.

Always Send TEST_STARTED

Send TEST_STARTED before each test runs. This enables real-time "currently running" indicator, accurate duration calculation, and better progress tracking.

Include suiteName

Group tests by suite/class for better organization in the Dashboard. Use the full class name (e.g., com.example.tests.LoginTests) for clear categorization.

Include Error Details

For failed tests, include both the error message and full stack trace. This helps with debugging directly from the Dashboard without switching to your IDE.

Error Handling

HTTP Status Codes

Code	Meaning
201	Event recorded successfully
200	Duplicate event (idempotent - same eventId)
401	Invalid or missing API key
403	Session belongs to a different company
404	Session not found
422	Session not accepting events (wrong type or ended)

Grace Period

Events are accepted for:

• Active sessions - Always accepted
• Ended sessions - Up to 5 minutes after session ends

This grace period ensures late-arriving events (e.g., final test results after session timeout) are not lost.

Troubleshooting

Events Not Appearing in Dashboard

1. Check session ID - Ensure you're using the correct session ID from the /wd/hub/session response
2. Check API key - Verify the API key is valid and has access to the session's company
3. Check session type - Only automation sessions (type: AUTOMATION) accept test events
4. Check WebSocket - Dashboard must be connected via WebSocket to receive live updates

Duplicate Events

If you're seeing duplicate events, ensure you're using a consistent eventId for each test event. The server ignores events with duplicate IDs within the same session.

Missing Duration

Duration is calculated from the TEST_STARTED event timestamp to the completion event timestamp. If TEST_STARTED is not sent, duration may be missing or inaccurate.