feat: add test run execution, stats, and metrics support - #2
Merged
Conversation
- Change status from Object to String with NameStringDeserializer to
handle both plain strings and nested {"name":"..."} objects from API
- Add createdAt, nanoId, testCount, passRate fields
- Update EntityTest to use builder pattern for TestRun construction
- Add ExecutionMode enum (Parallel/Sequential) - Add ExecutionRequest model with nested ExecutionOptions - Add TestSetClient.execute() and rescore() for triggering test runs - Add TestSetClient.lastRun() to fetch most recent completed run - Add metric management: getMetrics(), addMetric(), removeMetric() - Add test association: addTests(), removeTests()
- Add TestRunStatsMode and TestResultStatsMode enums - Add typed response models: TestRunStats, TestResultStats, and all nested models (StatusDistribution, ResultDistribution, MetricStats, OverallStats, TimelineData, TestRunSummary, etc.) - Update TestRunClient.stats() to return typed TestRunStats with overloads for mode, run IDs, and full filter params - Add TestResultClient.stats() with mode and 18 filter parameters matching the Python SDK
- Add 14 WireMock tests: execute, rescore, lastRun, nested status deserialization, typed run/result stats with modes and filters, metrics, test association - Add TestRunIntegrationTest with 18 tests covering run lifecycle, stats modes (summary, status, all), result stats (metrics, behavior, category, overall), filtered queries, execution, and last run
- Add ExecuteTestSetExample: parallel, sequential, custom metrics - Add TestRunWorkflowExample: list runs, results, stats, last run, rescore - Add TestSetMetricsExample: metric and test association management - Add TestRunStatsExample: summary, distribution, timeline, filtering - Add TestResultStatsExample: pass rates by metric, behavior, category, topic, per-run summaries - Add examples README with setup instructions and quick reference - Update project README with new features and all 9 examples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
execute,rescore,lastRun) toTestSetClient, matching the Python SDK'sTestSet.execute()/TestSet.rescore()/TestSet.last_run()capabilitiesTestRunClient.stats(),TestResultClient.stats()) with mode enums and full filter parameter supportgetMetrics,addMetric,removeMetric) and test association (addTests,removeTests) toTestSetClientTestRun.statusdeserialization to handle both plain strings and nested objects from the APIChanges
New files (27)
ExecutionMode,TestRunStatsMode,TestResultStatsModeenumsExecutionRequestmodel with nestedExecutionOptionsentities/stats/(TestRunStats,TestResultStats,MetricStats,OverallStats,StatusDistribution,ResultDistribution,TimelineData, etc.)TestRunIntegrationTest(18 tests against live API)README.mdModified files (7)
TestSetClient— execute, rescore, lastRun, metrics, test associationTestRunClient— typedstats()with mode/filter overloadsTestResultClient— typedstats()with 18 filter parametersTestRun— status deserialization fix, new fieldsClientWiremockTest— 14 new WireMock testsEntityTest— updated for new TestRun signatureREADME.md,CHANGELOG.md,pom.xml— docs and version bump to 0.1.2Test plan