CTT conformance: access rights, X509, aggregates, and matrix diagnostics#3978
Merged
Conversation
…ess Space WriteMask) The static Int32 test node carries explicit RolePermissions granting Browse|Read|Write|ReadHistory|ReadRolePermissions but not WriteAttribute. The CTT Address Space WriteMask unit writes the non-Value attributes the node's WriteMask advertises (DisplayName/Description); those require PermissionType.WriteAttribute. Because RolePermissions are enforced for every attribute once present, the authenticated user lacking WriteAttribute got BadUserAccessDenied (126 failures) instead of letting the per-attribute WriteMask decide Good/BadNotWritable. Added WriteAttribute to kTestNodePermissions so the WriteMask governs the result. Added AddressSpaceWriteMaskTests regression (write Description must not be BadUserAccessDenied). Green net48+net10.
…ings Security User X509 cluster (29): the SessionManager brute-force lockout (MaxFailedAuthenticationAttempts, default 5, keyed on the shared client application-instance certificate thumbprint) trips partway through the CTT's X509 negative-token suite - every rejected attempt counts, even the ones the CTT expects to fail - so from test 005 on every ActivateSession returns BadUserAccessDenied, masking the real per-token result (incl. the positive 011/013 cases that expect Good). Lockout is vendor hardening, not an OPC UA conformance requirement, so disable it for compliance testing via MaxFailedAuthenticationAttempts=0 in Ctt.ReferenceServer.Config.xml (production keeps the default 5). Behaviour proven by ClientLockoutTests.ClientIsNotLockedOutWhenLockoutDisabledAsync. plans/ctt-issues.md: add CTT-script defect #5 (HAAggregateHelper.js:1484 dereferences possibleNodeId without the isDefined guard used elsewhere) and a run5 findings section (X509 lockout, cert-trust provisioning for valid-cert 037/044/051/052, and the aggregate server-vs-CTT value comparison that needs the CTT loop to attribute).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves OPC UA CTT (Compliance Test Tool) conformance for the reference server by fixing an access-rights gap affecting non-Value attribute writes governed by WriteMask, disabling authentication lockout in the dedicated CTT server config to avoid masking X509-token results, and documenting identified CTT-side script defects and run findings.
Changes:
- Add a regression test ensuring
WriteMask-writable non-Value attribute writes are not rejected asBadUserAccessDenied. - Update the reference server conformance node’s explicit
RolePermissionsto includePermissionType.WriteAttribute. - Disable
MaxFailedAuthenticationAttemptsinCtt.ReferenceServer.Config.xmland document remaining CTT-side defects/findings inplans/ctt-issues.md.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| Tests/Opc.Ua.InformationModel.Tests/AddressSpaceWriteMaskTests.cs | Adds a regression test to prevent BadUserAccessDenied for WriteMask-writable non-Value attribute writes. |
| plans/ctt-issues.md | Documents an additional CTT script defect and summarizes “latest run” residual findings/attribution notes. |
| Applications/Quickstarts.Servers/ReferenceServer/ReferenceNodeManager.cs | Extends explicit RolePermissions for Scalar_Static_Int32 to include WriteAttribute so WriteMask governs non-Value writes. |
| Applications/ConsoleReferenceServer/Ctt.ReferenceServer.Config.xml | Disables auth lockout for CTT runs to prevent false BadUserAccessDenied cascades in negative-token suites. |
…nt32 The comment enumerated Browse+Read+Write+ReadHistory+ReadRolePermissions (omitting the newly added WriteAttribute) and framed WriteAttribute as a SecurityAdmin-only concern. WriteAttribute is part of kTestNodePermissions and is granted to Anonymous and AuthenticatedUser; SecurityAdmin simply gets the full permission set. Comment-only change, no behavior change.
…store Investigates the X509-only CTT re-run (Security User X509, 17 errors) taken after the auth-lockout fix. Server fix (CTT 018): a user-token signature built with a wrong algorithm returned the channel-level BadSecurityChecksFailed. SecurityPolicies. VerifySignatureData (shared with the channel signature checks) throws BadSecurityChecksFailed for an unexpected SignatureData.Algorithm, and X509IdentityTokenHandler.VerifyAsync propagated it unchanged. Map that algorithm-mismatch BadSecurityChecksFailed to the token-level BadIdentityTokenInvalid in the user-token path (Part 4), leaving the shared channel-signature behaviour untouched. Regression test added. Reference server enhancement (ease provisioning): the remaining 16 errors are the CTT's trusted user certs not being in the server's pki/trustedUser store (the server correctly rejects untrusted user certs per Part 4). The reference server now persists every rejected X509 user certificate to a dedicated pki/rejectedUser review store (sibling of pki/trustedUser) so an operator can move the legitimate ones into the trusted-user store and re-run. Also refactors VerifyX509IdentityToken to async (removing the sync-over-async validation call). Integration test + README note added. plans/ctt-issues.md: document CTT-script defect #6 (the '// to prevent user lockout' cleanup ActivateSession lacks error suppression, so it can fail an otherwise-passing negative test - UaR.js:219) and the run6 provisioning / rejectedUser workflow, with OPC UA spec references. Validated on net48 + net10 (X509IdentityTokenHandlerTests, SecurityX509UserTests).
marcschier
changed the title
Root-causes the 6016-error aggregate cluster using the CTT console log (Server Value / CTT Value diagnostics). Under UseServerCapabilitiesDefaults the CTT reads the node's HistoricalDataConfiguration -> AggregateConfiguration object and computes aggregates with those values, then compares to the server's ReadProcessed. The server computes with PercentDataGood/Bad=100, TreatUncertainAsBad=true (a partial no-data interval => Uncertain), but HistoricalDataConfigurationInstaller.PopulateProperties materialised the mandatory AggregateConfiguration child WITHOUT populating it, leaving the type's all-zero defaults (0/0/false) - both inconsistent with the server's computation and an invalid AggregateConfiguration (PercentDataGood < 100-PercentDataBad). A CTT reading 0/0/false classifies every partial interval as Bad while the server returns Uncertain -> the systematic divergence across every aggregate/node. Fix: HistorianNodeCapabilities gains DefaultAggregateConfiguration (defaulting to the Part 13 v1.05.07 4.2.1.2 defaults 100/100/true/false); the installer populates the AggregateConfiguration object from it, so a UseServerCapabilitiesDefaults client reads the same config the server computes with. Also fixes a latent client bug: HistoryClient.GetConfigurationAsync resolved the companion object via HasAddIn instead of HasHistoricalConfiguration (Part 11 5.2.3) and never read AggregateConfiguration; it now does both. Regression: HistoryClientIntegrationTests.GetConfigurationReturnsHistoricalData ConfigurationAsync asserts the advertised PercentDataGood/Bad=100, TreatUncertainAsBad=true. plans/ctt-issues.md documents the root cause + the remaining CTT-script items. Validated net48 + net10.
Alarms & Conditions (152): using the console log (ctt output 1.txt) - the 'After Acknowledge Retain in invalid state' (42) is spec-correct server behaviour: core GetRetainState returns false for an inactive+acked+confirmed condition with no branches and UpdateRetainState precedes ReportStateChange, so Retain=true reflects an outstanding condition branch (Part 9 5.8.2/5.7.3) that the CTT's ValidateRetain does not account for. ConditionDisabled (60) and Unable-to-read-input (42, AnalogSource is a readable variable) need the live A&C loop to attribute. AliasName (12): the reference server is FindAlias-only (InMemoryAliasNameStore) and does not materialise browsable AliasNameType instance nodes under the categories, so the CTT Category/Hierarchy units (which browse for instances) report 'No instance found' / 'not part of the current category'. Documented as Part 17 profile-dependent (expose instances vs gate the assertion). Plus a genuine CTT-script defect: AliasName Hierarchy/002.js:80 references undefined TC_Variables.ListOfNodes (should be OutputArguments). All with Part 9 / Part 17 references for the CTT developers.
# Conflicts: # Applications/Quickstarts.Servers/ReferenceServer/ReferenceServer.cs
…BadInvalidArgument) Per OPC 10000-11 (Part 11) v1.05.07 6.5.4.2, a processed history read whose StartTime equals EndTime has a zero-width time domain with no meaningful interpretation, so the Server shall return Bad_InvalidArgument. The server previously computed and returned Good, which CTT flags in the aggregate multi-node cluster (aggregate_002_01: 'Server Status [Good] not equal to ctt status [BadInvalidArgument]', and the single-node aggregate_001_01), both driven by the StartEqualsEnd time definition in HAAggregateHelper.PerformMultipleNodeTest / PerformSingleNodeTest. Add a narrow guard at the top of HistorianDispatcher.DispatchProcessedReadAsync that returns Bad_InvalidArgument per operation (matching CTT, which expects the HistoryRead service call to succeed with per-node BadInvalidArgument results rather than a service fault). Covers both async and sync custom node manager read paths. Tests (net10 + net48): faithful multi-node service-path reproduction (HistoryReadProcessedWithEqualStartAndEndTimeReturnsBadInvalidArgumentPerNodeAsync) plus dispatcher-level equal-time and reverse-order guard tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Feed reverse processed history reads in request order, preserve exact raw endpoints during interpolation, and honor uncertain classification and repeated Good quality semantics. Add direct calculator and live historian Part 11/13 oracle coverage for focused CTT aggregate scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add deterministic Part 6 matrix wire and server NumericRange tests, and document the run12 CTT failure cascades and script defects. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover Good, Uncertain, and Bad raw values exactly on a reverse interpolative slice boundary, including value, status bits, and timestamps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Return exact non-Bad raw values for Interpolative aggregates regardless of TreatUncertainAsBad, while continuing to interpolate across exact Bad points. Cover forward and reverse Good, Uncertain, and Bad boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Include only the request start bound for WorstQuality2 and preserve the first status code at the worst severity in request order. Add forward, reverse, and mixed-status coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Minimum/Maximum (and Minimum2/Maximum2) marked a result Calculated whenever the selected raw sample timestamp differed from the chronological lower slice bound. For reverse reads (startTime > endTime) the request-direction interval start is the later timestamp, so a min/max sample sitting on the returned interval timestamp was wrongly flagged Calculated instead of Good, Raw (Part 13 5.4.3.10/5.4.3.11, interval geometry Part 13 5.4.2.2 / Part 11). Compare the selected sample against GetTimestamp(slice) (the request-direction interval start) in both ComputeMinMax and ComputeMinMax2. Forward reads are unchanged because GetTimestamp(slice) == slice.StartTime there. Add independent forward/reverse oracle tests for Minimum/Maximum, Minimum2/ Maximum2 and the reverse ActualTime variants, and correct the shared CTT regression oracle that encoded the buggy reverse-Maximum expectation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
marcschier
changed the title
Correct one-sided and equal-time reads, enforce modified-read bound validation, count and synthesize Part 11 bounds correctly, and make history continuation points single-use. Add focused unit and integration coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reproduce the reference server's all-Good linear-ramp seed (SeedHistoricalNodeAsync) and pin the server's Part 13 aggregate behaviour for the residual value families reported by CTT run 14. Direct and live in-memory historian reads confirm the server is spec-correct and the residual mismatches are on the CTT oracle side: - Float/Double sloped interpolation bounds (Interpolative/StartBound/EndBound) are byte-exact; CTT only diverges on integer-typed nodes. - Integer interpolation rounds to nearest via Convert.ToInt32 (identical in 1.5.378/master378 and HEAD); Part 13 5.4.3.2.2 does not mandate truncation, so the CTT oracle flooring to the earlier integer is a convention difference, not a server regression. - Duration/Percent/WorstQuality2/DurationInState over all-Good data are full-Good; the CTT expectations assume a configured Bad-data region absent from the all-Good seed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Return the next bound for equal-time exact reads, and synthesize footnote LAST/FIRST bounds when one-sided reads exhaust available history before NumValuesPerNode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ilot/ctt-run5-fixes
romanett
approved these changes
Jul 13, 2026
7 tasks
marcschier
added a commit
that referenced
this pull request
Jul 18, 2026
…lidation (#4002) # Description Follow-up to #3978 based on the latest OPC UA CTT aggregate, matrix, Historical Access, Attribute, and Node Management runs. ## Aggregate run 16 - Ignore synthetic `BadBoundNotFound` raw-bound markers when feeding aggregate calculators. - Add direct and live historian regression coverage. ## Multi-dimensional Variant matrices - Fix `Variant.Copy()` so rank-2 Variant matrices preserve their payload and dimensions. - Enforce valid Variant matrix dimensions consistently across Binary, JSON, XML stream, and XML parser codecs. - Encoders return `BadEncodingError`; decoders return `BadDecodingError` for malformed matrix dimensions. - Add cross-codec valid/null/empty/malformed tests and a batched ReferenceServer Read regression. - Correct the previous CTT issue attribution: the invalid `Scalar_Static_Arrays2D_Variant` server encoding poisoned the batched CTT Read. ## Node Management AddNodes - Return `BadNodeIdRejected` for unsupported requested-NodeId namespaces. - Validate hierarchical ReferenceType source/target NodeClass constraints. - Document run-18 confirmation that all remaining 15 AddNodes errors are CTT project/script configuration issues. ## Historical Access and Attribute Services run 18 - Reject whitespace in `NumericRange` syntax with `BadIndexRangeInvalid`. - Add HistoryRead tests for operation-level invalid syntax and per-DataValue `BadIndexRangeNoData`. - Document CTT defects: - `012.js` expects `BadIndexRangeNoData` at the operation level instead of on returned DataValues; - `Err-012.js` uses a non-historizing node for an access-denied test; - generic Attribute array helpers omit NodeId[] and StatusCode[] conversion paths. ## Base Information model validation run 19 - Run the CTT main session as `sysadmin` over Basic256Sha256 SignAndEncrypt so SecurityAdmin RolePermissions and EncryptionRequired restrictions are both satisfied without weakening the server model. - Preserve predefined well-known NodeIds while making generated dynamic type and optional-child factories allocate compatible per-instance NodeIds, remap internal references, and update references from the owning subtree. - Materialize dynamic RoleType properties through generated helpers and keep role-child string NodeIds disjoint from numeric RoleManager root IDs. - Add net48/net10 regressions for generator output, NodeId allocation/reference remapping, DI mandatory children, dynamic roles, and low-privilege versus SecurityAdmin Browse visibility. - Document CTT defects for permission-unaware mandatory-child validation, the UA 1.04 reference model used against a UA 1.05 server, stale ServerDiagnostics cache, and the `ConformanceUnits` scalar/array mismatch. ## Related Issues - Follow-up to #3960 and #3978. ## Checklist - [x] I have signed the CLA and read the CONTRIBUTING document. - [x] Tests prove the fixes and improve coverage. - [x] Required CTT documentation is updated. - [x] Focused builds/tests introduce no warnings. - [x] Affected tests pass on .NET Framework and .NET 10. - [ ] CI and CodeQL are green. - [ ] All PR feedback is addressed. ## Latest CTT full run and A&C run 20 - Fix Monitor Value Change V2 `036.js`: an ambiguous `DataValue(StatusCode)` call encoded `BadIndexRangeNoData` as a Good value. - Prove all 19 configured matrix monitored items notify when the selected value actually changes; document the independent `042.js` script defects. - Rebase dynamic `CertificateExpired` / `TrustListOutOfDate` alarm roots and descendants, restore standard type declarations and ModellingRules, and expose the cross-node-manager references. - Deliver method-triggered events to every authorized Session, fixing the two-Session Confirm scenario. - Preserve locale-only `LocalizedText` values consistently across Binary, JSON, and XML and align Acknowledge/Confirm comment behavior and tests. - Document the remaining proven CTT defects for TransactionDiagnostics, SemanticChange, History `013`, PolicyId scope, A&C, durable subscriptions, and Subscription Minimum events. ## Base synchronization Merged `master` through `007cfaf90` (`#4010`, repository layout consolidation) and retained the matrix regression tests at their new `tests/` locations. ## Latest validation - Focused merged-tree tests pass on net10 and net48. - `Opc.Ua.Server` and `Quickstarts.Servers` Release builds complete with 0 warnings and 0 errors on net10 and net48. - The existing `DisableEnableViaTypeAndInstanceMethodNoEventsDuringDisabledAsync` timing test took its documented inconclusive timeout path after the layout merge; no test failure was reported. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: romanett <romanett98@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Investigates successive OPC UA CTT runs and fixes the proven server conformance defects while documenting proven CTT script/codec defects in
plans/ctt-issues.md.Server fixes
WriteAttributeonScalar_Static_Int32so attributes advertised byWriteMaskare writable.BadUserAccessDenied.BadIdentityTokenInvalid.pki/rejectedUserto simplify trusted-user provisioning.HistoricalDataConfiguration.AggregateConfigurationwith the server's actual defaults.BadInvalidArgumentfor processed reads with equal StartTime/EndTime.MultipleValues.HistoryClient.GetConfigurationAsynctraversal and aggregate-configuration reads.Tests and CTT evidence
NumericRangecoverage.plans/ctt-issues.mdwith exact evidence and recommended fixes for CTT aggregate, X509, multidimensional-array, A&C, and AliasName script defects.No CTT failure is suppressed or accepted without a spec-based classification.
Related Issues
Checklist