Use env variable (OPENSEARCH_FIPS_MODE) to enable opensearch to run in FIPS enforced mode instead of checking for existence of bcFIPS jars

[cwperks]

Description

This PR updates logic in ./bin/opensearch-env (and /bin/opensearch-env.bat) to change the condition for running in FIPS approved mode. Currently, opensearch-env checks for the existence of bc-fips* jars under the lib/ folder and accordingly adds system props to ensure opensearch enforces FIPS-approved security policy. bc-fips folders are not currently included in the default distribution, and need to be built in from source by creating a distribution with the FIPS build param (-Pcrypto.standard=FIPS-140-3).

When the min distribution and default distribution start building with -Pcrypto.standard=FIPS-140-3 by default, then it would mean that this block is always hit and FIPS approved mode would be enforced. We do not (yet) want that to be default and instead rely on a cluster administrator to explicitly enable that.

This PR introduces an env var (OPENSEARCH_FIPS_MODE) that a cluster admin can use to enable it.

For example

OPENSEARCH_FIPS_MODE=true ./bin/opensearch

Related Issues

Resolves opensearch-project/opensearch-build#5979

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The changes replace automatic FIPS mode detection based on BouncyCastle FIPS JAR presence with an explicit environment variable configuration (OPENSEARCH_FIPS_MODE). Startup scripts for both Unix and Windows now check this variable to enable FIPS, and test configurations are updated to set it for FIPS compliance testing.

Changes

Cohort / File(s)	Summary
Documentation CHANGELOG.md	Added changelog entry documenting the new OPENSEARCH_FIPS_MODE environment variable for FIPS mode configuration.
Startup Scripts distribution/src/bin/opensearch-env, distribution/src/bin/opensearch-env.bat	Replaced automatic FIPS detection (checking for bc-fips JAR files) with explicit OPENSEARCH_FIPS_MODE environment variable. When set to "true" (case-insensitive), enables FIPS by configuring Java security properties and BouncyCastle FIPS approved mode. Removed previous JAR detection logic and fixed minor documentation formatting.
Test Configuration qa/fips-compliance/build.gradle	Added OPENSEARCH_FIPS_MODE=true environment variable to javaRestTest cluster configuration to enable FIPS mode during test execution.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title clearly and concisely describes the main change: introducing OPENSEARCH_FIPS_MODE environment variable instead of checking for bcFIPS jars.
Description check	✅ Passed	The pull request description includes all required sections: detailed Description explaining the changes and rationale, Related Issues linking to #5979, and a completed Check List per the template.
Linked Issues check	✅ Passed	The PR meets the requirements from linked issue #5979 by introducing an explicit OPENSEARCH_FIPS_MODE environment variable allowing administrators to control FIPS enforcement, addressing the concern about unintended FIPS enforcement when distributions are built with -Pcrypto.standard=FIPS-140-3 by default.
Out of Scope Changes check	✅ Passed	All changes are directly related to implementing the OPENSEARCH_FIPS_MODE environment variable mechanism and updating related test configurations, with no extraneous modifications outside the linked issue's scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into main

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

Tip

Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cwperks]

@beanuwave let me know what you think of this change. I think we should change the check to something different and not have it be checking for the existence of bcFips jars under the lib/ folder.

If the change in this PR looks ok, then I will raise a docs PR to accompany this.

[github-actions]

❗ AI-powered Code-Diff-Analyzer found issues on commit 4cb633e.

Path	Line	Severity	Description
distribution/src/bin/opensearch-env	116	low	FIPS mode configuration changed from automatic detection (jar presence) to explicit opt-in via environment variable. This is a documented design change with legitimate use cases, but could allow FIPS mode to be disabled in environments where it was previously auto-enabled. Default value is 'false', requiring explicit enablement. No evidence of malicious intent - appears to be intentional architecture change per PR #20625.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 0 | Medium: 0 | Low: 1

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

❌ Gradle check result for 4cb633e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[beanuwave]

@cwperks I like this approach - it's one step closer to having bcfips.jar included on the class-path. However, I'm missing the trigger part. OPENSEARCH_FIPS_MODE would likly be set by gradle build script or docker's entrypoint.sh. All tests would need to pass when building with the -Pcrypto.standard=FIPS-140-3 parameter.

[github-actions]

❌ Gradle check result for 08723c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[cwperks]

@beanuwave agreed on running the entire gradle check against a distro built with -Pcrypto.standard=FIPS-140-3, but the scope of this PR is more focused on the logic within opensearch-env which then adds the system props to enforce that the OpenSearch process is run with FIPS approved algorithms.

[github-actions]

❌ Gradle check result for 08723c6: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 08723c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8c75738: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 8c75738: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.21%. Comparing base (3aed19d) to head (8c75738).
⚠️ Report is 131 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #20625      +/-   ##
============================================
- Coverage     73.30%   73.21%   -0.09%     
+ Complexity    71965    71914      -51     
============================================
  Files          5781     5781              
  Lines        329200   329200              
  Branches      47491    47491              
============================================
- Hits         241314   241040     -274     
- Misses        68491    68813     +322     
+ Partials      19395    19347      -48     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.