Add sample integ tests for latest systemd unit file #17410

RajatGupta02 · 2025-02-21T07:44:43Z

Description

Added few integ tests for the latest systemd unit file that has to be shipped from version 3.0 onwards.

Supporting References

Related PR: #17107

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

RajatGupta02 · 2025-02-21T07:45:21Z

@peterzhuamazon Here are the sample tests to verify the behavior

github-actions · 2025-02-21T08:11:27Z

❌ Gradle check result for beaa600: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2025-02-21T13:12:26Z

@RajatGupta02 I am not sure how exactly this test is supposed to be running, should we run it in Docker with RPM distribution installed? Thank you.

peterzhuamazon · 2025-02-21T15:08:05Z

@RajatGupta02 I am not sure how exactly this test is supposed to be running, should we run it in Docker with RPM distribution installed? Thank you.

It will be onboard to opensearch-build repo and run within the framework there.
Assuming a cluster is already deployed with deb/rpm in a docker on Jenkins.

peterzhuamazon · 2025-02-24T22:26:26Z

Hi @RajatGupta02 @kumargu I made a change to this PR, specially adding sudo for su command, as we are assuming a non-root user to run the test.

I also have changes to build repo which we can build the new deb/rpm with service file and env file from your previous PR (#17107):

Support new OS core systemd and integTests in 3.0.0 and manifest updates opensearch-build#5345

There might be some changes required for this line, as our docker container is assuming to have 4 CPUs, I wonder if you can use workers by default to avoid overflowing the system.

:qa:systemd-test:integTest.maxParallelForks (49) is larger than max-workers (4), forcing it to 4

cc: @getsaurabh02

The result log:


> Task :buildSrc:reaper:compileJava FROM-CACHE
> Task :buildSrc:reaper:processResources NO-SOURCE
> Task :buildSrc:reaper:classes UP-TO-DATE
> Task :buildSrc:reaper:jar
> Task :buildSrc:compileJava FROM-CACHE
> Task :buildSrc:compileGroovy FROM-CACHE
> Task :buildSrc:generateVersionProperties FROM-CACHE
> Task :buildSrc:pluginDescriptors
> Task :buildSrc:processResources
> Task :buildSrc:classes
> Task :buildSrc:jar
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.12
  OS Info               : Linux 5.10.234-205.895.amzn2int.x86_64 (amd64)
  JDK Version           : 23 (Eclipse Temurin JDK)
  JAVA_HOME             : /opt/java/openjdk-23
  Random Testing Seed   : 6B21ADE5FD63681D
  In FIPS 140 mode      : false
=======================================
> Task :libs:opensearch-cli:processResources NO-SOURCE
> Task :libs:opensearch-geo:processResources NO-SOURCE
> Task :libs:opensearch-plugin-classloader:processResources NO-SOURCE
> Task :libs:opensearch-compress:processResources
> Task :libs:opensearch-core:processResources
> Task :libs:opensearch-common:processResources
> Task :libs:opensearch-secure-sm:processResources NO-SOURCE
> Task :libs:opensearch-task-commons:processResources NO-SOURCE
> Task :libs:opensearch-telemetry:processResources NO-SOURCE
> Task :server:extractProto
> Task :libs:opensearch-x-content:processResources
> Task :libs:opensearch-plugin-classloader:compileJava FROM-CACHE
> Task :libs:opensearch-geo:compileJava FROM-CACHE
> Task :libs:opensearch-secure-sm:compileJava FROM-CACHE
> Task :client:sniffer:processResources NO-SOURCE
> Task :libs:opensearch-geo:classes UP-TO-DATE
> Task :libs:opensearch-geo:jar
> Task :libs:opensearch-nio:processResources NO-SOURCE
> Task :libs:opensearch-secure-sm:classes UP-TO-DATE
> Task :libs:opensearch-secure-sm:jar
> Task :qa:systemd-test:processTestResources NO-SOURCE
> Task :client:rest:compileJava FROM-CACHE
> Task :client:rest:processResources NO-SOURCE
> Task :client:rest:classes UP-TO-DATE
> Task :test:framework:processResources
> Task :client:rest:jar
> Task :test:telemetry:processResources NO-SOURCE
> Task :client:sniffer:compileJava FROM-CACHE
> Task :client:sniffer:classes UP-TO-DATE
> Task :client:sniffer:jar
> Task :libs:opensearch-common:compileJava FROM-CACHE
> Task :libs:opensearch-common:classes
> Task :libs:opensearch-common:jar
> Task :libs:opensearch-cli:compileJava FROM-CACHE
> Task :libs:opensearch-task-commons:compileJava FROM-CACHE
> Task :libs:opensearch-cli:classes UP-TO-DATE
> Task :libs:opensearch-cli:jar
> Task :libs:opensearch-telemetry:compileJava FROM-CACHE
> Task :libs:opensearch-task-commons:classes UP-TO-DATE
> Task :libs:opensearch-task-commons:jar
> Task :libs:opensearch-telemetry:classes UP-TO-DATE
> Task :server:extractIncludeProto
> Task :libs:opensearch-nio:compileJava FROM-CACHE
> Task :libs:opensearch-nio:classes UP-TO-DATE
> Task :server:generateProto FROM-CACHE
> Task :server:generateModulesList
> Task :server:generatePluginsList
> Task :libs:opensearch-telemetry:jar
> Task :libs:opensearch-nio:jar
> Task :server:processResources
> Task :libs:opensearch-core:compileJava FROM-CACHE
> Task :libs:opensearch-core:classes
> Task :libs:opensearch-core:jar
> Task :libs:opensearch-compress:compileJava FROM-CACHE
> Task :libs:opensearch-compress:classes
> Task :libs:opensearch-x-content:compileJava FROM-CACHE
> Task :libs:opensearch-x-content:classes
> Task :libs:opensearch-compress:jar
> Task :test:telemetry:compileJava FROM-CACHE
> Task :test:telemetry:classes UP-TO-DATE
> Task :libs:opensearch-x-content:jar
> Task :test:telemetry:jar
> Task :server:compileJava FROM-CACHE
> Task :server:classes
> Task :test:framework:compileJava FROM-CACHE
> Task :test:framework:classes
> Task :test:framework:jar
> Task :server:jar
> Task :qa:systemd-test:compileTestJava
> Task :qa:systemd-test:testClasses

> Task :qa:systemd-test:integTest

BUILD SUCCESSFUL in 17s
52 actionable tasks: 31 executed, 21 from cache

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

github-actions · 2025-02-26T18:55:16Z

❌ Gradle check result for 4afea87: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

RajatGupta02 · 2025-02-28T11:07:47Z

@reta Could you please review this PR? We were able to run these tests on opensearch-build repo (which now pulls the core repo and runs these tests in a separate environment in Jenkins there)
Thanks

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

github-actions · 2025-02-28T19:18:41Z

❌ Gradle check result for 928e5db: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

RajatGupta02 · 2025-02-28T19:35:37Z

After fixing one of the gradle precommit check that involved SystemdIT class to extend OpenSearchIntegTestCase class it causes issues while running and fails Gradle Check(Jenkins) as now JSM identifies this test and is restricting it to make certain calls required for executing commands:

java.security.AccessControlException: access denied ("java.io.FilePermission" "<<ALL FILES>>" "execute")
	at __randomizedtesting.SeedInfo.seed([161327F56878ADC2]:0)
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1085)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
	at java.base/java.lang.SecurityManager.checkExec(SecurityManager.java:653)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1115)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
	at java.base/java.lang.Runtime.exec(Runtime.java:681)
	at java.base/java.lang.Runtime.exec(Runtime.java:530)
	at org.opensearch.systemdinteg.SystemdIT.executeCommand(SystemdIT.java:98)
	at org.opensearch.systemdinteg.SystemdIT.getOpenSearchPid(SystemdIT.java:73)
	at org.opensearch.systemdinteg.SystemdIT.setup(SystemdIT.java:64)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:878)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1575)

reta · 2025-02-28T23:02:22Z

qa/systemd-test/src/test/java/org/opensearch/systemdinteg/SystemdIT.java

+*/
+
+/*
+* Licensed to Elasticsearch under one or more contributor


This is new file, no need to have Elasticsearch attribution

reta · 2025-02-28T23:05:24Z

qa/systemd-test/build.gradle

@@ -0,0 +1,7 @@
+apply plugin: 'opensearch.standalone-rest-test'
+
+tasks.register("integTest", Test) {


AFAIK you probably don't need to run this test as part of Gradle check, right?

Nope, this only needs to be executed as part of the build integTest.

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

kumargu · 2025-03-01T07:32:27Z

After fixing one of the gradle precommit check that involved SystemdIT class to extend OpenSearchIntegTestCase class it causes issues while running and fails Gradle Check(Jenkins) as now JSM identifies this test and is restricting it to make certain calls required for executing commands:

this is probably because we moved some scripts to a file and the test is trying access the file? If yes, can we keep the script to the same SystemIT file?

This was referenced Feb 21, 2025

[RELEASE] Release version 3.0.0 opensearch-project/opensearch-build#3747

Open

Support new OS core systemd and integTests in 3.0.0 and manifest updates opensearch-project/opensearch-build#5345

Merged

Rajat Gupta and others added 15 commits February 26, 2025 10:15

Remove unit file mount

79f7a79

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Use centos image

66d808c

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Change method name

c7e276e

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Add sample systemd integ tests to verify behavior

cbc79bc

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Update su with sudo probably need to have a privileged mode

a678a08

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

Additional tests

446e9af

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Wrap commands with su -c

ef4733a

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Add sudo

13a5345

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Remove sudo for test process exit

c28b917

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Minor fixes

95afea4

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Fixed script string

a6f6285

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Remove redundant code

92d3e5d

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Add terminate script

dee4eac

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Modified terminate script

0c4f27d

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Add Changelog-3.0 entry

4afea87

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

RajatGupta02 force-pushed the test-integ-branch branch from efccfcc to 4afea87 Compare February 26, 2025 18:18

RajatGupta02 mentioned this pull request Feb 26, 2025

Add systemd integ tests to run with docker #17308

Closed

3 tasks

kumargu approved these changes Feb 28, 2025

View reviewed changes

Fix for gradle precommit workflow

b6fda48

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

RajatGupta02 requested a review from bugmakerrrrrr as a code owner February 28, 2025 18:09

Rajat Gupta added 2 commits February 28, 2025 10:26

Fix testing conventions gradle precommit check

97bf56b

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Fix imports

928e5db

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

reta reviewed Feb 28, 2025

View reviewed changes

Only run as part of build integTest, remove gradle check

cbee189

Signed-off-by: Rajat Gupta <gptrajat@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add sample integ tests for latest systemd unit file #17410

Add sample integ tests for latest systemd unit file #17410

RajatGupta02 commented Feb 21, 2025 •

edited

Loading

RajatGupta02 commented Feb 21, 2025

github-actions bot commented Feb 21, 2025

reta commented Feb 21, 2025

peterzhuamazon commented Feb 21, 2025

peterzhuamazon commented Feb 24, 2025

github-actions bot commented Feb 26, 2025

RajatGupta02 commented Feb 28, 2025 •

edited

Loading

github-actions bot commented Feb 28, 2025

RajatGupta02 commented Feb 28, 2025

reta Feb 28, 2025

reta Feb 28, 2025

peterzhuamazon Mar 1, 2025

kumargu commented Mar 1, 2025

		@@ -0,0 +1,7 @@
		apply plugin: 'opensearch.standalone-rest-test'

		tasks.register("integTest", Test) {

Add sample integ tests for latest systemd unit file #17410

Are you sure you want to change the base?

Add sample integ tests for latest systemd unit file #17410

Conversation

RajatGupta02 commented Feb 21, 2025 • edited Loading

Description

Supporting References

Check List

RajatGupta02 commented Feb 21, 2025

github-actions bot commented Feb 21, 2025

reta commented Feb 21, 2025

peterzhuamazon commented Feb 21, 2025

peterzhuamazon commented Feb 24, 2025

github-actions bot commented Feb 26, 2025

RajatGupta02 commented Feb 28, 2025 • edited Loading

github-actions bot commented Feb 28, 2025

RajatGupta02 commented Feb 28, 2025

reta Feb 28, 2025

Choose a reason for hiding this comment

reta Feb 28, 2025

Choose a reason for hiding this comment

peterzhuamazon Mar 1, 2025

Choose a reason for hiding this comment

kumargu commented Mar 1, 2025

RajatGupta02 commented Feb 21, 2025 •

edited

Loading

RajatGupta02 commented Feb 28, 2025 •

edited

Loading