Update ELK images for 9.0.0-rc1 #18692
Open
+15
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
|
I have basically the same comment as the first part of #18475 (comment): From the failing CI: In this case, these are referencing images outside Docker Official Images, which is not allowed in Official Images. There are a few Redhat
|
alpar-t
commented
Mar 25, 2025
|
Thanks @yosifkit ! I switched the images to use |
This comment has been minimized.
This comment has been minimized.
alpar-t
force-pushed
the
update-official-images-9.0.0-rc1
branch
from
443f067 to
36c45bd
Compare
|
Thanks again! Rebased and force pushed. |
Diff for 36c45bd:diff --git a/_bashbrew-cat b/_bashbrew-cat
index 4fe169f..cc8e70e 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -19,6 +19,11 @@ Architectures: amd64, arm64v8
GitFetch: refs/heads/8.17
GitCommit: 92353caed0efb0cd7dabd9113167277fd4488856
+Tags: 9.0.0-rc1
+Architectures: amd64, arm64v8
+GitFetch: refs/heads/9.0
+GitCommit: 8fd67af47850632590faeca14ad65734bca0a163
+
# kibana
Maintainers: Jon Budzenski (@jbudz)
@@ -41,6 +46,11 @@ Architectures: amd64, arm64v8
GitFetch: refs/heads/8.17
GitCommit: 92353caed0efb0cd7dabd9113167277fd4488856
+Tags: 9.0.0-rc1
+Architectures: amd64, arm64v8
+GitFetch: refs/heads/9.0
+GitCommit: 8fd67af47850632590faeca14ad65734bca0a163
+
# logstash
Maintainers: João Duarte (@jsvd)
@@ -62,3 +72,8 @@ Tags: 8.17.4
Architectures: amd64, arm64v8
GitFetch: refs/heads/8.17
GitCommit: 92353caed0efb0cd7dabd9113167277fd4488856
+
+Tags: 9.0.0-rc1
+Architectures: amd64, arm64v8
+GitFetch: refs/heads/9.0
+GitCommit: 8fd67af47850632590faeca14ad65734bca0a163
diff --git a/_bashbrew-list b/_bashbrew-list
index ba21ac6..c6747ed 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -1,9 +1,12 @@
elasticsearch:7.17.28
elasticsearch:8.16.6
elasticsearch:8.17.4
+elasticsearch:9.0.0-rc1
kibana:7.17.28
kibana:8.16.6
kibana:8.17.4
+kibana:9.0.0-rc1
logstash:7.17.28
logstash:8.16.6
logstash:8.17.4
+logstash:9.0.0-rc1
diff --git a/_bashbrew-list-build-order b/_bashbrew-list-build-order
index ba21ac6..c6747ed 100644
--- a/_bashbrew-list-build-order
+++ b/_bashbrew-list-build-order
@@ -1,9 +1,12 @@
elasticsearch:7.17.28
elasticsearch:8.16.6
elasticsearch:8.17.4
+elasticsearch:9.0.0-rc1
kibana:7.17.28
kibana:8.16.6
kibana:8.17.4
+kibana:9.0.0-rc1
logstash:7.17.28
logstash:8.16.6
logstash:8.17.4
+logstash:9.0.0-rc1
diff --git a/elasticsearch_8.17.4/Dockerfile b/elasticsearch_9.0.0-rc1/Dockerfile
similarity index 81%
copy from elasticsearch_8.17.4/Dockerfile
copy to elasticsearch_9.0.0-rc1/Dockerfile
index 937f163..2639257 100644
--- a/elasticsearch_8.17.4/Dockerfile
+++ b/elasticsearch_9.0.0-rc1/Dockerfile
@@ -9,14 +9,14 @@
# Extract Elasticsearch artifact
################################################################################
-FROM ubuntu:20.04 AS builder
+FROM redhat/ubi9-minimal:latest AS builder
# Install required packages to extract the Elasticsearch distribution
RUN for iter in 1 2 3 4 5 6 7 8 9 10; do \
- apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl && \
+ microdnf install -y findutils tar gzip && \
exit_code=0 && break || \
- exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10; \
+ exit_code=$? && echo "microdnf error: retry $iter in 10s" && sleep 10; \
done; \
exit $exit_code
@@ -44,7 +44,7 @@ RUN for iter in 1 2 3 4 5 6 7 8 9 10; do \
RUN mkdir /usr/share/elasticsearch
WORKDIR /usr/share/elasticsearch
-RUN curl --retry 10 -S -L --output /tmp/elasticsearch.tar.gz https://artifacts-no-kpi.elastic.co/downloads/elasticsearch/elasticsearch-8.17.4-linux-$(arch).tar.gz
+RUN curl --retry 10 -S -L --output /tmp/elasticsearch.tar.gz https://artifacts-no-kpi.elastic.co/downloads/elasticsearch/elasticsearch-9.0.0-rc1-linux-$(arch).tar.gz
RUN tar -zxf /tmp/elasticsearch.tar.gz --strip-components=1
@@ -79,26 +79,20 @@ RUN sed -i -e 's/ES_DISTRIBUTION_TYPE=tar/ES_DISTRIBUTION_TYPE=docker/' bin/elas
# Add entrypoint
################################################################################
-FROM ubuntu:20.04
-
-# Change default shell to bash, then install required packages with retries.
-RUN yes no | dpkg-reconfigure dash && \
- for iter in 1 2 3 4 5 6 7 8 9 10; do \
- export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get upgrade -y && \
- apt-get install -y --no-install-recommends \
- ca-certificates curl netcat p11-kit unzip zip && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists/* && \
+FROM redhat/ubi9-minimal:latest
+
+RUN for iter in 1 2 3 4 5 6 7 8 9 10; do \
+ microdnf update --setopt=tsflags=nodocs -y && \
+ microdnf install --setopt=tsflags=nodocs -y \
+ nc shadow-utils zip unzip findutils procps-ng && \
+ microdnf clean all && \
exit_code=0 && break || \
- exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10; \
+ exit_code=$? && echo "microdnf error: retry $iter in 10s" && sleep 10; \
done; \
exit $exit_code
RUN groupadd -g 1000 elasticsearch && \
- adduser --uid 1000 --gid 1000 --home /usr/share/elasticsearch elasticsearch && \
- adduser elasticsearch root && \
+ adduser -u 1000 -g 1000 -G 0 -d /usr/share/elasticsearch elasticsearch && \
chown -R 0:0 /usr/share/elasticsearch
ENV ELASTIC_CONTAINER true
@@ -130,32 +124,39 @@ RUN chmod g=u /etc/passwd && \
chmod 0775 /usr/share/elasticsearch && \
chown elasticsearch bin config config/jvm.options.d data logs plugins
-# Update "cacerts" bundle to use Ubuntu's CA certificates (and make sure it
-# stays up-to-date with changes to Ubuntu's store)
-COPY bin/docker-openjdk /etc/ca-certificates/update.d/docker-openjdk
-RUN /etc/ca-certificates/update.d/docker-openjdk
+RUN ln -sf /etc/pki/ca-trust/extracted/java/cacerts /usr/share/elasticsearch/jdk/lib/security/cacerts
EXPOSE 9200 9300
-LABEL org.label-schema.build-date="2025-03-20T15:39:59.811110136Z" \
+LABEL org.label-schema.build-date="2025-03-20T10:06:19.154801632Z" \
org.label-schema.license="Elastic-License-2.0" \
org.label-schema.name="Elasticsearch" \
org.label-schema.schema-version="1.0" \
org.label-schema.url="https://www.elastic.co/products/elasticsearch" \
org.label-schema.usage="https://www.elastic.co/guide/en/elasticsearch/reference/index.html" \
- org.label-schema.vcs-ref="c63c7f5f8ce7d2e4805b7b3d842e7e792d84dda1" \
+ org.label-schema.vcs-ref="4b4c2d3901e40762d9b04f3e6321ef144f2627d8" \
org.label-schema.vcs-url="https://github.com/elastic/elasticsearch" \
org.label-schema.vendor="Elastic" \
- org.label-schema.version="8.17.4" \
- org.opencontainers.image.created="2025-03-20T15:39:59.811110136Z" \
+ org.label-schema.version="9.0.0-rc1" \
+ org.opencontainers.image.created="2025-03-20T10:06:19.154801632Z" \
org.opencontainers.image.documentation="https://www.elastic.co/guide/en/elasticsearch/reference/index.html" \
org.opencontainers.image.licenses="Elastic-License-2.0" \
- org.opencontainers.image.revision="c63c7f5f8ce7d2e4805b7b3d842e7e792d84dda1" \
+ org.opencontainers.image.revision="4b4c2d3901e40762d9b04f3e6321ef144f2627d8" \
org.opencontainers.image.source="https://github.com/elastic/elasticsearch" \
org.opencontainers.image.title="Elasticsearch" \
org.opencontainers.image.url="https://www.elastic.co/products/elasticsearch" \
org.opencontainers.image.vendor="Elastic" \
- org.opencontainers.image.version="8.17.4"
+ org.opencontainers.image.version="9.0.0-rc1"
+
+LABEL name="Elasticsearch" \
+ maintainer="infra@elastic.co" \
+ vendor="Elastic" \
+ version="9.0.0-rc1" \
+ release="1" \
+ summary="Elasticsearch" \
+ description="You know, for search."
+
+RUN mkdir /licenses && cp LICENSE.txt /licenses/LICENSE
# Our actual entrypoint is `tini`, a minimal but functional init program. It
# calls the entrypoint we provide, while correctly forwarding signals.
diff --git a/elasticsearch_8.16.6/bin/docker-entrypoint.sh b/elasticsearch_9.0.0-rc1/bin/docker-entrypoint.sh
similarity index 100%
copy from elasticsearch_8.16.6/bin/docker-entrypoint.sh
copy to elasticsearch_9.0.0-rc1/bin/docker-entrypoint.sh
diff --git a/elasticsearch_7.17.28/config/elasticsearch.yml b/elasticsearch_9.0.0-rc1/config/elasticsearch.yml
similarity index 100%
copy from elasticsearch_7.17.28/config/elasticsearch.yml
copy to elasticsearch_9.0.0-rc1/config/elasticsearch.yml
diff --git a/elasticsearch_8.17.4/config/log4j2.properties b/elasticsearch_9.0.0-rc1/config/log4j2.properties
similarity index 93%
copy from elasticsearch_8.17.4/config/log4j2.properties
copy to elasticsearch_9.0.0-rc1/config/log4j2.properties
index c0d67c8..bd79621 100644
--- a/elasticsearch_8.17.4/config/log4j2.properties
+++ b/elasticsearch_9.0.0-rc1/config/log4j2.properties
@@ -18,7 +18,7 @@ appender.deprecation_rolling.type = Console
appender.deprecation_rolling.name = deprecation_rolling
appender.deprecation_rolling.layout.type = ECSJsonLayout
# Intentionally follows a different pattern to above
-appender.deprecation_rolling.layout.dataset = deprecation.elasticsearch
+appender.deprecation_rolling.layout.dataset = elasticsearch.deprecation
appender.deprecation_rolling.filter.rate_limit.type = RateLimitingFilter
appender.header_warning.type = HeaderWarningAppender
@@ -70,6 +70,12 @@ logger.org_apache_fontbox.level = off
logger.org_apache_xmlbeans.name = org.apache.xmlbeans
logger.org_apache_xmlbeans.level = off
+logger.entitlements_ingest_attachment.name = org.elasticsearch.entitlement.runtime.policy.PolicyManager.ingest-attachment.ALL-UNNAMED
+logger.entitlements_ingest_attachment.level = error
+
+logger.entitlements_repository_gcs.name = org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-gcs.ALL-UNNAMED
+logger.entitlements_repository_gcs.level = error
+
logger.com_amazonaws.name = com.amazonaws
logger.com_amazonaws.level = warn
@@ -85,6 +91,9 @@ logger.com_amazonaws_auth_profile_internal_BasicProfileConfigFileLoader.level =
logger.com_amazonaws_services_s3_internal_UseArnRegionResolver.name = com.amazonaws.services.s3.internal.UseArnRegionResolver
logger.com_amazonaws_services_s3_internal_UseArnRegionResolver.level = error
+logger.entitlements_repository_s3.name = org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-s3.ALL-UNNAMED
+logger.entitlements_repository_s3.level = error
+
appender.audit_rolling.type = Console
appender.audit_rolling.name = audit_rolling
appender.audit_rolling.layout.type = PatternLayout
@@ -191,3 +200,6 @@ logger.samlxml_decrypt.name = org.opensaml.xmlsec.encryption.support.Decrypter
logger.samlxml_decrypt.level = fatal
logger.saml2_decrypt.name = org.opensaml.saml.saml2.encryption.Decrypter
logger.saml2_decrypt.level = fatal
+
+logger.entitlements_inference.name = org.elasticsearch.entitlement.runtime.policy.PolicyManager.inference.software.amazon.awssdk.profiles
+logger.entitlements_inference.level = error
diff --git a/kibana_8.16.6/Dockerfile b/kibana_9.0.0-rc1/Dockerfile
similarity index 79%
copy from kibana_8.16.6/Dockerfile
copy to kibana_9.0.0-rc1/Dockerfile
index 3672297..8bf339d 100644
--- a/kibana_8.16.6/Dockerfile
+++ b/kibana_9.0.0-rc1/Dockerfile
@@ -9,14 +9,14 @@
# Build stage 0 `builder`:
# Extract Kibana artifact
################################################################################
-FROM ubuntu:20.04 AS builder
+FROM redhat/ubi9-minimal:latest AS builder
-RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl
+RUN microdnf install -y findutils tar gzip
RUN cd /tmp && \
curl --retry 8 -s -L \
--output kibana.tar.gz \
- https://artifacts.elastic.co/downloads/kibana/kibana-8.16.6-linux-$(arch).tar.gz && \
+ https://artifacts.elastic.co/downloads/kibana/kibana-9.0.0-rc1-linux-$(arch).tar.gz && \
cd -
RUN mkdir /usr/share/kibana
@@ -61,17 +61,14 @@ RUN mkdir -p /usr/share/fonts/local && \
# Copy kibana from stage 0
# Add entrypoint
################################################################################
-FROM ubuntu:20.04
+FROM redhat/ubi9-minimal:latest
EXPOSE 5601
RUN for iter in {1..10}; do \
- export DEBIAN_FRONTEND=noninteractive && \
- apt-get update && \
- apt-get upgrade -y && \
- apt-get install -y --no-install-recommends \
- fontconfig fonts-liberation libnss3 curl ca-certificates && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists/* && exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && \
+ microdnf update --setopt=tsflags=nodocs -y && \
+ microdnf install --setopt=tsflags=nodocs -y \
+ fontconfig liberation-fonts-common freetype shadow-utils nss findutils && \
+ microdnf clean all && exit_code=0 && break || exit_code=$? && echo "microdnf error: retry $iter in 10s" && \
sleep 10; \
done; \
(exit $exit_code)
@@ -108,26 +105,36 @@ RUN groupadd --gid 1000 kibana && \
--home-dir /usr/share/kibana --no-create-home \
kibana
-LABEL org.label-schema.build-date="2025-03-20T11:10:41.087Z" \
+LABEL org.label-schema.build-date="2025-03-19T12:31:41.621Z" \
org.label-schema.license="Elastic License" \
org.label-schema.name="Kibana" \
org.label-schema.schema-version="1.0" \
org.label-schema.url="https://www.elastic.co/products/kibana" \
org.label-schema.usage="https://www.elastic.co/guide/en/kibana/reference/index.html" \
- org.label-schema.vcs-ref="39969cb4b1ab957faf1e78d25d83ec04192ddc21" \
+ org.label-schema.vcs-ref="d6960388820701573f4a79825ae057ff62e37d83" \
org.label-schema.vcs-url="https://github.com/elastic/kibana" \
org.label-schema.vendor="Elastic" \
- org.label-schema.version="8.16.6" \
- org.opencontainers.image.created="2025-03-20T11:10:41.087Z" \
+ org.label-schema.version="9.0.0-rc1" \
+ org.opencontainers.image.created="2025-03-19T12:31:41.621Z" \
org.opencontainers.image.documentation="https://www.elastic.co/guide/en/kibana/reference/index.html" \
org.opencontainers.image.licenses="Elastic License" \
- org.opencontainers.image.revision="39969cb4b1ab957faf1e78d25d83ec04192ddc21" \
+ org.opencontainers.image.revision="d6960388820701573f4a79825ae057ff62e37d83" \
org.opencontainers.image.source="https://github.com/elastic/kibana" \
org.opencontainers.image.title="Kibana" \
org.opencontainers.image.url="https://www.elastic.co/products/kibana" \
org.opencontainers.image.vendor="Elastic" \
- org.opencontainers.image.version="8.16.6"
-
+ org.opencontainers.image.version="9.0.0-rc1"
+
+LABEL name="Kibana" \
+ maintainer="infra@elastic.co" \
+ vendor="Elastic" \
+ version="9.0.0-rc1" \
+ release="1" \
+ summary="Kibana" \
+ description="Your window into the Elastic Stack."
+
+RUN mkdir /licenses && \
+ cp LICENSE.txt /licenses/LICENSE
ENTRYPOINT ["/bin/tini", "--"]
diff --git a/kibana_8.17.4/bin/kibana-docker b/kibana_9.0.0-rc1/bin/kibana-docker
similarity index 97%
copy from kibana_8.17.4/bin/kibana-docker
copy to kibana_9.0.0-rc1/bin/kibana-docker
index f4ae377..951d6a0 100755
--- a/kibana_8.17.4/bin/kibana-docker
+++ b/kibana_9.0.0-rc1/bin/kibana-docker
@@ -40,6 +40,7 @@ kibana_vars=(
csp.report_uri
csp.report_to
csp.report_only.form_action
+ csp.report_only.object_src
permissionsPolicy.report_to
data.autocomplete.valueSuggestions.terminateAfter
data.autocomplete.valueSuggestions.timeout
@@ -83,9 +84,6 @@ kibana_vars=(
elasticsearch.ssl.truststore.path
elasticsearch.ssl.verificationMode
elasticsearch.username
- enterpriseSearch.accessCheckTimeout
- enterpriseSearch.accessCheckTimeoutWarning
- enterpriseSearch.host
externalUrl.policy
i18n.locale
interactiveSetup.enabled
@@ -165,6 +163,7 @@ kibana_vars=(
server.name
server.port
server.protocol
+ server.prototypeHardening
server.publicBaseUrl
server.requestId.allowFromAnyIp
server.requestId.ipAllowlist
@@ -221,16 +220,13 @@ kibana_vars=(
xpack.actions.proxyBypassHosts
xpack.actions.proxyHeaders
xpack.actions.proxyOnlyHosts
- xpack.actions.proxyRejectUnauthorizedCertificates
xpack.actions.proxyUrl
- xpack.actions.rejectUnauthorized
xpack.actions.responseTimeout
xpack.actions.ssl.proxyVerificationMode
xpack.actions.ssl.verificationMode
xpack.alerting.healthCheck.interval
xpack.alerting.invalidateApiKeysTask.interval
xpack.alerting.invalidateApiKeysTask.removalDelay
- xpack.alerting.defaultRuleTaskTimeout
xpack.alerting.rules.run.timeout
xpack.alerting.rules.run.ruleTypeOverrides
xpack.alerting.cancelAlertsOnRuleTimeout
@@ -240,9 +236,6 @@ kibana_vars=(
xpack.alerting.rules.run.alerts.max
xpack.alerting.rules.run.actions.connectorTypeOverrides
xpack.alerting.maxScheduledPerMinute
- xpack.alerts.healthCheck.interval
- xpack.alerts.invalidateApiKeysTask.interval
- xpack.alerts.invalidateApiKeysTask.removalDelay
xpack.apm.indices.error
xpack.apm.indices.metric
xpack.apm.indices.onboarding
@@ -322,7 +315,6 @@ kibana_vars=(
xpack.productDocBase.artifactRepositoryUrl
xpack.reporting.capture.browser.autoDownload
xpack.reporting.capture.browser.chromium.disableSandbox
- xpack.reporting.capture.browser.chromium.inspect
xpack.reporting.capture.browser.chromium.maxScreenshotDimension
xpack.reporting.capture.browser.chromium.proxy.bypass
xpack.reporting.capture.browser.chromium.proxy.enabled
@@ -383,6 +375,7 @@ kibana_vars=(
xpack.security.audit.appender.strategy.type
xpack.security.audit.appender.type
xpack.security.audit.enabled
+ xpack.security.audit.include_saved_object_names
xpack.security.audit.ignore_filters
xpack.security.authc.http.autoSchemesEnabled
xpack.security.authc.http.enabled
diff --git a/kibana_7.17.28/config/kibana.yml b/kibana_9.0.0-rc1/config/kibana.yml
similarity index 100%
copy from kibana_7.17.28/config/kibana.yml
copy to kibana_9.0.0-rc1/config/kibana.yml
diff --git a/logstash_8.17.4/Dockerfile b/logstash_9.0.0-rc1/Dockerfile
similarity index 36%
copy from logstash_8.17.4/Dockerfile
copy to logstash_9.0.0-rc1/Dockerfile
index bd909fe..e6816ab 100644
--- a/logstash_8.17.4/Dockerfile
+++ b/logstash_9.0.0-rc1/Dockerfile
@@ -1,95 +1,93 @@
# This Dockerfile was generated from templates/Dockerfile.erb
+# Build env2yaml
+FROM golang:1.23 as builder-env2yaml
+COPY env2yaml/env2yaml.go /tmp/go/src/env2yaml/env2yaml.go
+COPY env2yaml/go.mod /tmp/go/src/env2yaml/go.mod
+COPY env2yaml/go.sum /tmp/go/src/env2yaml/go.sum
-FROM ubuntu:20.04
+WORKDIR /tmp/go/src/env2yaml
-RUN for iter in {1..10}; do \
- export DEBIAN_FRONTEND=noninteractive && \
- apt-get update -y && \
- apt-get upgrade -y && \
- apt-get install -y procps findutils tar gzip && \
- apt-get install -y locales && \
- apt-get install -y curl && \
- apt-get clean all && \
- locale-gen 'en_US.UTF-8' && \
- apt-get clean metadata && \
- exit_code=0 && break || exit_code=$? && \
-echo "packaging error: retry $iter in 10s" && \
-apt-get clean all && \
- apt-get clean metadata && \
-sleep 10; done; \
-(exit $exit_code)
-
-# Provide a non-root user to run the process.
-RUN groupadd --gid 1000 logstash && \
- adduser --uid 1000 --gid 1000 --home /usr/share/logstash --no-create-home logstash
+RUN go build
-# Add Logstash itself.
-RUN curl -Lo - https://artifacts.elastic.co/downloads/logstash/logstash-8.17.4-linux-$(arch).tar.gz | \
- tar zxf - -C /usr/share && \
- mv /usr/share/logstash-8.17.4 /usr/share/logstash && \
- chown --recursive logstash:logstash /usr/share/logstash/ && \
- chown -R logstash:root /usr/share/logstash && \
- chmod -R g=u /usr/share/logstash && \
- mkdir /licenses/ && \
- mv /usr/share/logstash/NOTICE.TXT /licenses/NOTICE.TXT && \
- mv /usr/share/logstash/LICENSE.txt /licenses/LICENSE.txt && \
- find /usr/share/logstash -type d -exec chmod g+s {} \; && \
- ln -s /usr/share/logstash /opt/logstash
+# Build main image
+# Minimal distributions do not ship with en language packs.
+FROM redhat/ubi9-minimal:latest
-WORKDIR /usr/share/logstash
ENV ELASTIC_CONTAINER true
ENV PATH=/usr/share/logstash/bin:$PATH
+ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
+
+WORKDIR /usr/share
-# Provide a minimal configuration, so that simple invocations will provide
-# a good experience.
- COPY config/logstash-full.yml config/logstash.yml
+COPY --from=builder-env2yaml /tmp/go/src/env2yaml/env2yaml /usr/local/bin/env2yaml
COPY config/pipelines.yml config/log4j2.properties config/log4j2.file.properties config/
+COPY config/logstash-full.yml config/logstash.yml
COPY pipeline/default.conf pipeline/logstash.conf
-
-RUN chown --recursive logstash:root config/ pipeline/
-# Ensure Logstash gets the correct locale by default.
-ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
-
-COPY env2yaml/env2yaml-amd64 env2yaml/env2yaml-arm64 env2yaml/
-# Copy over the appropriate env2yaml artifact
-RUN set -eux; env2yamlarch="$(dpkg --print-architecture)"; \
- case "${env2yamlarch}" in \
- 'x86_64'|'amd64') \
- env2yamlarch=amd64; \
- ;; \
- 'aarch64'|'arm64') \
- env2yamlarch=arm64; \
- ;; \
- *) echo >&2 "error: unsupported architecture '$env2yamlarch'"; exit 1 ;; \
- esac; \
- mkdir -p /usr/local/bin; \
- cp env2yaml/env2yaml-${env2yamlarch} /usr/local/bin/env2yaml; \
- rm -rf env2yaml
-# Place the startup wrapper script.
COPY bin/docker-entrypoint /usr/local/bin/
-RUN chmod 0755 /usr/local/bin/docker-entrypoint
+# Install packages
+RUN for iter in {1..10}; do \
+ microdnf update -y && \
+ microdnf upgrade -y && \
+ microdnf install -y procps findutils tar gzip && \
+ microdnf install -y openssl && \
+ microdnf install -y which shadow-utils && \
+ microdnf clean all && \
+ exit_code=0 && break || \
+ exit_code=$? && echo "packaging error: retry $iter in 10s" && \
+ microdnf clean all && sleep 10; \
+ done; \
+ (exit $exit_code)
+
+# Provide a non-root user to run the process
+# Add Logstash itself and set permissions
+RUN groupadd --gid 1000 logstash && \
+ adduser --uid 1000 --gid 1000 \
+ --home "/usr/share/logstash" \
+ --no-create-home \
+ logstash && \
+ curl -Lo - https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-rc1-linux-x86_64.tar.gz | \
+ tar zxf - -C /usr/share && \
+ mv /usr/share/logstash-9.0.0-rc1 /usr/share/logstash && \
+ chown -R logstash:root /usr/share/logstash config/ pipeline/ && \
+ chmod -R g=u /usr/share/logstash && \
+ mv config/* /usr/share/logstash/config && \
+ mv pipeline /usr/share/logstash/pipeline && \
+ mkdir /licenses && \
+ mv /usr/share/logstash/NOTICE.TXT /licenses/NOTICE.TXT && \
+ mv /usr/share/logstash/LICENSE.txt /licenses/LICENSE.txt && \
+ find /usr/share/logstash -type d -exec chmod g+s {} \; && \
+ ln -s /usr/share/logstash /opt/logstash && \
+ chmod 0755 /usr/local/bin/docker-entrypoint && \
+ rmdir config
+WORKDIR /usr/share/logstash
USER 1000
EXPOSE 9600 5044
-LABEL org.label-schema.schema-version="1.0" \
- org.label-schema.vendor="Elastic" \
- org.opencontainers.image.vendor="Elastic" \
+LABEL org.label-schema.build-date=2025-03-18T02:18:47+00:00 \
+ org.label-schema.license="Elastic License" \
org.label-schema.name="logstash" \
- org.opencontainers.image.title="logstash" \
- org.label-schema.version="8.17.4" \
- org.opencontainers.image.version="8.17.4" \
+ org.label-schema.schema-version="1.0" \
org.label-schema.url="https://www.elastic.co/products/logstash" \
org.label-schema.vcs-url="https://github.com/elastic/logstash" \
- org.label-schema.license="Elastic License" \
- org.opencontainers.image.licenses="Elastic License" \
+ org.label-schema.vendor="Elastic" \
+ org.label-schema.version="9.0.0-rc1" \
+ org.opencontainers.image.created=2025-03-18T02:18:47+00:00 \
org.opencontainers.image.description="Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite 'stash.'" \
- org.label-schema.build-date=2025-03-19T17:05:46+00:00 \
- org.opencontainers.image.created=2025-03-19T17:05:46+00:00
+ org.opencontainers.image.licenses="Elastic License" \
+ org.opencontainers.image.title="logstash" \
+ org.opencontainers.image.vendor="Elastic" \
+ org.opencontainers.image.version="9.0.0-rc1" \
+ description="Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite 'stash.'" \
+ license="Elastic License" \
+ maintainer="info@elastic.co" \
+ name="logstash" \
+ summary="Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite 'stash.'" \
+ vendor="Elastic"
ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
diff --git a/logstash_8.16.6/bin/docker-entrypoint b/logstash_9.0.0-rc1/bin/docker-entrypoint
similarity index 100%
copy from logstash_8.16.6/bin/docker-entrypoint
copy to logstash_9.0.0-rc1/bin/docker-entrypoint
diff --git a/logstash_8.16.6/config/log4j2.file.properties b/logstash_9.0.0-rc1/config/log4j2.file.properties
similarity index 100%
copy from logstash_8.16.6/config/log4j2.file.properties
copy to logstash_9.0.0-rc1/config/log4j2.file.properties
diff --git a/logstash_7.17.28/config/log4j2.properties b/logstash_9.0.0-rc1/config/log4j2.properties
similarity index 100%
copy from logstash_7.17.28/config/log4j2.properties
copy to logstash_9.0.0-rc1/config/log4j2.properties
diff --git a/logstash_7.17.28/config/logstash-full.yml b/logstash_9.0.0-rc1/config/logstash-full.yml
similarity index 73%
copy from logstash_7.17.28/config/logstash-full.yml
copy to logstash_9.0.0-rc1/config/logstash-full.yml
index 58e1a35..949b467 100644
--- a/logstash_7.17.28/config/logstash-full.yml
+++ b/logstash_9.0.0-rc1/config/logstash-full.yml
@@ -1,2 +1,2 @@
-http.host: "0.0.0.0"
+api.http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
diff --git a/logstash_7.17.28/config/pipelines.yml b/logstash_9.0.0-rc1/config/pipelines.yml
similarity index 100%
copy from logstash_7.17.28/config/pipelines.yml
copy to logstash_9.0.0-rc1/config/pipelines.yml
diff --git a/logstash_9.0.0-rc1/env2yaml/env2yaml.go b/logstash_9.0.0-rc1/env2yaml/env2yaml.go
new file mode 100644
index 0000000..dc2641e
--- /dev/null
+++ b/logstash_9.0.0-rc1/env2yaml/env2yaml.go
@@ -0,0 +1,198 @@
+// env2yaml
+//
+// Merge environment variables into logstash.yml.
+// For example, running Docker with:
+//
+// docker run -e pipeline.workers=6
+//
+// or
+//
+// docker run -e PIPELINE_WORKERS=6
+//
+// will cause logstash.yml to contain the line:
+//
+// pipeline.workers: 6
+package main
+
+import (
+ "errors"
+ "fmt"
+ "io/ioutil"
+ "log"
+ "os"
+ "strings"
+
+ "gopkg.in/yaml.v2"
+)
+
+var validSettings = []string{
+ "api.enabled",
+ "api.http.host",
+ "api.http.port",
+ "api.environment",
+ "node.name",
+ "path.data",
+ "pipeline.id",
+ "pipeline.workers",
+ "pipeline.output.workers",
+ "pipeline.batch.size",
+ "pipeline.batch.delay",
+ "pipeline.unsafe_shutdown",
+ "pipeline.ecs_compatibility",
+ "pipeline.ordered",
+ "pipeline.plugin_classloaders",
+ "pipeline.separate_logs",
+ "path.config",
+ "config.string",
+ "config.test_and_exit",
+ "config.reload.automatic",
+ "config.reload.interval",
+ "config.debug",
+ "config.support_escapes",
+ "config.field_reference.escape_style",
+ "queue.type",
+ "path.queue",
+ "queue.page_capacity",
+ "queue.max_events",
+ "queue.max_bytes",
+ "queue.checkpoint.acks",
+ "queue.checkpoint.writes",
+ "queue.checkpoint.interval",
+ "queue.drain",
+ "dead_letter_queue.enable",
+ "dead_letter_queue.max_bytes",
+ "dead_letter_queue.flush_interval",
+ "dead_letter_queue.storage_policy",
+ "dead_letter_queue.retain.age",
+ "path.dead_letter_queue",
+ "log.level",
+ "log.format",
+ "log.format.json.fix_duplicate_message_fields",
+ "metric.collect",
+ "path.logs",
+ "path.plugins",
+ "api.auth.type",
+ "api.auth.basic.username",
+ "api.auth.basic.password",
+ "api.auth.basic.password_policy.mode",
+ "api.auth.basic.password_policy.length.minimum",
+ "api.auth.basic.password_policy.include.upper",
+ "api.auth.basic.password_policy.include.lower",
+ "api.auth.basic.password_policy.include.digit",
+ "api.auth.basic.password_policy.include.symbol",
+ "allow_superuser",
+ "monitoring.cluster_uuid",
+ "xpack.monitoring.allow_legacy_collection",
+ "xpack.monitoring.enabled",
+ "xpack.monitoring.collection.interval",
+ "xpack.monitoring.elasticsearch.hosts",
+ "xpack.monitoring.elasticsearch.username",
+ "xpack.monitoring.elasticsearch.password",
+ "xpack.monitoring.elasticsearch.proxy",
+ "xpack.monitoring.elasticsearch.api_key",
+ "xpack.monitoring.elasticsearch.cloud_auth",
+ "xpack.monitoring.elasticsearch.cloud_id",
+ "xpack.monitoring.elasticsearch.sniffing",
+ "xpack.monitoring.elasticsearch.ssl.certificate_authority",
+ "xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint",
+ "xpack.monitoring.elasticsearch.ssl.verification_mode",
+ "xpack.monitoring.elasticsearch.ssl.truststore.path",
+ "xpack.monitoring.elasticsearch.ssl.truststore.password",
+ "xpack.monitoring.elasticsearch.ssl.keystore.path",
+ "xpack.monitoring.elasticsearch.ssl.keystore.password",
+ "xpack.monitoring.elasticsearch.ssl.certificate",
+ "xpack.monitoring.elasticsearch.ssl.key",
+ "xpack.monitoring.elasticsearch.ssl.cipher_suites",
+ "xpack.management.enabled",
+ "xpack.management.logstash.poll_interval",
+ "xpack.management.pipeline.id",
+ "xpack.management.elasticsearch.hosts",
+ "xpack.management.elasticsearch.username",
+ "xpack.management.elasticsearch.password",
+ "xpack.management.elasticsearch.proxy",
+ "xpack.management.elasticsearch.api_key",
+ "xpack.management.elasticsearch.cloud_auth",
+ "xpack.management.elasticsearch.cloud_id",
+ "xpack.management.elasticsearch.sniffing",
+ "xpack.management.elasticsearch.ssl.certificate_authority",
+ "xpack.management.elasticsearch.ssl.ca_trusted_fingerprint",
+ "xpack.management.elasticsearch.ssl.verification_mode",
+ "xpack.management.elasticsearch.ssl.truststore.path",
+ "xpack.management.elasticsearch.ssl.truststore.password",
+ "xpack.management.elasticsearch.ssl.keystore.path",
+ "xpack.management.elasticsearch.ssl.keystore.password",
+ "xpack.management.elasticsearch.ssl.certificate",
+ "xpack.management.elasticsearch.ssl.key",
+ "xpack.management.elasticsearch.ssl.cipher_suites",
+ "xpack.geoip.download.endpoint",
+ "xpack.geoip.downloader.enabled",
+}
+
+// Given a setting name, return a downcased version with delimiters removed.
+func squashSetting(setting string) string {
+ downcased := strings.ToLower(setting)
+ de_dotted := strings.Replace(downcased, ".", "", -1)
+ de_underscored := strings.Replace(de_dotted, "_", "", -1)
+ return de_underscored
+}
+
+// Given a setting name like "pipeline.workers" or "PIPELINE_UNSAFE_SHUTDOWN",
+// return the canonical setting name. eg. 'pipeline.unsafe_shutdown'
+func normalizeSetting(setting string) (string, error) {
+ for _, validSetting := range validSettings {
+ if squashSetting(setting) == squashSetting(validSetting) {
+ return validSetting, nil
+ }
+ }
+ return "", errors.New("Invalid setting: " + setting)
+}
+
+func main() {
+ if len(os.Args) != 2 {
+ log.Fatalf("usage: env2yaml FILENAME")
+ }
+ settingsFilePath := os.Args[1]
+
+ settingsFile, err := ioutil.ReadFile(settingsFilePath)
+ if err != nil {
+ log.Fatalf("error: %v", err)
+ }
+
+ // Read the original settings file into a map.
+ settings := make(map[string]interface{})
+ err = yaml.Unmarshal(settingsFile, &settings)
+ if err != nil {
+ log.Fatalf("error: %v", err)
+ }
+
+ // Merge any valid settings found in the environment.
+ foundNewSettings := false
+ for _, line := range os.Environ() {
+ kv := strings.SplitN(line, "=", 2)
+ key := kv[0]
+ setting, err := normalizeSetting(key)
+ if err == nil {
+ foundNewSettings = true
+ log.Printf("Setting '%s' from environment.", setting)
+ // we need to keep ${KEY} in the logstash.yml to let Logstash decide using ${KEY}'s value from either keystore or environment
+ settings[setting] = fmt.Sprintf("${%s}", key)
+ }
+ }
+
+ if foundNewSettings {
+ output, err := yaml.Marshal(&settings)
+ if err != nil {
+ log.Fatalf("error: %v", err)
+ }
+
+ stat, err := os.Stat(settingsFilePath)
+ if err != nil {
+ log.Fatalf("error: %v", err)
+ }
+
+ err = ioutil.WriteFile(settingsFilePath, output, stat.Mode())
+ if err != nil {
+ log.Fatalf("error: %v", err)
+ }
+ }
+}
diff --git a/logstash_9.0.0-rc1/env2yaml/go.mod b/logstash_9.0.0-rc1/env2yaml/go.mod
new file mode 100644
index 0000000..7600007
--- /dev/null
+++ b/logstash_9.0.0-rc1/env2yaml/go.mod
@@ -0,0 +1,5 @@
+module logstash/env2yaml
+
+go 1.21
+
+require gopkg.in/yaml.v2 v2.4.0
diff --git a/logstash_9.0.0-rc1/env2yaml/go.sum b/logstash_9.0.0-rc1/env2yaml/go.sum
new file mode 100644
index 0000000..7534661
--- /dev/null
+++ b/logstash_9.0.0-rc1/env2yaml/go.sum
@@ -0,0 +1,3 @@
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
diff --git a/logstash_7.17.28/pipeline/default.conf b/logstash_9.0.0-rc1/pipeline/default.conf
similarity index 100%
copy from logstash_7.17.28/pipeline/default.conf
copy to logstash_9.0.0-rc1/pipeline/default.confRelevant Maintainers:
|
|
Any thoughts on #18475 (comment)?
(on that last point, the best way to get "userspace architecture" from an RPM-based distribution that I've found is +LABEL name="Kibana" \
+ maintainer="infra@elastic.co" \
+ vendor="Elastic" \
+ version="9.0.0-rc1" \
+ release="1" \
+ summary="Kibana" \
+ description="Your window into the Elastic Stack."See #3540, especially #3540 (comment):
And #3540 (comment):
+RUN for iter in 1 2 3 4 5 6 7 8 9 10; do \This (and similar constructions in other images) are weird -- I understand that these package installations have been flaky in the past, but they shouldn't be, and our build system will auto-retry failing builds. Can you elaborate on why you're still carrying this? +RUN mkdir /licenses && cp LICENSE.txt /licenses/LICENSEThis is harmless but kind of odd -- can you elaborate on what it's for? (If it's important, would a symlink be sufficient? If not, can you make it a hard link instead, and created in the same layer as +COPY env2yaml/env2yaml.go /tmp/go/src/env2yaml/env2yaml.go
+COPY env2yaml/go.mod /tmp/go/src/env2yaml/go.mod
+COPY env2yaml/go.sum /tmp/go/src/env2yaml/go.sumThis is kind of verbose - it's not a problem, but any reason not to simply It's not ideal that this means the full Go file ends up as part of the build context we review, but that's mostly OK too, since it's pretty small (it just means we might make suggestions in that code too, or ask more strongly that it move later if it grows too much). +COPY --from=builder-env2yaml /tmp/go/src/env2yaml/env2yaml /usr/local/bin/env2yaml
COPY config/pipelines.yml config/log4j2.properties config/log4j2.file.properties config/
+COPY config/logstash-full.yml config/logstash.yml
COPY pipeline/default.conf pipeline/logstash.conf
...
COPY bin/docker-entrypoint /usr/local/bin/In general, every |
+RUN go buildIt's probably also a good idea to include |
+ microdnf update --setopt=tsflags=nodocs -y && \We'd prefer to see package upgrades happen in the base images directly (and the UBI images are updated pretty proactively), so this should be removed. |
# 1. Sync the user and group permissions of /etc/passwd
# 2. Set correct permissions of the entrypoint
# 3. Ensure that there are no files with setuid or setgid, in order to mitigate "stackclash" attacks.
# We've already run this in previous layers so it ought to be a no-op.
# 4. Replace OpenJDK's built-in CA certificate keystore with the one from the OS
# vendor. The latter is superior in several ways.
# REF: https://github.com/elastic/elasticsearch-docker/issues/171
# 5. Tighten up permissions on the ES home dir (the permissions of the contents are handled earlier)
# 6. You can't install plugins that include configuration when running as `elasticsearch` and the `config`
# dir is owned by `root`, because the installed tries to manipulate the permissions on the plugin's
# config directory.
RUN chmod g=u /etc/passwd && \
chmod 0555 /usr/local/bin/docker-entrypoint.sh && \
find / -xdev -perm -4000 -exec chmod ug-s {} + && \
chmod 0775 /usr/share/elasticsearch && \
chown elasticsearch bin config config/jvm.options.d data logs pluginsThis is all odd as an explicit layer -- modifications to |
|
@tianon Could we address your changes as a follow up, decoupled form this PR. I'm happy to facilitate that with the maintainers of the images and make sure this happens. Would like to make the new versions of ELK available through dockerhub as close as possible to the release into other channels. 9.0.0-RC1 is out everywhere else except for official images for several days now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.