Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

components/*: Forbid write access to root filesystem #1600

Merged
merged 2 commits into from
Jan 28, 2022
Merged

components/*: Forbid write access to root filesystem #1600

merged 2 commits into from
Jan 28, 2022

Conversation

ArthurSens
Copy link
Member

Description

This is part of the initiative to tighten security in kube-prometheus - #1595.

An attacker who has access to a container, can create files and download scripts as he wishes, and modify the underlying application running on the container. If a container needs to write to the filesystem, a strict volume mount should be used.

Following remediation docs from https://hub.armo.cloud/docs/c-0017

Fixes #1595

Type of change

What type of changes does your code introduce to the kube-prometheus? Put an x in the box that apply.

  • CHANGE (fix or feature that would cause existing functionality to not work as expected)
  • FEATURE (non-breaking change which adds functionality)
  • BUGFIX (non-breaking change which fixes an issue)
  • ENHANCEMENT (non-breaking change which improves existing functionality)
  • NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. Later this will be copied to the changelog file.

Forbid write access to root filesystem for all components

@ArthurSens
Copy link
Member Author

Failing test seems to be unrelated 🤔

@philipgough
Copy link
Contributor

@ArthurSens do we want to lower the threshold for make kubescape as a result of this and include it here?

@ArthurSens
components/*: Forbid write access to root filesystem
57c46a2 
Signed-off-by: GitHub <noreply@github.com>
@ArthurSens
Copy link
Member Author

Ah yes, thanks for the reminder

@ArthurSens
Adjust Kubescape threshold
98664db 
Signed-off-by: ArthurSens <arthursens2005@gmail.com>
@ArthurSens
Copy link
Member Author

ArthurSens commented Jan 27, 2022 •
edited
Loading

Ps: I haven't addressed the FIXME comments yet because, although PRs are getting merged upstream, we still haven't updated the dependencies to get them into this repository 🙂

philipgough
philipgough approved these changes Jan 27, 2022
View reviewed changes
Copy link
Contributor

@philipgough philipgough left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't addressed the FIXME comments yet because, although PRs are getting merged upstream, we still haven't updated the dependencies to get them into this repository

yes, makes sense.

This LGTM but will leave for @paulfantom to take a look prior to merge

paulfantom
paulfantom approved these changes Jan 28, 2022
View reviewed changes
Copy link
Member

@paulfantom paulfantom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@ArthurSens ArthurSens merged commit b113c45 into prometheus-operator:main Jan 28, 2022
@ArthurSens ArthurSens deleted the readOnlyRootFilesystem branch January 28, 2022 12:09
@ArthurSens ArthurSens mentioned this pull request Feb 2, 2022
Merged
5 tasks
@itspngu
Copy link

itspngu commented Feb 7, 2022

@ArthurSens sorry for commenting on a closed PR, but I've traced an issue with our Grafana deployment crashlooping to this change. We're using the jssonet libraries, and the Grafana pod kept crashlooping with this being the only log entry:

Error: ✗ failed to create temporary file: open /tmp/2943756055.zip: read-only file system

Is there something wrong with our configuration?

local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // Monitor all namespaces
  (import 'kube-prometheus/addons/all-namespaces.libsonnet') +
  // Soft anti-affinity to prevent Prometheus and Alertmanager from running on the same node (if possible)
  (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
  // Custom metrics API allows the HPA v2 to scale based on arbirary metrics
  (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
  // Remove some monitoring rules for inaccessible control plane components
  (import 'kube-prometheus/addons/managed-cluster.libsonnet') {
    values+:: {
      // [...]
      grafana+: {
        // TODO: externalize grafana setup into own file so both kube-prometheus and kube-thanos can be cleanly referenced
        datasources: [
          {
            name: 'thanos',
            type: 'prometheus',
            access: 'proxy',
            // TODO: un-hardcode
            url: 'http://thanos-query.' + common.namespace + '.svc:9090',
            orgId: 1,
            version: 1,
            editable: false,
          },
        ],
        plugins+: [
          'camptocamp-prometheus-alertmanager-datasource',
        ],
        dashboards+:
          thanosMixin.grafanaDashboards +
          certManagerMixin.grafanaDashboards,
        config+: {
          sections+: {
            server+: {
              root_url: 'https://grafana.' + common.ingress.domain,
            },
          },
        },
      },

Everything starts up fine once I set readOnlyFilesystem: false in the generated yaml manifest for the Grafana Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 8.3.4
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: grafana
      app.kubernetes.io/name: grafana
      app.kubernetes.io/part-of: kube-prometheus
  template:
    metadata:
      annotations:
        checksum/grafana-config: 1565e43c513d51f2bc2913fa57de62ae
        checksum/grafana-dashboardproviders: bc79f12017c019002ed650d44571a465
        checksum/grafana-datasources: c5eca10c070124efca8530cff8e56ac9
      labels:
        app.kubernetes.io/component: grafana
        app.kubernetes.io/name: grafana
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 8.3.4
    spec:
      automountServiceAccountToken: false
      containers:
        - env:
            - name: GF_INSTALL_PLUGINS
              value: camptocamp-prometheus-alertmanager-datasource,camptocamp-prometheus-alertmanager-datasource
          image: grafana/grafana:8.3.4
          name: grafana
          ports:
            - containerPort: 3000
              name: http
          readinessProbe:
            httpGet:
              path: /api/health
              port: http
          resources:
            limits:
              cpu: 200m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            # Line below is changed, everything else is "as-is" output from jsonnet-bundler
            readOnlyRootFilesystem: false
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
              readOnly: false
            - mountPath: /etc/grafana/provisioning/datasources
              name: grafana-datasources
              readOnly: false
            - mountPath: /etc/grafana/provisioning/dashboards
              name: grafana-dashboards
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/alertmanager-overview
              name: grafana-dashboard-alertmanager-overview
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/apiserver
              name: grafana-dashboard-apiserver
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/bucket-replicate
              name: grafana-dashboard-bucket-replicate
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/cert-manager
              name: grafana-dashboard-cert-manager
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/cluster-total
              name: grafana-dashboard-cluster-total
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/compact
              name: grafana-dashboard-compact
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/controller-manager
              name: grafana-dashboard-controller-manager
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/grafana-overview
              name: grafana-dashboard-grafana-overview
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/k8s-resources-cluster
              name: grafana-dashboard-k8s-resources-cluster
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/k8s-resources-namespace
              name: grafana-dashboard-k8s-resources-namespace
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/k8s-resources-node
              name: grafana-dashboard-k8s-resources-node
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/k8s-resources-pod
              name: grafana-dashboard-k8s-resources-pod
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/k8s-resources-workload
              name: grafana-dashboard-k8s-resources-workload
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/k8s-resources-workloads-namespace
              name: grafana-dashboard-k8s-resources-workloads-namespace
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/kubelet
              name: grafana-dashboard-kubelet
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/namespace-by-pod
              name: grafana-dashboard-namespace-by-pod
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/namespace-by-workload
              name: grafana-dashboard-namespace-by-workload
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/node-cluster-rsrc-use
              name: grafana-dashboard-node-cluster-rsrc-use
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/node-rsrc-use
              name: grafana-dashboard-node-rsrc-use
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/nodes
              name: grafana-dashboard-nodes
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/overview
              name: grafana-dashboard-overview
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/persistentvolumesusage
              name: grafana-dashboard-persistentvolumesusage
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/pod-total
              name: grafana-dashboard-pod-total
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/prometheus-remote-write
              name: grafana-dashboard-prometheus-remote-write
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/prometheus
              name: grafana-dashboard-prometheus
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/proxy
              name: grafana-dashboard-proxy
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/query-frontend
              name: grafana-dashboard-query-frontend
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/query
              name: grafana-dashboard-query
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/receive
              name: grafana-dashboard-receive
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/rule
              name: grafana-dashboard-rule
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/scheduler
              name: grafana-dashboard-scheduler
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/sidecar
              name: grafana-dashboard-sidecar
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/store
              name: grafana-dashboard-store
              readOnly: false
            - mountPath: /grafana-dashboard-definitions/0/workload-total
              name: grafana-dashboard-workload-total
              readOnly: false
            - mountPath: /etc/grafana
              name: grafana-config
              readOnly: false
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        fsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: grafana
      volumes:
        - emptyDir: {}
          name: grafana-storage
        - name: grafana-datasources
          secret:
            secretName: grafana-datasources
        - configMap:
            name: grafana-dashboards
          name: grafana-dashboards
        - configMap:
            name: grafana-dashboard-alertmanager-overview
          name: grafana-dashboard-alertmanager-overview
        - configMap:
            name: grafana-dashboard-apiserver
          name: grafana-dashboard-apiserver
        - configMap:
            name: grafana-dashboard-bucket-replicate
          name: grafana-dashboard-bucket-replicate
        - configMap:
            name: grafana-dashboard-cert-manager
          name: grafana-dashboard-cert-manager
        - configMap:
            name: grafana-dashboard-cluster-total
          name: grafana-dashboard-cluster-total
        - configMap:
            name: grafana-dashboard-compact
          name: grafana-dashboard-compact
        - configMap:
            name: grafana-dashboard-controller-manager
          name: grafana-dashboard-controller-manager
        - configMap:
            name: grafana-dashboard-grafana-overview
          name: grafana-dashboard-grafana-overview
        - configMap:
            name: grafana-dashboard-k8s-resources-cluster
          name: grafana-dashboard-k8s-resources-cluster
        - configMap:
            name: grafana-dashboard-k8s-resources-namespace
          name: grafana-dashboard-k8s-resources-namespace
        - configMap:
            name: grafana-dashboard-k8s-resources-node
          name: grafana-dashboard-k8s-resources-node
        - configMap:
            name: grafana-dashboard-k8s-resources-pod
          name: grafana-dashboard-k8s-resources-pod
        - configMap:
            name: grafana-dashboard-k8s-resources-workload
          name: grafana-dashboard-k8s-resources-workload
        - configMap:
            name: grafana-dashboard-k8s-resources-workloads-namespace
          name: grafana-dashboard-k8s-resources-workloads-namespace
        - configMap:
            name: grafana-dashboard-kubelet
          name: grafana-dashboard-kubelet
        - configMap:
            name: grafana-dashboard-namespace-by-pod
          name: grafana-dashboard-namespace-by-pod
        - configMap:
            name: grafana-dashboard-namespace-by-workload
          name: grafana-dashboard-namespace-by-workload
        - configMap:
            name: grafana-dashboard-node-cluster-rsrc-use
          name: grafana-dashboard-node-cluster-rsrc-use
        - configMap:
            name: grafana-dashboard-node-rsrc-use
          name: grafana-dashboard-node-rsrc-use
        - configMap:
            name: grafana-dashboard-nodes
          name: grafana-dashboard-nodes
        - configMap:
            name: grafana-dashboard-overview
          name: grafana-dashboard-overview
        - configMap:
            name: grafana-dashboard-persistentvolumesusage
          name: grafana-dashboard-persistentvolumesusage
        - configMap:
            name: grafana-dashboard-pod-total
          name: grafana-dashboard-pod-total
        - configMap:
            name: grafana-dashboard-prometheus-remote-write
          name: grafana-dashboard-prometheus-remote-write
        - configMap:
            name: grafana-dashboard-prometheus
          name: grafana-dashboard-prometheus
        - configMap:
            name: grafana-dashboard-proxy
          name: grafana-dashboard-proxy
        - configMap:
            name: grafana-dashboard-query-frontend
          name: grafana-dashboard-query-frontend
        - configMap:
            name: grafana-dashboard-query
          name: grafana-dashboard-query
        - configMap:
            name: grafana-dashboard-receive
          name: grafana-dashboard-receive
        - configMap:
            name: grafana-dashboard-rule
          name: grafana-dashboard-rule
        - configMap:
            name: grafana-dashboard-scheduler
          name: grafana-dashboard-scheduler
        - configMap:
            name: grafana-dashboard-sidecar
          name: grafana-dashboard-sidecar
        - configMap:
            name: grafana-dashboard-store
          name: grafana-dashboard-store
        - configMap:
            name: grafana-dashboard-workload-total
          name: grafana-dashboard-workload-total
        - name: grafana-config
          secret:
            secretName: grafana-config

Are we expected to declare an explicit mount for /tmp, e.g. an EmptyDir? If yes, I feel like this should be part of the generated yaml, shouldn't it? If this is actually a bug, let me know and I'll open an issue.

@philipgough
Copy link
Contributor

@itspngu - this is indeed a bug - it would be great if you can open the issue. Thanks

@itspngu itspngu mentioned this pull request Feb 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@paulfantom paulfantom paulfantom approved these changes

@philipgough philipgough philipgough approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Investigate updating the default 'readOnlyRootFilesystem' to enforce immutable container fs
4 participants
@ArthurSens @philipgough @itspngu @paulfantom