|
| 1 | +--- |
| 2 | +slug: leverage-kubernetes-data-oriented-projects-with-portworx |
| 3 | +title: "Leveraging Kubernetes Data-Oriented Projects with Portworx" |
| 4 | +title_meta: "How to Leverage Kubernetes Data-Oriented Projects with Portworx" |
| 5 | +description: "Explore how Portworx, a leading cloud-native storage platform for Kubernetes, enhances data and SaaS management with efficient cloud storage solutions." |
| 6 | +authors: ["Cameron Laird"] |
| 7 | +contributors: ["Cameron Laird","Adam Overa"] |
| 8 | +published: 2024-06-25 |
| 9 | +keywords: ['portworx cloud-native storage platform','kubernetes enterprise storage platform','cloud storage','data storage','data management','saas management','dbaas management'] |
| 10 | +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' |
| 11 | +external_resources: |
| 12 | +- '[Portworx: Database Services](https://portworx.com/products/portworx-data-services/)' |
| 13 | +- '[Portworx Documentation: Kubernetes Operations](https://docs.portworx.com/portworx-enterprise/operations/operate-kubernetes)' |
| 14 | +- '[Portworx: Kubernetes Storage Use Case](https://portworx.com/use-case/kubernetes-storage/)' |
| 15 | +--- |
| 16 | + |
| 17 | +Management of data at scale is crucial for deriving actionable insights, and an effective data platform can provide those kinds of insights. A data platform is the technology infrastructure used for the collection, storage, transaction processing, and analysis of varied data at scale. It simplifies engineering tasks such as expanding the storage available to an application or encrypting project secrets. |
| 18 | + |
| 19 | +Portworx handles advanced storage and data management capabilities for cloud-native environments. This guide provides step-by-step instructions for installing Portworx on an existing Kubernetes cluster. It then walks through setting up a model project to demonstrate Portworx' capabilities. |
| 20 | + |
| 21 | +## What Is Portworx? |
| 22 | + |
| 23 | +Portworx enables deployment and management of storage and data services specifically in containerized environments. It also handles data replication, snapshots, backups, and data recovery, allowing application systems to focus on their own specific requirements. Since Portworx itself is cloud-native, it plays a crucial role in helping other systems maximize the capabilities of the cloud. |
| 24 | + |
| 25 | +Current cloud computing practices face several challenges, particularly the difficulty of managing Kubernetes instances in the real-world. Portworx mitigates some of these challenges. |
| 26 | + |
| 27 | +A limited version of the Portworx Storage Platform is available for free. It allows for an implementation of object storage for a single distributed cluster. This guide focuses on the free, downloadable software that you can install and run for your own educational and small-scale uses. |
| 28 | + |
| 29 | +## How Portworx Relates to Kubernetes, Kafka, and Cassandra |
| 30 | + |
| 31 | +Portworx integrates with widely known software systems such as Kubernetes, Kafka, and Cassandra: |
| 32 | + |
| 33 | +- [**Kubernetes**](/docs/guides/kubernetes/) serves as the foundation of most Portworx implementations. However, Portworx is also compatible with other container orchestration systems. |
| 34 | + |
| 35 | +- [**Cassandra**](/docs/guides/databases/cassandra/) is an open source distributed database management system that emphasizes economical operation, high availability, and wide-column semantics. Portworx addresses several of the challenges involved in configuring and operating Cassandra. For example, when running Cassandra in containers managed by Kubernetes, Portworx can effectively control memory, resource quotas, and/or CPU cores per Kubernetes cluster. |
| 36 | + |
| 37 | +- [**Kafka**](/docs/guides/what-is-apache-kafka/) is a widely used open source distributed event store and stream-processing platform. In much the same way a traditional database system manages **records** of data, Kafka manages **events**. For Kafka to perform optimally, it needs a high-performance underlying storage system, and Portworx is a good choice. Teams and individuals often initially adopt Portworx to meet requirements for hosting or upgrading Kafka. Portworx also offers white papers specifically on the [operation of Kafka in a Kubernetes environment](https://portworx.com/blog/deploying-kafka-on-kubernetes-using-portworx-data-services/). |
| 38 | + |
| 39 | +## Before You Begin |
| 40 | + |
| 41 | +1. Create a Kubernetes cluster that meets the [Portworx installation prerequisites](https://docs.portworx.com/portworx-enterprise/install-portworx/prerequisites). A **Shared CPU**, **Linode 8 GB** plan is suitable. You must have `kubectl` configured on your local machine to interact with the cluster. See our [Getting Stated with Kubernetes](/docs/products/compute/kubernetes/get-started/) guide for instructions. Also, take note of the Kubernetes version running on your cluster as it is needed later. |
| 42 | + |
| 43 | +1. The Portworx installation prerequisites also include a backing drive (i.e. Volume) for each of three nodes, which must be at least 8 GB. Follow our [Getting Started with Block Storage](/docs/products/storage/block-storage/get-started/) guide to create and attach a 10 GB Volume to each node. Creating volumes via the **Storage** tab of the individual Kubernetes instances is more efficient than via **Volumes**, as it creates *and* attaches in one step. |
| 44 | + |
| 45 | +1. Sign up for a personal account on [Portworx Central](https://central.portworx.com). |
| 46 | + |
| 47 | +{{< note >}} |
| 48 | +This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/docs/guides/linux-users-and-groups/) guide. |
| 49 | +{{< /note >}} |
| 50 | + |
| 51 | +## Portworx Installation |
| 52 | + |
| 53 | +To install Portworx, use the basic installation model on an existing Kubernetes cluster. This can be any existing Kubernetes cluster, whether using Linode Kubernetes Engine or a manually constructed setup. You can also use [kind](https://kind.sigs.k8s.io/) for an installation purely within your desktop development environment. |
| 54 | + |
| 55 | +Portworx is not an open source system, though it supports many [individual open source components](https://portworx.com/products/open-source/), and some of its [licenses](https://docs.portworx.com/portworx-enterprise/operations/licensing) involve no fee. However, installation is generally done through the Portworx website and not via standard command-line package managers such as `apt` or `brew`. |
| 56 | + |
| 57 | +Follow the steps in the below sections to install Portworx on an existing Kubernetes cluster. |
| 58 | + |
| 59 | +### The Wizard |
| 60 | + |
| 61 | +1. Open a web browser and log in to [Portworx Central](https://central.portworx.com). |
| 62 | + |
| 63 | +1. Select **Get Started** from the **Welcome to Portworx** section of the Portworx Central home page: |
| 64 | + |
| 65 | +  |
| 66 | + |
| 67 | +1. Choose the **Portworx Essentials/Portworx CSI** fee-free license for demonstration or proof-of-concept workloads: |
| 68 | + |
| 69 | +  |
| 70 | + |
| 71 | +1. Choose `DAS/SAN` as **Platform** and `None` for **Distribution Name**. Retain `portworx` as the default **Namespace**, but change the **K8s Version** to match the Kubernetes version of your cluster (e.g. `1.30.2`). |
| 72 | + |
| 73 | +  |
| 74 | + |
| 75 | + {{< note >}} |
| 76 | + Use the following command to check your version of Kubernetes: |
| 77 | + |
| 78 | + ```command |
| 79 | + kubectl version |
| 80 | + ``` |
| 81 | + {{< /note >}} |
| 82 | + |
| 83 | +1. Select **Save Spec** to generate `kubectl` commands for `Operator` and `StorageCluster`, which reflect the specifications chosen for the Portworx installation. Copy the `kubectl` commands for use in the next section. |
| 84 | + |
| 85 | +  |
| 86 | + |
| 87 | +1. To save this configuration, fill in **Spec Name** and **Spec Tags** then click **Save Spec** again. |
| 88 | + |
| 89 | +1. Your generated spec manifest is now available in the **Spec List** section of Portworx Central. You can download it at any time by clicking the three vertical dots under **Actions** and choosing **Download**. |
| 90 | + |
| 91 | +  |
| 92 | +
|
| 93 | +### Deployment |
| 94 | +
|
| 95 | +1. Use the first `kubectl` command generated in the previous section to deploy the `Operator` specification. The command structure should follow that of the example command below, with {{< placeholder "PORTWORX_VERSION_NUMBER" >}} and {{< placeholder "KUBERNETES_VERSION_NUMBER" >}} matching your respective Portworx and Kubernetes versions: |
| 96 | +
|
| 97 | + ```command |
| 98 | + kubectl apply -f 'https://install.portworx.com/{{< placeholder "PORTWORX_VERSION_NUMBER" >}}?comp=pxoperator&kbver={{< placeholder "KUBERNETES_VERSION_NUMBER" >}}&ns=portworx' |
| 99 | + ``` |
| 100 | +
|
| 101 | + Sample output: |
| 102 | +
|
| 103 | + ```output |
| 104 | + namespace/portworx created |
| 105 | + serviceaccount/portworx-operator created |
| 106 | + clusterrole.rbac.authorization.k8s.io/portworx-operator created |
| 107 | + clusterrolebinding.rbac.authorization.k8s.io/portworx-operator created |
| 108 | + deployment.apps/portworx-operator created |
| 109 | + ``` |
| 110 | +
|
| 111 | +1. Use the second `kubectl` command generated in the previous section to deploy the `StorageCluster` specification. The command structure should resemble the example command below, with {{< placeholder "PX_USER_ID" >}} and {{< placeholder "PX_CLUSTER_ID" >}} being unique to your Portworx Central account: |
| 112 | +
|
| 113 | + ```command |
| 114 | + kubectl apply -f 'https://install.portworx.com/{{< placeholder "PORTWORX_VERSION_NUMBER" >}}?operator=true&mc=false&kbver={{< placeholder "KUBERNETES_VERSION_NUMBER" >}}&ns=portworx&oem=esse&user={{< placeholder "PX_USER_ID" >}}&b=true&iop=6&c=px-cluster-{{< placeholder "PX_CLUSTER_ID" >}}&stork=true&csi=true&mon=true&tel=true&st=k8s&promop=true' |
| 115 | + ``` |
| 116 | +
|
| 117 | + Sample output: |
| 118 | +
|
| 119 | + ```output |
| 120 | + storagecluster.core.libopenstorage.org/px-cluster-{{< placeholder "PX_CLUSTER_ID" >}} created |
| 121 | + secret/px-essential created |
| 122 | + ``` |
| 123 | +
|
| 124 | + {{< note >}} |
| 125 | + Should you receive any errors, you can use the **Generate Spec** screen to view up-to-date commands. |
| 126 | + {{< /note >}} |
| 127 | +
|
| 128 | +### Verification |
| 129 | +
|
| 130 | +1. Monitor the status of Portworx nodes with the following command: |
| 131 | +
|
| 132 | + ```command |
| 133 | + kubectl -n portworx get storagenodes -l name=portworx |
| 134 | + ``` |
| 135 | +
|
| 136 | + Once the deployments finish, each Portworx node appears as `Online`: |
| 137 | +
|
| 138 | + ```output |
| 139 | + NAME ID STATUS VERSION AGE |
| 140 | + lke194968-280433-369bf4810000 f2522d07-0b59-482a-a8ae-bd2854fd7bc4 Online 3.1.2.0-fb52ced 4m46s |
| 141 | + lke194968-280433-438a8b610000 b1920afd-5326-48dc-9572-af8e638fd92b Online 3.1.2.0-fb52ced 4m45s |
| 142 | + lke194968-280433-527b95040000 58649b41-b4c9-4c56-b983-e27a61c9f582 Online 3.1.2.0-fb52ced 4m46s |
| 143 | + ``` |
| 144 | +
|
| 145 | +1. Use the following command to monitor the status of an individual node, replacing {{< placeholder "NODE_NAME" >}} with the `NAME` of one of the nodes listed in the prior command's output: |
| 146 | + |
| 147 | + ```command |
| 148 | + kubectl -n portworx describe storagenode {{< placeholder "NODE_NAME" >}} |
| 149 | + ``` |
| 150 | + |
| 151 | +At this point, your working Kubernetes cluster includes a small Portworx deployment with a permanent fee-free license. You can use your cluster for educational practice, proofs-of-concept, or other demonstrations of Portworx' capabilities. |
| 152 | +
|
| 153 | +## Run a Model Portworx Project |
| 154 | +
|
| 155 | +Among the examples found in the [Portworx documentation](https://docs.portworx.com/) is [Run Kafka on Kubernetes at Scale with Portworx](https://blog.purestorage.com/purely-technical/run-kafka-kubernetes-scale-portworx/). While thousands of organizations already deploy Kafka manually, Portworx can enhance the process. Replacement of a manual deployment with Portworx' mediation automates disaster recovery, application-specific high availability, backup services, and capacity management. |
| 156 | + |
| 157 | +Using the Portworx installation from the preceding section, follow the steps below to get started: |
| 158 | + |
| 159 | +1. Create a specification file named `sc-kafka-rf2.yaml`: |
| 160 | + |
| 161 | + ```command |
| 162 | + nano sc-kafka-rf2.yaml |
| 163 | + ``` |
| 164 | + |
| 165 | + Paste in the following contents, and save your changes: |
| 166 | + |
| 167 | + ```file {title="sc-kafka-rf2.yaml" lang="yaml"} |
| 168 | + kind: StorageClass |
| 169 | + apiVersion: storage.k8s.io/v1 |
| 170 | + metadata: |
| 171 | + name: px-sc-kafka-repl2 |
| 172 | + provisioner: kubernetes.io/portworx-volume |
| 173 | + allowVolumeExpansion: true |
| 174 | + parameters: |
| 175 | + repl: "2" |
| 176 | + priority_io: "high" |
| 177 | + io_profile: "db_remote" |
| 178 | + cow_ondemand: "true" |
| 179 | + disable_io_profile_protection: "1" |
| 180 | + nodiscard: "false" |
| 181 | + group: "kafka-broker-rep2" |
| 182 | + fg: "false" |
| 183 | + ``` |
| 184 | + |
| 185 | + This storage specification provides several automations, including replication. The `repl: "2"` parameter maintains two full replicas of broker data (i.e. Kafka's content) across the failure domains of the hosting Kubernetes cluster. This ensures that Kafka continues without downtime should a node fail. |
| 186 | +
|
| 187 | +1. Use the following command to apply the `StorageClass`: |
| 188 | +
|
| 189 | + ```command |
| 190 | + kubectl apply -f sc-kafka-rf2.yaml |
| 191 | + ``` |
| 192 | +
|
| 193 | + Sample output: |
| 194 | +
|
| 195 | + ```output |
| 196 | + storageclass.storage.k8s.io/px-sc-kafka-repl2 created |
| 197 | + ``` |
| 198 | +
|
| 199 | +Storage-level replication makes it possible for Portworx to identify and re-assign healthy storage in the event of failure. This keeps data available while replicating almost immediately. |
| 200 | +
|
| 201 | +While Portworx generally requires several dozens of lines of configuration files, it's typically less than what administrators may use to maintain a Kubernetes cluster. When terabytes of data are involved, Portworx's efficient data utilization can result in large cost savings. |
0 commit comments