Add persistent Storage

Persistent Storage

When your application needs to ship a database, blob store, or other means of persisting its data, it’s useful to use kubernetes Persistent Volumes to manage the persistence and redundancy of that storage. This chapter give you a brief intro to how Replicated manages Persistent Volumes in Kubernetes, and some tips on how to leverage them to ship databases and other storage technology alongside your application.

Leveraging PVCs in StatefulSets

For most stateful deployments, you’ll want to include a Kubernetes StatefulSet in your application. These are similar to Deployments, but with some extra features for stateful components, including direct integration with Persistent Volumes.

An example StatefulSet is shown below, note especially the volumeClaimTemplates field, which is used to configure how Persistent Volumes are allocated.

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: "mysql"
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5
        ports:
        - containerPort: 3306
          name: mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "default"
      resources:
        requests:
          storage: 100Gi

Rook

Replicated’s Kubernetes scheduler uses Rook to dynamically provision storage for Persistent Volume Claims required by Replicated and your app.

Replication

Replicated’s install script will default to creating a cluster backed by the host directory /opt/replicated/rook. Each file in the cluster begins with a single copy, i.e. without replication. When a node is added or removed from the cluster, Replicated will automatically adjust the replication level up to a maximum of 3.

Prior to Replicated 2.30, customers had to manually increase the replication level with the following command:

kubectl -n rook-ceph patch pool replicapool --type='json' -p='[{"op": "replace", "path": "/spec/replicated/size", "value": 2}]'

Dynamic Provisioning in Cloud Environments

The Cloud Controller Manager (CCM) was introduced in Kuberentes 1.6 to offload control loops that integrate with cloud services from the Kubernetes Controller Manager (KCM). Three of the four cloud dependent controllers - node, route, and service - have been moved to the Cloud Controller Manager. A fourth, the volume controller, was replaced with the Flex Volume plugin framework rather than being ported directlly from the KCM to the CCM. As of Kubernetes 1.10, a newer plugin system, Container Storage Interface (CSI) has been designated as the successor to Flex Volumes. CSI plugins are similar to Flex Volumes but run in pods rather than as binaries on the host.

Replicated is monitoring progress of the CSI framework and may add support for plugins that will dynamically provision volumes such as EBS on AWS. For now, customers who wish to use EBS volumes with Replicated may choose to mount a volume on each host at /opt/replicated/rook, effectively backing Rook with EBS storage. Alternatively, Rook can be disabled entirely by setting the storage_provisioner parameter to 0, which would require sysadmins to manually provision an EBS volume for each of the four Persistent Volume Claims defined by Replicated in addition to any required to run your app.

Tips

• Deployments with Pods that mount Persistent Volumes should specify an update strategy type of Recreate or a RollingUpdate with a maxUnavailable value of 1.
• The default RollingUpdate with maxUnavailable value of 25% will prevent old Pods from terminating and yielding their Persistent Volumes, which will keep new Pods from starting.

For more information on managing the storage needs of Kubernetes in customer environments, see Managing Storage.