Merge remote-tracking branch 'origin/master' into feature/arangodbexporter

Author: lamai93

CHANGELOG.md

@@ -1,8 +1,19 @@
 # Change Log
 
-## [0.3.10](---) (XXXX-XX-XX)
+## [0.3.10](---) (2019-04-04)
 - Added Pod Disruption Budgets for all server groups in production mode.
 - Added Priority Class Name to be specified per server group.
+- Forward resource requirements to k8s.
+- Automatic creation of randomized root password on demand.
+- Volume resizing (only enlarge).
+- Allow to disable liveness probes, increase timeouts in defaults.
+- Handle case of all coordinators gone better.
+- Added `MY_NODE_NAME` and `NODE_NAME` env vars for all pods.
+- Internal communications with ArangoDB more secure through tokens which
+  are limited to certain API paths.
+- Rolling upgrade waits till all shards are in sync before proceeding to
+  next dbserver, even if it takes longer than 15 min.
+- Improve installation and upgrade instructions in README.
 
 ## [0.3.9](https://github.com/arangodb/kube-arangodb/tree/0.3.9) (2019-02-28)
 [Full Changelog](https://github.com/arangodb/kube-arangodb/compare/0.3.8...0.3.9)

MAINTAINERS.md

@@ -2,20 +2,35 @@
 
 ## Running tests
 
-To run the entire test set, run:
+To run the entire test set, first set the following environment variables:
+
+  - `DOCKERNAMESPACE` to your docker hub account
+  - `VERBOSE` to `1` (default is empty)
+  - `LONG` to `1` (default is empty, which skips lots of tests)
+  - `ARANGODB` to the name of a community image you want to test,
+    default is `arangodb/arangodb:latest`
+  - `ENTERPRISEIMAGE` to the name of an enterprise image, you want to
+    test, if not set, some tests are skipped
+  - `ARANGO_LICENSE_KEY` to the enterpise license key
+  - `KUBECONFIG` to the path to some k8s configuration with
+    credentials, this indicates which cluster to use
 
 ```bash
-export DOCKERNAMESPACE=<your docker hub account>
 make clean
 make build
 make run-tests
 ```
 
+To run only a single test, set `TESTOPTIONS` to something like
+`-test.run=TestRocksDBEncryptionSingle` where
+`TestRocksDBEncryptionSingle` is the name of the test.
+
 ## Preparing a release
 
 To prepare for a release, do the following:
 
 - Make sure all tests are OK.
+- To run a complete set of tests, see above.
 - Update the CHANGELOG manually, since the automatic CHANGELOG
   generation is switched off (did not work in many cases).
 

README.md

@@ -37,32 +37,79 @@ it is intended to be.
 | Docker for Mac Edge  | 1.10               | >= 3.3.13        | Runs  | Not intended     |                       |
 | Scaleway Kubernetes  | 1.10               | >= 3.3.13        | ?     | No               |                       |
 
-## Installation of latest release using Helm
+## WARNING
 
 **WARNING**: There is a problem with rolling upgrades in version 0.3.8.
 **DO NOT USE 0.3.8 FOR ROLLING UPGRADES.** If you are still using 0.3.8,
 then upgrade to 0.3.9 of the operator first before running any rolling
 upgrade.
 
+## Installation of latest release using Kubectl
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.10/manifests/arango-crd.yaml
+kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.10/manifests/arango-deployment.yaml
+# To use `ArangoLocalStorage`, also run
+kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.10/manifests/arango-storage.yaml
+# To use `ArangoDeploymentReplication`, also run
+kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.10/manifests/arango-deployment-replication.yaml
+```
+
+This procedure can also be used for upgrades and will not harm any
+running ArangoDB deployments.
+
+## Installation of latest release using Helm
+
+Only use this procedure for a new install of the operator. See below for
+upgrades.
+
 ```bash
 # The following will install the custom resources required by the operators.
-helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.9/kube-arangodb-crd.tgz
+helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.10/kube-arangodb-crd.tgz
 # The following will install the operator for `ArangoDeployment` &
 # `ArangoDeploymentReplication` resources.
-helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.9/kube-arangodb.tgz
+helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.10/kube-arangodb.tgz
 # To use `ArangoLocalStorage`, also run
-helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.9/kube-arangodb-storage.tgz
+helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.10/kube-arangodb-storage.tgz
 ```
 
-## Installation of latest release using Kubectl
+## Upgrading the operator using Helm
+
+To upgrade the operator to the latest version with Helm, you have to
+delete the previous deployment and then install the latest. **HOWEVER**:
+You *must not delete* the deployment of the custom resource definitions
+(CRDs), or your ArangoDB deployments will be deleted!
+
+Therefore, you have to use `helm list` to find the deployments for the
+operator (`kube-arangodb`) and of the storage operator
+(`kube-arangodb-storage`) and use `helm delete` to delete them using the
+automatically generated deployment names. Here is an example of a `helm
+list` output:
+
+```
+% helm list
+NAME            	REVISION	UPDATED                 	STATUS  	CHART                               	APP VERSION	NAMESPACE
+intent-camel    	1       	Mon Apr  8 11:37:52 2019	DEPLOYED	kube-arangodb-storage-0.3.10-preview	           	default  
+steely-mule     	1       	Sun Mar 31 21:11:07 2019	DEPLOYED	kube-arangodb-crd-0.3.9             	           	default  
+vetoed-ladybird 	1       	Mon Apr  8 11:36:58 2019	DEPLOYED	kube-arangodb-0.3.10-preview        	           	default  
+```
+
+So here, you would have to do
 
 ```bash
-kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.9/manifests/arango-crd.yaml
-kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.9/manifests/arango-deployment.yaml
+helm delete intent-camel
+helm delete vetoed-ladybird
+```
+
+but **not delete `steely-mule`**. Then you could install the new version
+with `helm install` as normal:
+
+```bash
+# The following will install the operator for `ArangoDeployment` &
+# `ArangoDeploymentReplication` resources.
+helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.10/kube-arangodb.tgz
 # To use `ArangoLocalStorage`, also run
-kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.9/manifests/arango-storage.yaml
-# To use `ArangoDeploymentReplication`, also run
-kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/0.3.9/manifests/arango-deployment-replication.yaml
+helm install https://github.com/arangodb/kube-arangodb/releases/download/0.3.10/kube-arangodb-storage.tgz
 ```
 
 ## Building

VERSION

@@ -1 +1 @@
-0.3.9+git
+0.3.10+git

deps/github.com/arangodb/go-driver/jwt/jwt.go

@@ -54,3 +54,29 @@ func CreateArangodJwtAuthorizationHeader(jwtSecret, serverID string) (string, er
 
 	return "bearer " + signedToken, nil
 }
+
+// CreateArangodJwtAuthorizationHeaderAllowedPaths calculates a JWT authorization header, for authorization
+// of a request to an arangod server, based on the given secret.
+// If the secret is empty, nothing is done.
+// Use the result of this function as input for driver.RawAuthentication.
+// Additionally allowed paths can be specified
+func CreateArangodJwtAuthorizationHeaderAllowedPaths(jwtSecret, serverID string, paths []string) (string, error) {
+	if jwtSecret == "" || serverID == "" {
+		return "", nil
+	}
+	// Create a new token object, specifying signing method and the claims
+	// you would like it to contain.
+	token := jg.NewWithClaims(jg.SigningMethodHS256, jg.MapClaims{
+		"iss":           issArangod,
+		"server_id":     serverID,
+		"allowed_paths": paths,
+	})
+
+	// Sign and get the complete encoded token as a string using the secret
+	signedToken, err := token.SignedString([]byte(jwtSecret))
+	if err != nil {
+		return "", driver.WithStack(err)
+	}
+
+	return "bearer " + signedToken, nil
+}

manifests/arango-deployment-replication.yaml

@@ -103,7 +103,7 @@ spec:
       containers:
       - name: operator
         imagePullPolicy: IfNotPresent
-        image: arangodb/kube-arangodb@sha256:3db337b992d0535b3caba590d70b089f27b812a77274a0a1b9f290b93caf7ff9
+        image: arangodb/kube-arangodb@sha256:bc17d77ca6c8f134a065db4f5dcbc07b84feb74ae10dd6666e822334f9ff9680
         args:
           - --operator.deployment-replication
         env:

manifests/arango-deployment.yaml

@@ -35,6 +35,9 @@ rules:
 - apiGroups: ["apps"]
   resources: ["deployments", "replicasets"]
   verbs: ["get"]
+- apiGroups: ["policy"]
+  resources: ["poddisruptionbudgets"]
+  verbs: ["get", "create", "delete"]
 - apiGroups: ["storage.k8s.io"]
   resources: ["storageclasses"]
   verbs: ["get", "list"]
@@ -103,7 +106,7 @@ spec:
       containers:
       - name: operator
         imagePullPolicy: IfNotPresent
-        image: arangodb/kube-arangodb@sha256:3db337b992d0535b3caba590d70b089f27b812a77274a0a1b9f290b93caf7ff9
+        image: arangodb/kube-arangodb@sha256:bc17d77ca6c8f134a065db4f5dcbc07b84feb74ae10dd6666e822334f9ff9680
         args:
           - --operator.deployment
           - --chaos.allowed=false

manifests/arango-storage.yaml

@@ -141,7 +141,7 @@ spec:
       containers:
       - name: operator
         imagePullPolicy: IfNotPresent
-        image: arangodb/kube-arangodb@sha256:3db337b992d0535b3caba590d70b089f27b812a77274a0a1b9f290b93caf7ff9
+        image: arangodb/kube-arangodb@sha256:bc17d77ca6c8f134a065db4f5dcbc07b84feb74ae10dd6666e822334f9ff9680
         args:
         - --operator.storage
         env:

manifests/arango-test.yaml

@@ -11,10 +11,13 @@ rules:
   resources: ["nodes"]
   verbs: ["list"]
 - apiGroups: [""]
-  resources: ["pods", "services", "persistentvolumes", "persistentvolumeclaims", "secrets", "serviceaccounts"]
+  resources: ["pods", "services", "persistentvolumes", "persistentvolumeclaims", "secrets", "serviceaccounts", "pods/log"]
   verbs: ["*"]
 - apiGroups: ["apps"]
-  resources: ["daemonsets"]
+  resources: ["daemonsets", "deployments"]
+  verbs: ["*"]
+- apiGroups: ["scheduling.k8s.io"]
+  resources: ["priorityclasses"]
   verbs: ["*"]
 
 ---

pkg/deployment/context_impl.go

@@ -380,3 +380,13 @@ func (d *Deployment) GetExpectedPodArguments(apiObject metav1.Object, deplSpec a
 	agents api.MemberStatusList, id string, version driver.Version) []string {
 	return d.resources.GetExpectedPodArguments(apiObject, deplSpec, group, agents, id, version)
 }
+
+// GetShardSyncStatus returns true if all shards are in sync
+func (d *Deployment) GetShardSyncStatus() bool {
+	return d.resources.GetShardSyncStatus()
+}
+
+// InvalidateSyncStatus resets the sync state to false and triggers an inspection
+func (d *Deployment) InvalidateSyncStatus() {
+	d.resources.InvalidateSyncStatus()
+}