You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/server-admin-4.8/modules/operator/pages/introduction-to-nomad-cluster-operation.adoc
+32-25Lines changed: 32 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ image::guides:ROOT:nomad-diagram-v2.png[Diagram of the Nomad cluster]
15
15
16
16
- **Nomad client:** Nomad clients execute the jobs they are allocated by Nomad servers. Usually a Nomad client runs on a dedicated machine (often a VM) to take full advantage of machine power. You can have multiple Nomad clients to form a cluster and the Nomad server allocates jobs to the cluster with its scheduling algorithm.
17
17
18
-
- **Nomad jobs:** A Nomad job is a specification, provided by a user, that declares a workload for Nomad. A Nomad job corresponds to an execution of a CircleCI job. If the job uses xref:guides:optimize:parallelism-faster-jobs.adoc#[parallelism], for example `parallelism: 10`, then Nomad runs 10 jobs.
18
+
- **Nomad jobs:** A Nomad job is a specification, provided by a user, that declares a workload for Nomad. A Nomad job corresponds to an execution of a CircleCI job. If the job uses parallelism, for example `parallelism: 10`, then Nomad runs 10 jobs. For more information on parallelism, see the xref:guides:optimize:parallelism-faster-jobs.adoc#[Test Splitting and Parallelism] guide.
19
19
20
20
- **Build agent:** Build agent is a Go program written by CircleCI that executes steps in a job and reports the results. Build agent is executed as the main process inside a Nomad job.
21
21
@@ -31,9 +31,9 @@ The `nomad` CLI is installed in the Nomad pod. It is preconfigured to talk to th
31
31
32
32
The get a list of statuses for all jobs in your cluster, run the following command:
33
33
34
-
[source,shell]
34
+
[source,console]
35
35
----
36
-
kubectl exec -it <nomad-server-pod-ID> -- nomad status
36
+
$ kubectl exec -it <nomad-server-pod-ID> -- nomad status
37
37
----
38
38
39
39
The `Status` is the most important field in the output, with the following status type definitions:
@@ -49,18 +49,18 @@ The `Status` is the most important field in the output, with the following statu
49
49
50
50
To get a list of your Nomad clients, run the following command:
NOTE: `nomad node-status` reports both Nomad clients that are currently serving (status `active`) and Nomad clients that were taken out of the cluster (status `down`). Therefore, you need to count the number of `active` Nomad clients to know the current capacity of your cluster.
58
58
59
59
To get more information about a specific client, run the following command from that client:
This gives information such as how many jobs are running on the client and the resource utilization of the client.
@@ -70,9 +70,9 @@ This gives information such as how many jobs are running on the client and the r
70
70
71
71
A Nomad job corresponds to an execution of a CircleCI job. Therefore, Nomad job logs can sometimes help to understand the status of a CircleCI job if there is a problem. To get logs for a specific job, run the following command:
NOTE: Be sure to specify the `-stderr` flag, as this is where most Build Agent logs appear.
@@ -83,30 +83,37 @@ Complete the following steps to get logs from the allocation of the specified jo
83
83
84
84
. Get the job ID with `nomad status` command.
85
85
. Get the allocation ID of the job with `nomad status <job-id>` command.
86
-
. Get the logs from the allocation with `nomad logs -stderr <allocation-id>`
86
+
. Get the logs from the allocation with `nomad logs -stderr <allocation-id>`.
87
87
88
88
[#accessing-the-nomad-web-ui]
89
89
=== Accessing the Nomad Web UI
90
90
91
91
Nomad provides a web UI for inspecting your Nomad cluster. If you are using an internalized Nomad deployment, which is the default setup with CircleCI server, follow the instructions in this section to temporarily access the UI for troubleshooting purposes. For a more permanent solution, consider externalizing Nomad and consult the official Nomad documentation for setting up routing.
92
92
93
-
. Nomad binds to its Pod IP. To port-forward to the Nomad service, you need to set up a lightweight tunneling mechanism within the cluster as follows:
93
+
. When using Nomad in Kubernetes, it binds to its Pod IP address. Use the command below to fetch the IP address of the Nomad server:
94
94
+
95
-
[source,bash]
95
+
[source,console]
96
96
----
97
-
export NOMAD_IP=$(kubectl get svc nomad-server -o jsonpath='{.spec.clusterIP}' -n <server-namespace>)
97
+
$ export NOMAD_ADDR=$(kubectl get svc nomad-server -o jsonpath='{.spec.clusterIP}' -n <server-namespace>)
98
+
----
99
+
100
+
. If using Externalized Nomad, Use Nomad Network Load Balancer or DNS address to port-forward to access the web UI.
. Navigate to ++`http://localhost:4646/ui`++ in your browser to access the Nomad UI. For more information on utilizing the Nomad UI, refer to the link:https://developer.hashicorp.com/nomad/tutorials/web-ui[Nomad documentation].
116
+
. Navigate to `++http[s]://localhost:4646/ui++` in your browser to access the Nomad UI. For more information on utilizing the Nomad UI, refer to the link:https://developer.hashicorp.com/nomad/tutorials/web-ui[Nomad documentation].
110
117
111
118
[#shutting-down-a-nomad-client]
112
119
=== Shutting down a Nomad client
@@ -115,22 +122,22 @@ When you want to shut down a Nomad client, you must first set the client to `dra
115
122
116
123
. To drain a client, log in to the client and set the client to drain mode with `node-drain` command as follows:
117
124
+
118
-
[source,shell]
125
+
[source,console]
119
126
----
120
-
nomad node-drain -self -enable
127
+
$ nomad node-drain -self -enable
121
128
----
122
129
. Then, make sure the client is in drain mode using the `node-status` command:
123
130
+
124
-
[source,shell]
131
+
[source,console]
125
132
----
126
-
nomad node-status -self
133
+
$ nomad node-status -self
127
134
----
128
135
129
136
Alternatively, you can drain a remote node with the following command, substituting the node ID:
130
137
131
-
[source,shell]
138
+
[source,console]
132
139
----
133
-
nomad node-drain -enable -yes <node-id>
140
+
$ nomad node-drain -enable -yes <node-id>
134
141
----
135
142
136
143
[#scaling-down-the-client-cluster]
@@ -145,7 +152,7 @@ The script should use the commands in the section above to do the following:
145
152
. Terminate the instance.
146
153
147
154
[#externalize-servers]
148
-
== Externalize your Nomad Servers
155
+
== Externalize your Nomad servers
149
156
From server v4.8, Nomad Servers may now be deployed externally to your Kubernetes cluster that hosts your installation of CircleCI server. Externalization of Nomad Servers is optional. Externalization of Nomad Servers can improve their stability. If you already have a CircleCI server instance with _internal_ Nomad Servers, the process to switch to external Nomad Servers is as follows:
150
157
151
158
. Stop all builds on your CircleCI server instance.
@@ -155,5 +162,5 @@ ifndef::pdf[]
155
162
[#next-steps]
156
163
== Next steps
157
164
158
-
* Read the xref:managing-user-accounts.adoc#[Managing user accounts] guide.
165
+
* Read the xref:managing-user-accounts.adoc#[Managing User Accounts] guide.
0 commit comments