|
1 | 1 | **Note**: Kubernetes now manages dependencies using go modules. |
2 | | -See [current documentation for working with dependencies](./vendor.md) for master branch development. |
3 | | -This document only applies to Kubernetes 1.14.x and earlier, |
4 | | -and should be removed once Kubernetes 1.14.x is no longer supported. |
5 | | - |
6 | | -# Using godep to manage dependencies |
7 | | - |
8 | | -This document is intended to show a way for managing `vendor/` tree dependencies |
9 | | -in Kubernetes. If you do not need to manage vendored dependencies, you probably |
10 | | -do not need to read this. |
11 | | - |
12 | | -## Background |
13 | | - |
14 | | -As a tool, `godep` leaves much to be desired. It builds on `go get`, and adds |
15 | | -the ability to pin dependencies to exact git version. The `go get` tool itself |
16 | | -doesn't have any concept of versions, and tends to blow up if it finds a git |
17 | | -repo synced to anything but `master`, but that is exactly the state that |
18 | | -`godep` leaves repos. This is a recipe for frustration when people try to use |
19 | | -the tools. |
20 | | - |
21 | | -This doc will focus on predictability and reproducibility. |
22 | | - |
23 | | -## Justifications for an update |
24 | | - |
25 | | -Before you update a dependency, take a moment to consider why it should be |
26 | | -updated. Valid reasons include: |
27 | | - 1. We need new functionality that is in a later version. |
28 | | - 2. New or improved APIs in the dependency significantly improve Kubernetes code. |
29 | | - 3. Bugs were fixed that impact Kubernetes. |
30 | | - 4. Security issues were fixed even if they don't impact Kubernetes yet. |
31 | | - 5. Performance, scale, or efficiency was meaningfully improved. |
32 | | - 6. We need dependency A and there is a transitive dependency B. |
33 | | - 7. Kubernetes has an older level of a dependency that is precluding being able |
34 | | -to work with other projects in the ecosystem. |
35 | | - |
36 | | -## Theory of operation |
37 | | - |
38 | | -The `go` toolchain assumes a global workspace that hosts all of your Go code. |
39 | | - |
40 | | -The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`. |
41 | | -This reads the `Godeps.json` file, downloads all of the dependencies from the |
42 | | -internet, and syncs them to the specified revisions. You can then make |
43 | | -changes - sync to different revisions or edit Kubernetes code to use new |
44 | | -dependencies (and satisfy them with `go get`). When ready, you tell `godep` to |
45 | | -"save" everything, which it does by walking the Kubernetes code, finding all |
46 | | -required dependencies, copying them from `$GOPATH` into the `vendor/` directory, |
47 | | -and rewriting `Godeps.json`. |
48 | | - |
49 | | -This does not work well, when combined with a global Go workspace. Instead, we |
50 | | -will set up a private workspace for this process. |
51 | | - |
52 | | -The Kubernetes build process uses this same technique, and offers a tool called |
53 | | -`run-in-gopath.sh` which sets up and switches to a local, private workspace, |
54 | | -including setting up `$GOPATH` and `$PATH`. If you wrap commands with this |
55 | | -tool, they will use the private workspace, which will not conflict with other |
56 | | -projects and is easily cleaned up and recreated. |
57 | | - |
58 | | -To see this in action, you can run an interactive shell in this environment: |
59 | | - |
60 | | -```sh |
61 | | -# Run a shell, but don't run your own shell initializations. |
62 | | -hack/run-in-gopath.sh bash --norc --noprofile |
63 | | -``` |
64 | | - |
65 | | -## Restoring deps |
66 | | - |
67 | | -To extract and download dependencies into `$GOPATH` we provide a script: |
68 | | -`hack/godep-restore.sh`. If you run this tool, it will restore into your own |
69 | | -`$GOPATH`. If you wrap it in `run-in-gopath.sh` it will restore into your |
70 | | -`_output/` directory. |
71 | | - |
72 | | -```sh |
73 | | -hack/run-in-gopath.sh hack/godep-restore.sh |
74 | | -``` |
75 | | - |
76 | | -This script will try to optimize what it needs to download, and if it seems the |
77 | | -dependencies are all present already, it will return very quickly. |
78 | | - |
79 | | -If there's ever any doubt about the correctness of your dependencies, you can |
80 | | -simply `make clean` or `rm -rf _output`, and run it again. |
81 | | - |
82 | | -Now you should have a clean copy of all of the Kubernetes dependencies. |
83 | | - |
84 | | -Downloading dependencies might take a while, so if you want to see progress |
85 | | -information use the `-v` flag: |
86 | | - |
87 | | -```sh |
88 | | -hack/run-in-gopath.sh hack/godep-restore.sh -v |
89 | | -``` |
90 | | - |
91 | | -## Making changes |
92 | | - |
93 | | -The most common things people need to do with deps are add and update them. |
94 | | -These are similar but different. |
95 | | - |
96 | | -### Adding a dep |
97 | | - |
98 | | -For the sake of examples, consider that we have discovered a wonderful Go |
99 | | -library at `example.com/go/frob`. The first thing you need to do is get that |
100 | | -code into your workspace: |
101 | | - |
102 | | -```sh |
103 | | -hack/run-in-gopath.sh go get -d example.com/go/frob |
104 | | -``` |
105 | | - |
106 | | -This will fetch, but not compile (omit the `-d` if you want to compile it now), |
107 | | -the library into your private `$GOPATH`. It will pull whatever the default |
108 | | -revision of that library is, typically the `master` branch for git repositories. |
109 | | -If this is not the revision you need, you can change it, for example to |
110 | | -`v1.0.0`: |
111 | | - |
112 | | -```sh |
113 | | -hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0' |
114 | | -``` |
115 | | - |
116 | | -Now that the code is present, you can start to use it in Kubernetes code. |
117 | | -Because it is in your private workspace's `$GOPATH`, it might not be part of |
118 | | -your own `$GOPATH`, so tools like `goimports` might not find it. This is an |
119 | | -unfortunate side-effect of this process. You can either add the whole private |
120 | | -workspace to your own `$GOPATH` or you can `go get` the library into your own |
121 | | -`$GOPATH` until it is properly vendored into Kubernetes. |
122 | | - |
123 | | -Another possible complication is a dep that uses `gopdep` itself. In that case, |
124 | | -you need to restore its dependencies, too: |
125 | | - |
126 | | -```sh |
127 | | -hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore' |
128 | | -``` |
129 | | - |
130 | | -If the transitive deps collide with Kubernetes deps, you may have to manually |
131 | | -resolve things. This is where the ability to run a shell in this environment |
132 | | -comes in handy: |
133 | | - |
134 | | -```sh |
135 | | -hack/run-in-gopath.sh bash --norc --noprofile |
136 | | -``` |
137 | | - |
138 | | -### Updating a dep |
139 | | - |
140 | | -Sometimes we already have a dep, but the version of it is wrong. Because of the |
141 | | -way that `godep` and `go get` interact (badly) it's generally easiest to hit it |
142 | | -with a big hammer: |
143 | | - |
144 | | -```sh |
145 | | -hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob' |
146 | | -hack/run-in-gopath.sh go get -d example.com/go/frob |
147 | | -hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0' |
148 | | -``` |
149 | | - |
150 | | -This will remove the code, re-fetch it, and sync to your desired version. |
151 | | - |
152 | | -### Removing a dep |
153 | | - |
154 | | -This happens almost for free. If you edit Kubernetes code and remove the last |
155 | | -use of a given dependency, you only need to restore and save the deps, and the |
156 | | -`godep` tool will figure out that you don't need that dep any more: |
157 | | - |
158 | | -## Saving deps |
159 | | - |
160 | | -Now that you have made your changes - adding, updating, or removing the use of a |
161 | | -dep - you need to rebuild the dependency database and make changes to the |
162 | | -`vendor/` directory. |
163 | | - |
164 | | -```sh |
165 | | -hack/run-in-gopath.sh hack/godep-save.sh |
166 | | -``` |
167 | | - |
168 | | -This will run through all of the primary targets for the Kubernetes project, |
169 | | -calculate which deps are needed, and rebuild the database. It will also |
170 | | -regenerate other metadata files which the project needs, such as BUILD files and |
171 | | -the LICENSE database. |
172 | | - |
173 | | -Commit the changes before updating deps in staging repos. |
174 | | - |
175 | | -## Saving deps in staging repos |
176 | | - |
177 | | -Kubernetes stores some code in a directory called `staging` which is handled |
178 | | -specially, and is not covered by the above. If you modified any code under |
179 | | -staging, or if you changed a dependency of code under staging (even |
180 | | -transitively), you'll also need to update deps there: |
181 | | - |
182 | | -```sh |
183 | | -./hack/update-staging-godeps.sh |
184 | | -``` |
185 | | - |
186 | | -Then commit the changes generated by the above script. |
187 | | - |
188 | | -## Commit messages |
189 | | - |
190 | | -Terse messages like "Update foo.org/bar to 0.42" are problematic |
191 | | -for maintainability. Please include in your commit message the |
192 | | -detailed reason why the dependencies were modified. |
193 | | - |
194 | | -Too commonly dependency changes have a ripple effect where something |
195 | | -else breaks unexpectedly. The first instinct during issue triage |
196 | | -is to revert a change. If the change was made to fix some other |
197 | | -issue and that issue was not documented, then a revert simply |
198 | | -continues the ripple by fixing one issue and reintroducing another |
199 | | -which then needs refixed. This can needlessly span multiple days |
200 | | -as CI results bubble in and subsequent patches fix and refix and |
201 | | -rerefix issues. This may be avoided if the original modifications |
202 | | -recorded artifacts of the change rationale. |
203 | | - |
204 | | -## Sanity checking |
205 | | - |
206 | | -After all of this is done, `git status` should show you what files have been |
207 | | -modified and added/removed. Make sure to sanity-check them with `git diff`, and |
208 | | -to `git add` and `git rm` them, as needed. It is commonly advised to make one |
209 | | -`git commit` which includes just the dependencies and Godeps files, and |
210 | | -another `git commit` that includes changes to Kubernetes code to use (or stop |
211 | | -using) the new/updated/removed dependency. These commits can go into a single |
212 | | -pull request. |
213 | | - |
214 | | -Before sending your PR, it's a good idea to sanity check that your |
215 | | -Godeps.json file and the contents of `vendor/ `are ok: |
216 | | - |
217 | | -```sh |
218 | | -hack/run-in-gopath.sh hack/verify-godeps.sh |
219 | | -``` |
220 | | - |
221 | | -All this script will do is a restore, followed by a save, and then look for |
222 | | -changes. If you followed the above instructions, it should be clean. If it is |
223 | | -not, you get to figure out why. |
224 | | - |
225 | | -## Manual updates |
226 | | - |
227 | | -It is sometimes expedient to manually fix the `Godeps.json` file to |
228 | | -minimize the changes. However, without great care this can lead to failures |
229 | | -with the verifier scripts. The kubernetes codebase does "interesting things" |
230 | | -with symlinks between `vendor/` and `staging/` to allow multiple Go import |
231 | | -paths to coexist in the same git repo. |
232 | | - |
233 | | -The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull |
234 | | -request. |
235 | | - |
236 | | -## Reviewing and approving dependency changes |
237 | | - |
238 | | -Particular attention to detail should be exercised when reviewing and approving |
239 | | -PRs that add/remove/update dependencies. Importing a new dependency should bring |
240 | | -a certain degree of value as there is a maintenance overhead for maintaining |
241 | | -dependencies into the future. |
242 | | - |
243 | | -When importing a new dependency, be sure to keep an eye out for the following: |
244 | | -- Is the dependency maintained? |
245 | | -- Does the dependency bring value to the project? Could this be done without |
246 | | - adding a new dependency? |
247 | | -- Is the target dependency the original source, or a fork? |
248 | | -- Is there already a dependency in the project that does something similar? |
249 | | -- Does the dependency have a license that is compatible with the Kubernetes |
250 | | - project? |
251 | | - |
252 | | -Additionally: |
253 | | -- Look at the godeps file. Check that the only changes are what the PR claims |
254 | | - them to be. |
255 | | -- Check if there is a tagged release we can vendor instead of a random hash |
256 | | -- Scan the imported code for things like init() functions |
257 | | -- Look at the Kubernetes code changes and make sure they are appropriate |
258 | | - (e.g. renaming imports or similar). You do not need to do feature code review. |
259 | | -- If this is all good, approve, but don't LGTM, unless you also do code review |
260 | | - or unless it is trivial (e.g. moving from k/k/pkg/utils -> k/utils). |
261 | | - |
262 | | -All new dependency licenses should be reviewed by @kubernetes/dep-approvers to ensure that they |
263 | | -are compatible with the Kubernetes project license. It is also important to note |
264 | | -and flag if a license has changed when updating a dependency, so that these can |
265 | | -also be reviewed. |
266 | | - |
267 | | -For reference, whitelisted licenses as per the CNCF Whitelist Policy are |
268 | | -mentioned [here](https://git.k8s.io/sig-release/licensing/README.md#licenses-for-dependencies). |
| 2 | +See [current documentation for working with dependencies](./vendor.md) for master branch development. |
0 commit comments