Skip to content

Commit 3d22ac3

Browse files
sgnn7sgnn7
authored
Merge pull request #1 from conjurinc/initial_libs
Initialise bash-lib repo
2 parents a620e57 + 1a441a3 commit 3d22ac3

File tree

152 files changed

+9350
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

152 files changed

+9350
-2
lines changed

‎.gitmodules‎

Whitespace-only changes.

‎.gittrees‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Git Subtrees
2+
3+
# The advantage of subtrees is that users don't have to care about them - its
4+
# just a single repo. The disadvantage is that git doesn't track the metadata
5+
# as it does for submodules.
6+
7+
# This file provides an enumeration of the subtrees in this repo, and the URLs
8+
# they came from.
9+
10+
# subtree_path remote_url remote_name
11+
test-utils/bats https://github.com/bats-core/bats bats
12+
test-utils/bats-support https://github.com/ztombol/bats-support bats-support
13+
test-utils/bats-assert-1 https://github.com/jasonkarns/bats-assert-1 bats-assert-1

‎Jenkinsfile‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#!/usr/bin/env groovy
2+
3+
pipeline {
4+
agent { label 'executor-v2' }
5+
6+
options {
7+
timestamps()
8+
buildDiscarder(logRotator(numToKeepStr: '30'))
9+
}
10+
11+
triggers {
12+
cron(getDailyCronString())
13+
}
14+
15+
environment {
16+
BATS_OUTPUT_FORMAT="junit"
17+
}
18+
19+
stages {
20+
21+
stage('BATS Tests') {
22+
steps {
23+
sh './tests-for-this-repo/run-bats-tests'
24+
}
25+
}
26+
27+
stage('Python Linting') {
28+
steps {
29+
sh './tests-for-this-repo/run-python-lint'
30+
}
31+
}
32+
33+
stage('Secrets Leak Check') {
34+
steps {
35+
sh './tests-for-this-repo/run-gitleaks'
36+
}
37+
}
38+
39+
}
40+
41+
post {
42+
always {
43+
junit '*-junit.xml'
44+
cleanupAndNotify(currentBuild.currentResult)
45+
}
46+
}
47+
}

‎README.md‎

Lines changed: 196 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,196 @@
1-
# bashlib
2-
Common bash functions for use in test pipelines
1+
# bash-lib
2+
```
3+
_______________ _______________
4+
.' .' .|
5+
.' .' .' |
6+
.'_______________.'______________ .' |
7+
| ___ _____ ___ || ___ _____ ___ | |
8+
||_=_|__=__|_=_||||_=_|__=__|_=_|| |
9+
______||_____===_____||||_____===_____|| | __________
10+
.' ||_____===_____||||_____===_____|| .' .'|
11+
.' ||_____===_____||||_____===_____|| .' .' |
12+
.'___________|_______________||_______________|.'__________.' |
13+
|.----------.|.-----___-----.||.-----___-----.|| |_____.----------.
14+
|] |||_____________||||_____________||| .' [ |
15+
|| ||.-----___-----.||.-----___-----.||.' | |
16+
|| |||_____________||||_____________|||==========| |
17+
|| ||.-----___-----.||.-----___-----.|| |_____| |
18+
|] o|||_____________||||_____________||| .' [ 'o|
19+
|| ||.-----___-----.||.-----___-----.||.' | |
20+
|| ||| ||||_____________|||==========| |
21+
|| ||| |||.-----___-----.|| |_____| |
22+
|] ||| |||| ||| .' [ |
23+
||__________|||_____________||||_____________|||.'________|__________|
24+
''----------'''------------------------------'''----------''
25+
(o)LGB (o)
26+
```
27+
28+
The place to store functions that are used in pipelines for multiple repos.
29+
30+
Please add whatever is useful to you, but keep it tidy so its still useful to everyone else :)
31+
32+
## Usage
33+
34+
Add bash-lib into your project in the way that best fits your workflow. The only requirement is that you **pin the version of
35+
bash-lib that you use**. This is important so that changes to bash-lib do not have the power to break all projects that use
36+
bash-lib. Your project can then test updates to bash-lib and roll forward periodicly.
37+
38+
Options:
39+
* Add a submodule: they are an easy way to integrate bash-lib and automatically use a single SHA until manually updated. Submodules add a pointer from a mount point in your repo to the external repo (bash-lib), and require workflow changes to ensure that pointer is derferenced during clone, checkout and some other opertaions.
40+
* Add a subtree: This repo uses subtrees to pull in test dependencies. Subtrees copy an external repo into a subdirectory of the host repo, no workflow changes are required. Subtrees naturally keep a single version of bash-lib until explicitly updated. Note that subtree merge commits do not rebase well :warning:, so best to keep subtree updates in separate PRs from normal commits.
41+
* Clone bash-lib in your deployment process, bash-lib doesn't have to be within your repo, just needs to be somewhere where your scripts can source [init](init). This is where it's most important that you implement a mechanism to always use the same SHA, as a **clone will track master by default, which is not an allowed use of bash-lib**.
42+
43+
Once you have bash-lib cloned in your project, you source two things:
44+
45+
1. Source `bash-lib/init`. This ensures submodules are initalised and sets the BASH_LIB_DIR env var to the absolute path to the bash-lib dir. This makes it easy to source libraries from other scripts.
46+
2. Source `${BASH_LIB_DIR}/lib-name/lib` for any libraries you are interested in.
47+
48+
You are now ready to use bash-lib functions :)
49+
50+
## Structure
51+
The `/init` script sets up everything required to use the library, most
52+
importantly the `BASH_LIB_DIR` variable which gives the absolute path to the root
53+
of the library and should be used for sourcing the modules.
54+
55+
The repo is organized into libraries, each library is a directory that has a
56+
lib file. Sourcing the lib for a library should expose all the functions
57+
that library offers. The lib file may source or reference other supporting
58+
files within it's directory.
59+
60+
```
61+
.
62+
├── libname
63+
│ ├── lib
64+
│ └── supporting-file
65+
├── init # init script, source this first
66+
├── run-tests # top level test script, executes all tests
67+
├── secrets.yml # secrets required for executing tests
68+
├── test-utils
69+
│ ├── bats # subtree
70+
│ ├── bats-assert-1 # subtree
71+
│ ├── bats-support # subtree
72+
│ ├── lib
73+
│ └── tap2junit
74+
└── tests-for-this-repo
75+
├── filehandling.bats
76+
├── fixtures #
77+
│ └── libname # Dir containing test fixtures for a library
78+
├── tap2junit
79+
├── libname.bats # contains tests for libname/lib
80+
├── python-lint # supporting files for python lint
81+
├── run-bats-tests # script to run bats tests
82+
├── run-gitleaks # script to check for leaked secrets
83+
└── run-python-lint # script to run python lint
84+
```
85+
## Style Guide
86+
Follow the [google shell style guide](https://google.github.io/styleguide/shell.xml#Naming_Conventions).
87+
TL;DR:
88+
1. Use snake_case function and variable names
89+
1. Use `function` when declaring functions.
90+
91+
92+
## Contents
93+
94+
<!-- html table due to markdown's lack of support for lists within tables -->
95+
<table>
96+
<thead>
97+
<tr>
98+
<th>Library</th>
99+
<th>Description</th>
100+
<th>Functions</th>
101+
</tr>
102+
</thead>
103+
<tbody>
104+
<tr>
105+
<td><a href="filehandling/lib">filehandling</a></td>
106+
<td>Functions relating to file and path handling
107+
<td>
108+
<ol>
109+
<li> <b>abs_path</b>: Ensure a path is absolute</li>
110+
</ol>
111+
</td>
112+
</tr>
113+
<tr>
114+
<td><a href="git/lib">git</a></td>
115+
<td>Git helpers</td>
116+
<td>
117+
<ol>
118+
<li><b>repo_root</b>: Find the root of the current git repo.</li>
119+
<li><b>all_files_in_repo</b>: List files tracked by git.</li>
120+
<li><b>remote_latest_tag</b>: Returns the symbolic name of the latest tag from a remote.</li>
121+
<li><b>remote_latest_tagged_commit</b>: Returns the SHA of the most recently tagged commit in a remote repo (<code>tag^{}</code>).</li>
122+
<li><b>remote_sha_for_ref</b>: Returns the SHA for a given ref from a named remote.</li>
123+
<li><b>remote_tag_for_sha</b>: Returns the tag corresponding to a SHA from a named remote - if there is one.</li>
124+
<li>tracked_files_excluding_subtrees: List files tracked by git, but excluding any files that are in paths listed in <code>.gittrees</code>.</li>
125+
<li><b>cat_gittrees</b>: Returns the contents of .gittrees from the top level of the repo, excluding any comments. Fails if .gittrees is not present.</li>
126+
</ol>
127+
</td>
128+
</tr>
129+
<td><a href="helpers/lib">helpers</a></td>
130+
<td>Bash scripting helpers</td>
131+
<td>
132+
<ol>
133+
<li><b>die</b>: print message and exit 1</li>
134+
<li><b>spushd/spopd</b>: Safe verisons of pushd & popd that call die if the push/pop fails, they also drop stdout. </li>
135+
</ol>
136+
</td>
137+
</tr>
138+
<tr>
139+
<td><a href="k8s/lib">k8s</a></td>
140+
<td>Utils for connecting to K8s</td>
141+
<td>
142+
<ol>
143+
<li><b>build_gke_image</b>: Build docker image for running kubectl commands against GKE.</li>
144+
<li><b>delete_gke_image</b>: Delete image from GKE.</li>
145+
<li><b>run_docker_gke_command</b>: Run command in gke-utils container, already authenticated to k8s cluster.</li>
146+
</ol>
147+
</td>
148+
</tr>
149+
<tr>
150+
<td><a href="logging/lib">logging</a></td>
151+
<td>Helpers related to login</td>
152+
<td>
153+
<ol>
154+
<li><b>announce</b>: Echo message in ascii banner to distinguish it from other log messages.</li>
155+
</ol>
156+
</td>
157+
</tr>
158+
<tr>
159+
<td><a href="test-utils/lib">test-utils</a></td>
160+
<td>Helpers for executing tests</td>
161+
<td>
162+
<ol>
163+
<li><b>shellcheck_script</b>: Execute shellcheck against a script, uses docker.</li>
164+
<li><b>find_scripts</b>: Find git tracked files with extension.</li>
165+
<li><b>tap2junit</b>: Convert a subset of <a href="http://testanything.org/">TAP</a> to JUnit XML. Retains logs for errors.</li>
166+
</ol>
167+
</td>
168+
</tr>
169+
</tbody>
170+
</table>
171+
172+
## Testing
173+
Tests are written using [BATS](https://github.com/bats-core/bats). Each libould have a `lib-name.bats` file in [tests-for-this-repo](/tests-for-this-repo).
174+
Asserts are provided by [bats-assert-1](https://github.com/jasonkarns/bats-assert-1). The value in these is that they provide useful debugging output when the assertion fails, eg expected x got y.
175+
176+
Example:
177+
```bash
178+
# source support and assert libraries
179+
. "${BASH_LIB_DIR}/test-utils/bats-support/load.bash"
180+
. "${BASH_LIB_DIR}/test-utils/bats-assert-1/load.bash"
181+
182+
# source the library under test
183+
. "${BASH_LIB_DIR}/git/lib"
184+
185+
# define a test that calls a library function
186+
@test "it does the thing" {
187+
some_prep_work
188+
# run is a wrapper that catches failures so that assertsions can be run,
189+
# otherwise the test would immediately fail.
190+
run does_the_thing
191+
assert_success
192+
assert_output "thing done"
193+
}
194+
```
195+
196+
Test fixtures should go in /tests-for-this-repo/[fixtures](tests-for-this-repo/fixtures)/lib-name.

‎filehandling/lib‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/bin/bash
2+
3+
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
4+
. "${BASH_LIB_DIR}/helpers/lib"
5+
6+
#https://stackoverflow.com/a/23002317
7+
function abs_path() {
8+
# generate absolute path from relative path
9+
# $1 : relative filename
10+
# return : absolute path
11+
if [ -d "$1" ]; then
12+
# dir
13+
(spushd "$1"; pwd)
14+
elif [ -f "$1" ]; then
15+
# file
16+
if [[ $1 = /* ]]; then
17+
echo "$1"
18+
elif [[ $1 == */* ]]; then
19+
echo "$(spushd "${1%/*}"; pwd)/${1##*/}"
20+
else
21+
echo "$(pwd)/$1"
22+
fi
23+
fi
24+
}

‎git/lib‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
#!/bin/bash
2+
3+
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
4+
. "${BASH_LIB_DIR}/helpers/lib"
5+
6+
# Get the top level of a git repo
7+
function repo_root(){
8+
git rev-parse --show-toplevel
9+
}
10+
11+
# List files tracked by git
12+
function all_files_in_repo(){
13+
git ls-tree -r HEAD --name-only
14+
}
15+
16+
# Find the latest tag available at a repo url
17+
# Returns tag name, not sha
18+
function remote_latest_tag(){
19+
local -r remote_url="${1}"
20+
# In ls-remote the ^{} suffix refers to a peeled/dereferenced object.
21+
# eg refs/tags/v0.0.1^{} shows the SHA of the commit that was tagged,
22+
# not the SHA of the tag itself.
23+
# Adding --refs hides peeled tags
24+
git ls-remote --tags --refs --quiet \
25+
"${remote_url}" \
26+
| tail -n 1 \
27+
| cut -f 2 \
28+
| sed -e 's+refs/tags/++'
29+
}
30+
31+
# Find the SHA of the latests commit to be tagged in a remote repo
32+
function remote_latest_tagged_commit(){
33+
local -r remote="${1}"
34+
local -r tag="$(remote_latest_tag "${remote}")"
35+
git ls-remote "${remote}" | awk "/refs\/tags\/${tag}\^/{print \$1}"
36+
}
37+
38+
function remote_sha_for_ref(){
39+
local -r remote="${1}"
40+
local -r ref="${2}"
41+
42+
# First try adding ^{} to the ref, incase it's a tag
43+
# and needs peeling. If nothing is found for that,
44+
# try without.
45+
peeled_ref=$(
46+
git ls-remote "${remote}" \
47+
| awk "/${ref}[^$]/{print \$1}"
48+
)
49+
50+
if [[ -n "${peeled_ref}" ]]; then
51+
echo "${peeled_ref}"
52+
else
53+
git ls-remote "${remote}" \
54+
| awk "/${ref}/{print \$1}"
55+
fi
56+
}
57+
58+
function remote_tag_for_sha(){
59+
local -r remote="${1}"
60+
local -r sha="${2}"
61+
git ls-remote "${remote}" \
62+
| awk -F'/' "/${sha}.*tag/{ gsub(/\^\{\}\$/, \"\"); print \$3 }"
63+
}
64+
65+
66+
## Minimal git subtree functionality required for tests to pass
67+
# full subtree functionality is not ready for merge.
68+
function cat_gittrees(){
69+
local -r git_trees="$(repo_root)/.gittrees"
70+
local -r subtrees_file_format=".gittrees should contain one subtree per line,\
71+
space seperated with three fields: subtree_path renmote_url remote_name"
72+
[[ -e "${git_trees}" ]] || die ".gittrees file ${git_trees} not found. ${subtrees_file_format}"
73+
grep -E -v '^\s*$|^\s*#' "$(repo_root)/.gittrees"
74+
}
75+
76+
function tracked_files_excluding_subtrees(){
77+
subtrees="$(cat_gittrees | awk '{print $1}' | paste -sd '|' -)"
78+
all_files_in_repo | grep -E -v "${subtrees}"
79+
}

‎helpers/lib‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/bash
2+
3+
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
4+
5+
function die(){
6+
echo "${@}"
7+
exit 1
8+
}
9+
10+
#safe pushd
11+
function spushd(){
12+
if ! pushd "${1}" >/dev/null; then
13+
die "pushd ${1} failed :("
14+
fi
15+
}
16+
17+
#safe popd
18+
function spopd(){
19+
popd >/dev/null || die "popd failed :("
20+
}

0 commit comments

Comments
 (0)