Skip to content

Added more documentation #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Added more documentation #6

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Sage Storage Loader

This service loads files staged in a directory (on Beehive) into [OSN](https://www.openstoragenetwork.org/) or [Pelican](https://pelicanplatform.org/). Based on env variable **STORAGE_TYPE** it will use OSN or Pelican.
This service loads files staged in a directory (on Beehive) into [OSN](https://www.openstoragenetwork.org/) or [Pelican](https://pelicanplatform.org/). Based on env variable **STORAGE_TYPE** it will use OSN or Pelican. All uploaded files will be in a folder called `node-data` in **STORAGE_TYPE**.

The expected data flow is:

Expand Down Expand Up @@ -64,4 +64,22 @@ You can check the uploads by curling for them in **LOADER_PELICAN_ENDPOINT**. Fo
```sh
#create jwt token first then run:
curl -v -H "Authorization: Bearer $(cat token)" $(LOADER_PELICAN_ENDPOINT)/$(LOADER_PELICAN_BUCKET)/
```
```

# Environment Variable Definitions
- **STORAGE_TYPE**: Type of storage to upload files to (Pelican or OSN).
- **LOADER_NUM_WORKERS**: The number of workers uploading files to STORAGE_TYPE.
- **LOADER_DATA_DIR**: The staging directory the workers will be grabbing files from.

## Pelican Only
- **LOADER_PELICAN_ENDPOINT**: The Pelican endpoint to use.
- **LOADER_PELICAN_BUCKET**: The directory to place `/node-data` in Pelican. *If left blank `/node-data` will be in root.*
- **JWT_PUBLIC_KEY_CONFIG_URL**: The url to retrieve Jwt public key configuration.
- **JWT_ISSUER_KEY_PATH**: The issuer key cert to generate Jwt tokens.
- **JWT_PUBLIC_KEY_ID**: The id of the public key to use.

## OSN Only
- **LOADER_S3_ENDPOINT**: The S3 endpoint to use.
- **LOADER_S3_ACCESS_KEY_ID**: The Access Key ID used for authenticating and authorizing access to the S3 service.
- **LOADER_S3_SECRET_ACCESS_KEY**: The Secret Access Key paired with LOADER_S3_ACCESS_KEY_ID.
- **LOADER_S3_BUCKET**: The S3 bucket to place `/node-data` in.