You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: code-engine-cos2cos/README.md
+6-77Lines changed: 6 additions & 77 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,9 +34,9 @@ Before you begin, ensure you have the following:
34
34
1. ce
35
35
2. cos
36
36
3. iam
37
-
-**Podman** or **Docker**(Refer, [Build Steps](#build-steps)): For building the container.
37
+
38
38
-**IBM Cloud Object Storage (COS)**(Not required when using config.sh script): Ensure you have created two buckets (primary and secondary). Refer, [Getting started with IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage).
39
-
-**IBM Code Engine**(Not required when using config.sh script): Set up and create a project on IBM Code Engine. Refer, [Getting started with IBM Cloud Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-getting-started).
39
+
-**IBM Code Engine Project**(Not required when using config.sh script): Set up and create a project on IBM Code Engine. Refer, [Getting started with IBM Cloud Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-getting-started).
40
40
-**Go**: For modifying and testing the source code.
41
41
42
42
### Setup & Configuration
@@ -56,11 +56,11 @@ Before you begin, ensure you have the following:
56
56
```
57
57
4.**Update [data.sh](/data.sh)**:
58
58
59
-
Please update the variable values in the file as per your requirement. The details for creating the cos2cos job with config.sh and building image with build.sh is fetched from this file.
59
+
Please update the variable values in the file as per your requirement. The details for creating the cos2cos job with config.sh is fetched from this file.
60
60
61
-
Note:- The variable CRTokenFilePath should remain same as mentioned in the data.sh file.
62
61
5.**Build the Image from the source code**:
63
-
Refer [Build Steps](#build-steps)
62
+
63
+
Image build is done automatically in the config.sh file from source.
64
64
6.**Setup the required IBM Cloud resources**:
65
65
66
66
To automatically set up the required IBM Cloud resources, including COS buckets and secrets, simply run the provided `config.sh` script:
@@ -77,36 +77,17 @@ Before you begin, ensure you have the following:
77
77
Note:
78
78
- The script will either take the existing project with the project name provided in data.sh or create a new project with the same name.
79
79
- The script will create a trusted profile if it does not exists.
80
-
- The script will work only with authentication mechanism of Trusted Profile. In-case the user wants to use service-credentials then he/she needs to manually update the AUTH_SECRET and create the API_KEY for both COS Instance as well as add second command-line argument as "false". The first argument can be set to "true" or "false". See below for more.
81
-
- The two command line arguments are:
82
-
1. The first argument "isInCodeEngine", a boolean value, is set to true by-default (meaning the job is running in IBM Cloud Code-Engine). It can be set to "false" when the user wants to run it locally.
83
-
2. The second argument "isUsingTrustedProfile", a boolean value, is set to true by-default (meaning authentication mechanism for COS bucket is TrustedProfile). It can be set to "false" when the user wants to use service-credentials. Please make sure that the AUTH_SECRET is also updated to have the following (in-case using service-credentials):
84
-
```bash
85
-
IBM_COS_API_KEY_PRIMARY=<COS-API-KEY-Primary>
86
-
IBM_COS_API_KEY_SECONDARY=<COS-API-KEY-Secondary>
87
-
```
88
-
- IMP: TrustedProfile Authentication method won't work when running the program locally.
89
80
7.**Run the Program**:
90
81
Once everything is configured, run it
91
-
92
-
- Locally using
93
-
```bash
94
-
go run .
95
-
```
96
-
OR
97
82
- On IBM Cloud Code-Engine using CLI:
98
83
```bash
99
84
ibmcloud ce jobrun submit --job ${JOB_NAME} --name ${JOB_NAME}
100
85
```
101
86
102
-
103
87
This will:
104
88
- Trigger the job in**IBM Code Engine** to process the files in the primary bucket.
105
89
- Upload the processed files to the secondary bucket.
106
90
107
-
Note:
108
-
- If you are running the program locally, pass 2 command-line arguments "false" "false" while running main.go
109
-
- Also make sure that env file is configured accordingly.
110
91
8. **Check the Logs**:
111
92
After the job completes, check the IBM Cloud UI or the logs to confirm the processing status.
112
93
@@ -116,9 +97,6 @@ The system supports parallel execution using IBM Code Engine Job arrays. This en
116
97
A sample version of the parallel job setup is included in the repository:
To run the project locally, you need to create a `.env` file in the root directory. You can refer to the [`env_sample`](/env_sample) file for the required environment variables and their format.
122
100
123
101
## Testing
124
102
@@ -129,61 +107,12 @@ To run the tests:
129
107
go test -v
130
108
```
131
109
132
-
## Build Steps
133
-
134
-
You can build and push the container image using one of the following methods.
135
-
136
-
**Note**: If you are using the [build.sh](/build.sh), by-default the [Method-1](#1-build-using-source-code-local-source) is used.
137
-
138
-
#### 1. Build Using Source Code (Local Source)
139
-
140
-
To build the image from local source code using IBM Cloud Code Engine:
141
-
142
-
```bash
143
-
ibmcloud ce build create --name ${BUILD_NAME} --build-type local --image ${REGISTRY}/${NAMESPACE}/${IMAGE_NAME}
144
-
ibmcloud ce buildrun submit --build ${BUILD_NAME} --name ${BUILD_NAME}-build-run
145
-
```
146
-
147
-
#### 2. Build Using Git-based Source
148
-
149
-
To build the image using a Git repository:
150
-
151
-
1. Create a deploy key or user-access key in your GitHub repository.
152
-
2. Add the private key by creating an SSH secret in IBM Cloud Code Engine.
153
-
3. Create a build using the Git repository:
154
-
155
-
```bash
156
-
ibmcloud ce build create \
157
-
--name ${BUILD_NAME} \
158
-
--image ${REGISTRY}/${NAMESPACE}/${IMAGE_NAME} \
159
-
--source ${GIT_SSH_URL} \
160
-
--context-dir / \
161
-
--strategy dockerfile \
162
-
--git-repo-secret ${GIT_SSH_SECRET}
163
-
```
164
-
165
-
4. Submit the build:
166
-
167
-
```bash
168
-
ibmcloud ce buildrun submit --build ${BUILD_NAME}
169
-
```
170
-
171
-
### View Build Logs
172
-
173
-
To view the logs of a build run, use the following command:
174
-
175
-
```bash
176
-
ibmcloud ce buildrun logs -f -n <build-run-name>
177
-
```
178
-
179
-
Replace `<build-run-name>` with the actual name of your build run.
180
-
181
110
## Performance
182
111
183
112
The program is optimized forhandling large files (up to several GBs). For example, when tested with 50 files (each 65MB), the program processed the filesin 70 to 100 seconds, with 13 parallel jobs.
184
113
185
114
## Troubleshooting
186
115
187
116
- **Error: Object Not Found**: Ensure that the primary bucket is correctly configured and contains the objects you expect.
188
-
- **Authentication Failure**: Check that the authentication method (trusted profile or service credentials) is correctly set up.
117
+
- **Authentication Failure**: Check that the authentication method (trusted profile) is correctly set up.
189
118
- **Job Timeout or Failure**: If the job takes longer than expected, check the logs for any performance bottlenecks or errors.
0 commit comments