Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
name: Bug report
about: Create a report to help us improve
title: "[BUG] "
labels: bug
assignees: ''
---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Run '...'
3. See error

**Expected behavior**
A concise description of what you expected to happen.

**Screenshots or logs**
If applicable, add screenshots or paste logs to help explain your problem.

**Environment (please complete the following information):**
- OS: (e.g., Windows 10, Ubuntu 22.04)
- Python version: (e.g., 3.10)

**Additional context**
Add any other context about the problem here.

<!-- Thank you for helping improve this project! -->
21 changes: 21 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
name: Feature request
about: Suggest an idea for this project
title: "[FEATURE] "
labels: enhancement
assignees: ''
---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

<!-- Thanks — maintained by achrafS133 -->
30 changes: 0 additions & 30 deletions .github/ISSUE_TEMPLATE/user-story.md

This file was deleted.

13 changes: 13 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!-- Describe the purpose of this PR and what changed. -->

## Summary

Describe changes at a high level.

## Checklist
- [ ] I have run the test suite locally (`pytest -q`).
- [ ] I updated any relevant documentation.
- [ ] This change is ready for review.

---
_Maintained by achrafS133_
52 changes: 0 additions & 52 deletions .github/workflows/validate-template.yaml

This file was deleted.

4 changes: 0 additions & 4 deletions .gitignore

This file was deleted.

16 changes: 16 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Contributing

Thanks for your interest in contributing to this project. The repo is maintained by `achrafS133`.

Guidelines
- Fork the repo and create a branch for your change.
- Open a pull request describing your changes and link any related issues.
- Run tests locally with `pytest -q` and ensure they pass.

Code style
- Follow the existing project style. Use `ruff` and `black` where available.

Reporting bugs
- Use the issue templates in `.github/ISSUE_TEMPLATE` to report bugs or request features.

Thank you — maintained by `achrafS133`.
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2025 achrafS133

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Prerequisites:

```bash
curl -fsSL https://pixi.sh/install.sh | bash
git clone https://github.com/l-mds/local-data-stack.git
git clone https://github.com/achrafS133/local-data-stack.git
cd local-data-stack
pixi run tpl-init-cruft

Expand Down
58 changes: 58 additions & 0 deletions docs/github-push-and-metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# GitHub: push fixes and metrics (local guidance)

This page explains how to fix push permission errors and how to view common GitHub repo metrics.

## Fixing `403` when pushing (common options)

1) Push to your fork (recommended if you don't have push access to the upstream repo)

```powershell
# create a fork on github.com (web UI), then in your local repo:
git remote add my-fork https://github.com/<your-github-username>/local-data-stack.git
git fetch my-fork
git push -u my-fork cleanup/remove-generated
```

2) Push via SSH to the original repo (requires an SSH key added to GitHub and write permissions)

```powershell
# test SSH auth
ssh -T git@github.com
# push using SSH origin
git push -u origin cleanup/remove-generated
```

3) Use a Personal Access Token (HTTPS) if required

```powershell
# set origin to HTTPS to push with PAT
git remote set-url origin https://github.com/<your-github-username>/local-data-stack.git
git push -u origin cleanup/remove-generated
# When prompted, use your username and PAT (with repo scope)
```

## Useful `gh` commands (install GitHub CLI first)

```powershell
# install (Windows)
winget install --id GitHub.cli -e
gh auth login

# fork the repo and add remote automatically
gh repo fork l-mds/local-data-stack --remote=true

# view actions runs
gh api repos/l-mds/local-data-stack/actions/runs --paginate
```

## GitHub web UI (links in repo)
- Pulse: `Repo -> Insights -> Pulse`
- Contributors: `Repo -> Insights -> Contributors`
- Community: `Repo -> Community`
- Code frequency: `Repo -> Insights -> Code frequency`
- Dependency graph: `Repo -> Insights -> Dependency graph`
- Network: `Repo -> Insights -> Network`
- Actions: `Repo -> Actions` (workflow runs and logs)

## Want me to do this for you?
- I can attempt to add your fork remote and push the `cleanup/remove-generated` branch (I will fail if the fork doesn't exist or you have no auth). Tell me the preferred option: `fork` / `ssh` / `https-with-pat`.
Binary file removed img/featured.png
Binary file not shown.
26 changes: 26 additions & 0 deletions local-to-cloud-data-platform/.cruft.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"template": "https://github.com/l-mds/local-data-stack.git",
"commit": "cda07524f9e0c97a0ab7ccd438073e82116063b6",
"checkout": null,
"context": {
"cookiecutter": {
"project_name": "local-to-cloud-data-platform",
"organization": "achraf",
"project_slug": "local-to-cloud-data-platform",
"project_slug_pixi": "local-to-cloud-data-platform",
"author": "achrafS133",
"author_email": "errahouti.2003.achraf@gmail.com",
"cloudflare_api_key": "<<your-secret-key>>",
"cloudflare_email": "<<your-e-mail>>",
"root_domain": "<<yourdomain.com>>",
"time_zone": "Europe/Vienna",
"state_path": "C:\\Users\\MSI\\Desktop\\local-to-cloud-data-platform\\local-data-stack/local-to-cloud-data-platform-z_state",
"_extensions": [
"jinja2_ospath.extensions.OSPathExtension"
],
"_template": "https://github.com/l-mds/local-data-stack.git",
"_commit": "cda07524f9e0c97a0ab7ccd438073e82116063b6"
}
},
"directory": null
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# rsync -avn . /dev/shm --exclude-from .dockerignore
.pixi/envs
.pixi/solve-group-envs
src/{{ cookiecutter.project_slug }}/code_location_{{ cookiecutter.project_slug }}_dbt/dbt_packages
src/local-to-cloud-data-platform/code_location_local-to-cloud-data-platform_dbt/dbt_packages
.dockerignore
.env
Dockerfile
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -683,7 +683,8 @@ gradle-app.setting
.ionide

# End of https://www.toptal.com/developers/gitignore/api/python,visualstudiocode,intellij+all,pycharm,jupyternotebooks,deno,angular,react,reactnative
src/{{ cookiecutter.project_slug }}/code_location_{{ cookiecutter.project_slug }}_dbt/config/.user.yml
src/{{ cookiecutter.project_slug }}/code_location_{{ cookiecutter.project_slug }}_dbt/analytics_database_dev.duckdb
src/{{ cookiecutter.project_slug }}/code_location_{{ cookiecutter.project_slug }}_dbt/analytics_database_prod.duckdb
.dagster_home/
src/code_location_local_to_cloud_data_platform/code_location_local_to_cloud_data_platform_dbt/.user.yml
src/code_location_local_to_cloud_data_platform/code_location_local_to_cloud_data_platform_dbt/analytics_database_dev.duckdb
src/code_location_local_to_cloud_data_platform/code_location_local_to_cloud_data_platform_dbt/analytics_database_prod.duckdb
z_state/
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ RUN echo 'exec "$@"' >> /app/entrypoint_dagster-daemon.sh
# sub package?
COPY src src

RUN pixi install -e codelocation-{{ cookiecutter.project_slug_pixi }} --locked
RUN pixi shell-hook -e codelocation-{{ cookiecutter.project_slug_pixi }} -s bash > /shell-hook
RUN echo "#!/bin/bash" > /app/entrypoint_codelocation-{{ cookiecutter.project_slug_pixi }}.sh
RUN cat /shell-hook >> /app/entrypoint_codelocation-{{ cookiecutter.project_slug_pixi }}.sh
RUN echo 'exec "$@"' >> /app/entrypoint_codelocation-{{ cookiecutter.project_slug_pixi }}.sh
RUN pixi install -e codelocation-local-to-cloud-data-platform --locked
RUN pixi shell-hook -e codelocation-local-to-cloud-data-platform -s bash > /shell-hook
RUN echo "#!/bin/bash" > /app/entrypoint_codelocation-local-to-cloud-data-platform.sh
RUN cat /shell-hook >> /app/entrypoint_codelocation-local-to-cloud-data-platform.sh
RUN echo 'exec "$@"' >> /app/entrypoint_codelocation-local-to-cloud-data-platform.sh

RUN pixi install -e codelocation-foo --locked
RUN pixi shell-hook -e codelocation-foo -s bash > /shell-hook
Expand Down Expand Up @@ -76,18 +76,18 @@ COPY --from=build /app/src/code_location_foo /app/src/code_location_foo
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "--package-name", "quickstart_etl"]


FROM dagster_basics AS codelocation-{{ cookiecutter.project_slug_pixi }}
FROM dagster_basics AS codelocation-local-to-cloud-data-platform
#RUN apt-get update && \
# apt-get upgrade -y && \
# export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends osmctools osm2pgrouting wget vim git osm2pgsql && \
# rm -rf /var/lib/apt/lists/*
ENV DO_NOT_TRACK=1
COPY --from=build /app/.pixi/envs/codelocation-{{ cookiecutter.project_slug_pixi }} /app/.pixi/envs/codelocation-{{ cookiecutter.project_slug_pixi }}
COPY --from=build --chmod=0755 /app/entrypoint_codelocation-{{ cookiecutter.project_slug_pixi }}.sh /app/entrypoint.sh
COPY --from=build /app/src/code_location_{{ cookiecutter.project_slug }} /app/src/code_location_{{ cookiecutter.project_slug }}
COPY --from=build /app/.pixi/envs/codelocation-local-to-cloud-data-platform /app/.pixi/envs/codelocation-local-to-cloud-data-platform
COPY --from=build --chmod=0755 /app/entrypoint_codelocation-local-to-cloud-data-platform.sh /app/entrypoint.sh
COPY --from=build /app/src/code_location_local-to-cloud-data-platform /app/src/code_location_local-to-cloud-data-platform
COPY --from=build /app/src/shared_library /app/src/shared_library
RUN cd src/code_location_{{ cookiecutter.project_slug }}/code_location_{{ cookiecutter.project_slug }}_dbt && /app/.pixi/envs/codelocation-{{ cookiecutter.project_slug_pixi }}/bin/dbt deps && /app/.pixi/envs/codelocation-{{ cookiecutter.project_slug_pixi }}/bin/dbt parse
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "--package-name", "code_location_{{ cookiecutter.project_slug }}"]
RUN cd src/code_location_local-to-cloud-data-platform/code_location_local-to-cloud-data-platform_dbt && /app/.pixi/envs/codelocation-local-to-cloud-data-platform/bin/dbt deps && /app/.pixi/envs/codelocation-local-to-cloud-data-platform/bin/dbt parse
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "--package-name", "code_location_local-to-cloud-data-platform"]


FROM builder_base AS all_codelocations_ci
Expand All @@ -114,7 +114,7 @@ RUN cat /shell-hook >> /app/entrypoint.sh
RUN echo 'exec "$@"' >> /app/entrypoint.sh
RUN chmod 0755 /app/entrypoint.sh

RUN cd src/{{ cookiecutter.project_slug_pixi }}/code_location_{{ cookiecutter.project_slug_pixi }}_dbt && /app/.pixi/envs/ci-validation/bin/dbt deps && /app/.pixi/envs/ci-validation/bin/dbt parse
RUN cd src/local-to-cloud-data-platform/code_location_local-to-cloud-data-platform_dbt && /app/.pixi/envs/ci-validation/bin/dbt deps && /app/.pixi/envs/ci-validation/bin/dbt parse

# docker buildx build --target build --platform=linux/amd64 -t build:x86 -f Dockerfile .
# docker buildx build --target build -t build:arm -f Dockerfile .
Expand All @@ -126,8 +126,8 @@ RUN cd src/{{ cookiecutter.project_slug_pixi }}/code_location_{{ cookiecutter.pr
# docker run --rm -ti foo:server
# docker buildx build --target dagster-daemon --platform=linux/amd64 -t foo:daemon -f Dockerfile .
# docker run --rm -ti foo:daemon
# docker buildx build --target codelocation-{{ cookiecutter.project_slug }} --platform=linux/amd64 -t foo:codelocation-{{ cookiecutter.project_slug }} -f Dockerfile .
# docker buildx build --target codelocation-{{ cookiecutter.project_slug_pixi }} --platform=linux/amd64 -t bar:latest -f Dockerfile .
# docker buildx build --target codelocation-local-to-cloud-data-platform --platform=linux/amd64 -t foo:codelocation-local-to-cloud-data-platform -f Dockerfile .
# docker buildx build --target codelocation-local-to-cloud-data-platform --platform=linux/amd64 -t bar:latest -f Dockerfile .
# docker buildx build --target codelocation-foo --platform=linux/amd64 -t foo:latest -f Dockerfile .
# docker buildx build --target codelocation-foo -t foo:latest -f Dockerfile .

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ pixi update

# launch a shell
pixi shell --frozen -e dev
cd src/code_location_{{ cookiecutter.project_slug }}/code_location_{{ cookiecutter.project_slug }}_dbt/ && dbt deps
cd src/code_location_local-to-cloud-data-platform/code_location_local-to-cloud-data-platform_dbt/ && dbt deps
cd ../../../
```

Expand Down Expand Up @@ -66,7 +66,7 @@ DAGSTER_IS_DEV_CLI=True

# only set in the container (should be done from CI pipeline)
#DAGSTER_CLOUD_DEPLOYMENT_NAME=techexploration
#DAGSTER_CLOUD_GIT_URL=https://github.com/{{ cookiecutter.organization }}/{{ cookiecutter.project_slug }}
#DAGSTER_CLOUD_GIT_URL=https://github.com/achraf/local-to-cloud-data-platform
#DAGSTER_CLOUD_GIT_SHA=<<your sha>>
#DAGSTER_CLOUD_GIT_BRANCH=main
```
Expand Down
Loading