Skip to content

Commit 3b29c78

Browse files
ngmislngmisl
committed
update readme and gh action
1 parent 5b87c8f commit 3b29c78

File tree

5 files changed

+305
-283
lines changed

5 files changed

+305
-283
lines changed

‎.github/workflows/update-llms.yml‎

Lines changed: 25 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -2,79 +2,53 @@ name: Update llms.txt
22

33
on:
44
push:
5-
branches: [ main, master ]
5+
branches: [main, master]
66
pull_request:
7-
branches: [ main, master ]
8-
workflow_dispatch: # Allow manual triggering
9-
repository_dispatch: # Allow external triggers
10-
types: [update-llms]
7+
branches: [main, master]
8+
workflow_dispatch: # Allow manual triggering
119

1210
permissions:
1311
contents: write
14-
pull-requests: write
1512

1613
jobs:
1714
update-llms:
1815
runs-on: ubuntu-latest
1916
steps:
20-
- uses: actions/checkout@v4
21-
with:
22-
fetch-depth: 0
17+
- name: Checkout repository
18+
uses: actions/checkout@v4
2319

2420
- name: Set up Python
2521
uses: actions/setup-python@v4
2622
with:
27-
python-version: '3.12'
28-
cache: 'pip'
29-
30-
- name: Install uv
31-
run: |
32-
curl -LsSf https://astral.sh/uv/install.sh | sh
33-
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
23+
python-version: "3.12"
24+
cache: "pip"
3425

35-
- name: Create venv and install dependencies
26+
- name: Install llmstxt tool
3627
run: |
37-
uv venv
28+
python -m venv .venv
3829
. .venv/bin/activate
39-
if [ "${{ github.event_name }}" = "repository_dispatch" ]; then
40-
uv pip install git+https://github.com/ngmisl/llmstxt.git
41-
else
42-
uv pip install -e .
43-
fi
30+
python -m pip install --upgrade pip
31+
pip install git+https://github.com/ngmisl/llmstxt.git
4432
45-
- name: Clone target repository
46-
if: github.event_name == 'repository_dispatch'
33+
- name: Generate llms.txt
4734
run: |
48-
git clone ${{ github.event.client_payload.repository }} target_repo
49-
cd target_repo
35+
. .venv/bin/activate
36+
# Remove existing llms.txt if it exists
37+
rm -f llms.txt
38+
# Generate new llms.txt
5039
python -c "from llmstxt import generate_llms_txt; generate_llms_txt()"
5140
52-
- name: Generate llms.txt for current repository
53-
if: github.event_name != 'repository_dispatch'
41+
- name: Configure Git
5442
run: |
55-
. .venv/bin/activate
56-
python -c "from llmstxt import generate_llms_txt; generate_llms_txt()"
43+
git config --local user.email "github-actions[bot]@users.noreply.github.com"
44+
git config --local user.name "github-actions[bot]"
5745
58-
- name: Check for changes
59-
id: changes
46+
- name: Commit and push changes
6047
run: |
61-
if git diff --quiet llms.txt; then
62-
echo "No changes to llms.txt"
63-
echo "changes=false" >> $GITHUB_OUTPUT
48+
git add llms.txt
49+
if git diff --staged --quiet; then
50+
echo "No changes to commit"
6451
else
65-
echo "Changes detected in llms.txt"
66-
echo "changes=true" >> $GITHUB_OUTPUT
52+
git commit -m "chore: update llms.txt"
53+
git push
6754
fi
68-
69-
- name: Create Pull Request
70-
if: steps.changes.outputs.changes == 'true'
71-
uses: peter-evans/create-pull-request@v7
72-
with:
73-
commit-message: "chore: update llms.txt"
74-
title: "chore: update llms.txt"
75-
body: |
76-
This PR updates the llms.txt file with the latest code compression.
77-
78-
Changes were automatically generated by [llmstxt](https://github.com/ngmisl/llmstxt).
79-
branch: update-llms
80-
delete-branch: true

‎README.md‎

Lines changed: 85 additions & 148 deletions
Original file line numberDiff line numberDiff line change
@@ -3,181 +3,118 @@
33

44
# llmstxt
55

6-
A Python tool for compressing and organizing code files into a single, LLM-friendly text file. This tool is designed to help prepare codebases for analysis by Large Language Models by removing unnecessary content while preserving important semantic information.
6+
A Python tool to compress code files into a single, LLM-friendly text file.
77

88
## Features
99

10-
- **Smart Code Compression**
11-
- Preserves docstrings and important comments
12-
- Removes redundant whitespace and formatting
13-
- Maintains code structure and readability
14-
- Handles multiple programming languages
15-
16-
- **Language Support**
17-
- Python (with AST-based compression)
18-
- JavaScript
19-
- Java
20-
- C/C++
21-
- Shell scripts
22-
- HTML/CSS
23-
- Configuration files (JSON, YAML, TOML, INI)
24-
- Markdown
25-
26-
- **LLM-Friendly Output**
27-
- XML-style semantic markers
28-
- File metadata and type information
29-
- Organized imports section
30-
- Clear file boundaries
31-
- Consistent formatting
10+
- Preserves important comments and docstrings
11+
- Removes unnecessary content
12+
- Structured, LLM-friendly output
13+
- GitHub Actions integration for automatic updates
3214

3315
## Installation
3416

35-
This project uses [uv](https://github.com/astral-sh/uv) for dependency management.
36-
3717
```bash
38-
# Install uv if you haven't already
39-
curl -LsSf https://astral.sh/uv/install.sh | sh
40-
41-
# Install the package and its dependencies
42-
uv pip install .
43-
44-
# For development
45-
uv pip install -e ".[dev]"
18+
pip install git+https://github.com/ngmisl/llmstxt.git
4619
```
4720

4821
## Usage
4922

50-
### Local Usage
23+
### Command Line
5124

5225
```bash
53-
# Generate llms.txt from current directory
54-
python llms.py
55-
```
26+
# Generate llms.txt in current directory
27+
python -m llmstxt
5628

57-
The script will:
58-
1. Scan the current directory recursively
59-
2. Process files according to .gitignore rules
60-
3. Generate `llms.txt` with compressed content
29+
# Or import and use in your code
30+
from llmstxt import generate_llms_txt
31+
generate_llms_txt()
32+
```
6133

6234
### GitHub Actions Integration
6335

64-
There are two ways to use this tool with GitHub Actions:
65-
66-
1. **For Your Own Repository**
67-
```bash
68-
# Create a workflow file
69-
mkdir -p .github/workflows
70-
curl -o .github/workflows/update-llms.yml https://raw.githubusercontent.com/ngmisl/llmstxt/main/.github/workflows/update-llms.yml
71-
```
72-
73-
The workflow will:
74-
- Run automatically on pushes to main/master
75-
- Create a PR with updated llms.txt when changes are detected
76-
- Can be manually triggered from the Actions tab
77-
78-
2. **For Remote Repositories**
79-
You can trigger the action for any repository using the GitHub API:
80-
81-
```bash
82-
curl -X POST \
83-
-H "Authorization: token $GITHUB_TOKEN" \
84-
-H "Accept: application/vnd.github.v3+json" \
85-
https://api.github.com/repos/ngmisl/llmstxt/dispatches \
86-
-d '{"event_type": "update-llms", "client_payload": {"repository": "https://github.com/user/repo.git"}}'
87-
```
88-
89-
This will:
90-
- Clone the target repository
91-
- Generate llms.txt
92-
- Create a PR with the changes
93-
94-
The workflow uses GitHub's PR system to ensure changes are reviewed before being merged.
95-
96-
## Output Format
97-
98-
The generated `llms.txt` file follows this structure:
99-
100-
```python
101-
# Project: llmstxt
102-
103-
## Project Structure
104-
This file contains the compressed and processed contents of the project.
105-
106-
### File Types
107-
- .py
108-
- .js
109-
- .java
110-
...
111-
112-
<file>src/main.py</file>
113-
<metadata>
114-
path: src/main.py
115-
type: py
116-
size: 1234 bytes
117-
</metadata>
118-
119-
<imports>
120-
import ast
121-
from typing import Optional
122-
</imports>
123-
124-
<code lang='python'>
125-
def example():
126-
"""Docstring preserved."""
127-
return True
128-
</code>
129-
130-
<file>src/utils.js</file>
131-
<metadata>
132-
path: src/utils.js
133-
type: js
134-
size: 567 bytes
135-
</metadata>
136-
137-
<code lang='javascript'>
138-
function helper() {
139-
return true;
140-
}
141-
</code>
36+
To automatically update `llms.txt` in your repository:
37+
38+
1. Create `.github/workflows/update-llms.yml` with:
39+
40+
```yaml
41+
name: Update llms.txt
42+
43+
on:
44+
push:
45+
branches: [main, master]
46+
pull_request:
47+
branches: [main, master]
48+
workflow_dispatch: # Allow manual triggering
49+
50+
permissions:
51+
contents: write
52+
53+
jobs:
54+
update-llms:
55+
runs-on: ubuntu-latest
56+
steps:
57+
- name: Checkout repository
58+
uses: actions/checkout@v4
59+
60+
- name: Set up Python
61+
uses: actions/setup-python@v4
62+
with:
63+
python-version: "3.12"
64+
cache: "pip"
65+
66+
- name: Install llmstxt tool
67+
run: |
68+
python -m venv .venv
69+
. .venv/bin/activate
70+
python -m pip install --upgrade pip
71+
pip install git+https://github.com/ngmisl/llmstxt.git
72+
73+
- name: Generate llms.txt
74+
run: |
75+
. .venv/bin/activate
76+
rm -f llms.txt
77+
python -c "from llmstxt import generate_llms_txt; generate_llms_txt()"
78+
79+
- name: Configure Git
80+
run: |
81+
git config --local user.email "github-actions[bot]@users.noreply.github.com"
82+
git config --local user.name "github-actions[bot]"
83+
84+
- name: Commit and push changes
85+
run: |
86+
git add llms.txt
87+
if git diff --staged --quiet; then
88+
echo "No changes to commit"
89+
else
90+
git commit -m "chore: update llms.txt"
91+
git push
92+
fi
14293
```
14394
144-
## Configuration
145-
146-
The tool can be configured through function parameters:
147-
148-
```python
149-
generate_llms_txt(
150-
output_file="llms.txt", # Output filename
151-
max_file_size=100 * 1024, # Max file size (100KB)
152-
allowed_extensions=( # Supported file types
153-
".py", ".js", ".java",
154-
".c", ".cpp", ".h", ".hpp",
155-
".sh", ".txt", ".md",
156-
".json", ".xml", ".yaml",
157-
".yml", ".toml", ".ini"
158-
)
159-
)
160-
```
95+
The workflow will:
96+
- Run on push to main/master
97+
- Run on pull requests
98+
- Can be triggered manually
99+
- Generate and commit `llms.txt` automatically
161100

162101
## Development
163102

164-
Requirements:
165-
166-
- Python 3.8+
167-
- [uv](https://github.com/astral-sh/uv) for dependency management
168-
169103
```bash
170-
# Install dev dependencies
171-
uv pip install -e ".[dev]"
104+
# Clone the repository
105+
git clone https://github.com/ngmisl/llmstxt.git
106+
cd llmstxt
107+
108+
# Install development dependencies
109+
pip install -e ".[dev]"
172110
173111
# Run type checking
174-
mypy llms.py
112+
mypy llmstxt
175113
176-
# Run linting and formatting
177-
ruff check .
178-
ruff format .
114+
# Run linting
115+
ruff check llmstxt
179116
```
180117

181118
## License
182119

183-
MIT License - See LICENSE file for details
120+
MIT

0 commit comments

Comments
 (0)