Skip to content

Commit 0e7a2df

Browse files
authored
feat: add .ts_ignore pattern ignoring system (#897)
* feat: add `.ts_ignore` pattern ignoring system * fix: add wcmatch dependency * search: add ".TemporaryItems" to GLOBAL_IGNORE * add `desktop.ini` and `.localized` to global ignore * add ".fhdx" and ".ts" filetypes * chore: remove logging statement * chore: format with ruff * feat: use ripgrep for scanning if available * docs: add ignore.md * search: remove ts_ignore filtering on queries * feat: detect if files are added but ignored * fix: render edges on all unlinked thumbs * perf: don't search for cached unlinked thumbs * fix(ui): ensure newlines in file stats * fix: use ignore_to_glob for wcmatch * fix(tests): remove inconsistent test The test hinged on the timing of refresh_dir()'s yield's rather than actual values * ui: change ignored icon and color
1 parent d00546d commit 0e7a2df

File tree

23 files changed

+911
-87
lines changed

23 files changed

+911
-87
lines changed

docs/install.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,14 @@ Don't forget to rebuild!
211211

212212
## Third-Party Dependencies
213213

214+
<!-- prettier-ignore -->
215+
!!! tip
216+
You can check to see if any of these dependencies are correctly located by launching TagStudio and going to "About TagStudio" in the menu bar.
217+
218+
### FFmpeg/FFprobe
219+
214220
For audio/video thumbnails and playback you'll need [FFmpeg](https://ffmpeg.org/download.html) installed on your system. If you encounter any issues with this, please reference our [FFmpeg Help](./help/ffmpeg.md) guide.
215221

216-
You can check to see if FFmpeg and FFprobe are correctly located by launching TagStudio and going to "About TagStudio" in the menu bar.
222+
### ripgrep
223+
224+
A recommended tool to improve the performance of directory scanning is [`ripgrep`](https://github.com/BurntSushi/ripgrep), a Rust-based directory walker that natively integrates with our [`.ts_ignore`](./utilities/ignore.md) (`.gitignore`-style) pattern matching system for excluding files and directories. Ripgrep is already pre-installed on some Linux distributions and also available from several package managers.

docs/utilities/ignore.md

Lines changed: 321 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,321 @@
1+
---
2+
title: Ignore Files
3+
---
4+
5+
# :material-file-document-remove: Ignore Files & Directories
6+
7+
<!-- prettier-ignore -->
8+
!!! warning "Legacy File Extension Ignoring"
9+
TagStudio versions prior to v9.5.4 use a different, more limited method to exclude or include file extensions from your library and subsequent searches. Opening a pre-exiting library in v9.5.4 or later will non-destructively convert this to the newer, more extensive `.ts_ignore` format.
10+
11+
If you're still running an older version of TagStudio in the meantime, you can access the legacy system by going to "Edit -> Manage File Extensions" in the menubar.
12+
13+
TagStudio offers the ability to ignore specific files and directories via a `.ts_ignore` file located inside your [library's](../library/index.md) `.TagStudio` folder. This file is designed to use very similar [glob](<https://en.wikipedia.org/wiki/Glob_(programming)>)-style pattern matching as the [`.gitignore`](https://git-scm.com/docs/gitignore) file used by Git™[^1]. It can be edited within TagStudio or opened to edit with an external program by going to the "Edit -> Ignore Files" option in the menubar.
14+
15+
This file is only referenced when scanning directories for new files to add to your library, and does not apply to files that have already been added to your library.
16+
17+
<!-- prettier-ignore -->
18+
!!! tip
19+
If you just want some specific examples of how to achieve common tasks with the ignore patterns (e.g. ignoring a single file type, ignoring a specific folder) then jump to the "[Use Cases](#use-cases)" section!
20+
21+
<!-- prettier-ignore-start -->
22+
=== "Example .ts_ignore file"
23+
```toml title="My Library/.TagStudio/.ts_ignore"
24+
# TagStudio .ts_ignore file.
25+
26+
# Code
27+
__pycache__
28+
.pytest_cache
29+
.venv
30+
.vs
31+
32+
# Projects
33+
Minecraft/**/Metadata
34+
Minecraft/Website
35+
!Minecraft/Website/*.png
36+
!Minecraft/Website/*.css
37+
38+
# Documents
39+
*.doc
40+
*.docx
41+
*.ppt
42+
*.pptx
43+
*.xls
44+
*.xlsx
45+
```
46+
<!-- prettier-ignore-end -->
47+
48+
## Pattern Format
49+
50+
<!-- prettier-ignore -->
51+
!!! note ""
52+
_This section sourced and adapted from Git's[^1] `.gitignore` [documentation](https://git-scm.com/docs/gitignore)._
53+
54+
### Internal Processes
55+
56+
When scanning your library directories, the `.ts_ignore` file is read by either the [`wcmatch`](https://facelessuser.github.io/wcmatch/glob/) library or [`ripgrep`](https://github.com/BurntSushi/ripgrep) in glob mode depending if you have the later installed on your system and it's detected by TagStudio. Ripgrep is the preferred method for scanning directories due to its improved performance and identical pattern matching to `.gitignore`. This mixture of tools may lead to slight inconsistencies if not using `ripgrep`.
57+
58+
---
59+
60+
### Comments ( `#` )
61+
62+
A `#` symbol at the start of a line indicates that this line is a comment, and match no items. Blank lines are used to enhance readability and also match no items.
63+
64+
- Can be escaped by putting a backslash ("`\`") in front of the `#` symbol.
65+
66+
<!-- prettier-ignore-start -->
67+
=== "Example comment"
68+
```toml
69+
# This is a comment! I can say whatever I want on this line.
70+
file_that_is_being_matched.txt
71+
72+
# file_that_is_NOT_being_matched.png
73+
file_that_is_being_matched.png
74+
```
75+
=== "Organizing with comments"
76+
```toml
77+
# TagStudio .ts_ignore file.
78+
79+
# Minecraft Stuff
80+
Minecraft/**/Metadata
81+
Minecraft/Website
82+
!Minecraft/Website/*.png
83+
!Minecraft/Website/*.css
84+
85+
# Microsoft Office
86+
*.doc
87+
*.docx
88+
*.ppt
89+
*.pptx
90+
*.xls
91+
*.xlsx
92+
```
93+
=== "Escape a # symbol"
94+
```toml
95+
# To ensure a file named '#hashtag.jpg' is ignored:
96+
\#hashtag.jpg
97+
```
98+
<!-- prettier-ignore-end -->
99+
100+
---
101+
102+
### Directories ( `/` )
103+
104+
The forward slash "`/`" is used as the directory separator. Separators may occur at the beginning, middle or end of the `.ts_ignore` search pattern.
105+
106+
- If there is a separator at the beginning or middle (or both) of the pattern, then the pattern is relative to the directory level of the particular `.TagStudio` library folder itself. Otherwise the pattern may also match at any level below the `.TagStudio` folder level.
107+
108+
- If there is a separator at the end of the pattern then the pattern will only match directories, otherwise the pattern can match both files and directories.
109+
110+
<!-- prettier-ignore-start -->
111+
=== "Example folder pattern"
112+
```toml
113+
# Matches "frotz" and "a/frotz" if they are directories.
114+
frotz/
115+
```
116+
=== "Example nested folder pattern"
117+
```toml
118+
# Matches "doc/frotz" but not "a/doc/frotz".
119+
doc/frotz/
120+
```
121+
<!-- prettier-ignore-end -->
122+
123+
---
124+
125+
### Negation ( `!` )
126+
127+
A `!` prefix before a pattern negates the pattern, allowing any files matched matched by previous patterns to be un-matched.
128+
129+
- Any matching file excluded by a previous pattern will become included again.
130+
- **It is not possible to re-include a file if a parent directory of that file is excluded.**
131+
132+
<!-- prettier-ignore-start -->
133+
=== "Example negation"
134+
```toml
135+
# All .jpg files will be ignored, except any located in the 'Photos' folder.
136+
*.jpg
137+
Photos/!*.jpg
138+
```
139+
=== "Escape a ! Symbol"
140+
```toml
141+
# To ensure a file named '!wowee.jpg' is ignored:
142+
\!wowee.jpg
143+
```
144+
<!-- prettier-ignore-end -->
145+
146+
---
147+
148+
### Wildcards
149+
150+
#### Single Asterisks ( `*` )
151+
152+
An asterisk "`*`" matches anything except a slash.
153+
154+
<!-- prettier-ignore-start -->
155+
=== "File examples"
156+
```toml
157+
# Matches all .png files in the "Images" folder.
158+
Images/*.png
159+
160+
# Matches all .png files in all folders
161+
*.png
162+
```
163+
=== "Folder examples"
164+
```toml
165+
# Matches any files or folders directly in "Images/" but not deeper levels.
166+
# Matches file "Images/mario.jpg"
167+
# Matches folder "Images/Mario"
168+
# Does not match file "Images/Mario/cat.jpg"
169+
Images/*
170+
```
171+
<!-- prettier-ignore-end -->
172+
173+
#### Question Marks ( `?` )
174+
175+
The character "`?`" matches any one character except "`/`".
176+
177+
<!-- prettier-ignore-start -->
178+
=== "File examples"
179+
```toml
180+
# Matches any .png file starting with "IMG_" and ending in any four characters.
181+
# Matches "IMG_0001.png"
182+
# Matches "Photos/IMG_1234.png"
183+
# Does not match "IMG_1.png"
184+
IMG_????.png
185+
186+
# Same as above, except matches any file extension instead of only .png
187+
IMG_????.*
188+
```
189+
=== "Folder examples"
190+
```toml
191+
# Matches all files in any direct subfolder of "Photos" beginning in "20".
192+
# Matches "Photos/2000"
193+
# Matches "Photos/2024"
194+
# Matches "Photos/2099"
195+
# Does not match "Photos/1995"
196+
Photos/20??/
197+
```
198+
<!-- prettier-ignore-end -->
199+
200+
#### Double Asterisks ( `**` )
201+
202+
Two consecutive asterisks ("`**`") in patterns matched against full pathname may have special meaning:
203+
204+
- A leading "`**`" followed by a slash means matches in all directories.
205+
- A trailing "`/**`" matches everything inside.
206+
- A slash followed by two consecutive asterisks then a slash ("`/**/`") matches zero or more directories.
207+
- Other consecutive asterisks are considered regular asterisks and will match according to the previous rules.
208+
209+
<!-- prettier-ignore-start -->
210+
=== "Leading **"
211+
```toml
212+
# Both match file or directory "foo" anywhere
213+
**/foo
214+
foo
215+
216+
# Matches file or directory "bar" anywhere that is directly under directory "foo"
217+
**/foo/bar
218+
```
219+
=== "Trailing /**"
220+
```toml
221+
# Matches all files inside directory "abc" with infinite depth.
222+
abc/**
223+
```
224+
=== "Middle /**/"
225+
```toml
226+
# Matches "a/b", "a/x/b", "a/x/y/b" and so on.
227+
a/**/b
228+
```
229+
<!-- prettier-ignore-end -->
230+
231+
#### Square Brackets ( `[a-Z]` )
232+
233+
Character sets and ranges are specific and powerful forms of wildcards that use characters inside of brackets (`[]`) to leverage very specific matching. The range notation, e.g. `[a-zA-Z]`, can be used to match one of the characters in a range.
234+
235+
<!-- prettier-ignore -->
236+
!!! tip
237+
For more in-depth examples and explanations on how to use ranges, please reference the [`glob`](https://man7.org/linux/man-pages/man7/glob.7.html) man page.
238+
239+
<!-- prettier-ignore-start -->
240+
=== "Range examples"
241+
```toml
242+
# Matches all files that start with "IMG_" and end in a single numeric character.
243+
# Matches "IMG_0.jpg", "IMG_7.png"
244+
# Does not match "IMG_10.jpg", "IMG_A.jpg"
245+
IMG_[0-9]
246+
247+
# Matches all files that start with "IMG_" and end in a single alphabetic character
248+
IMG_[a-z]
249+
```
250+
=== "Set examples"
251+
```toml
252+
# Matches all files that start with "IMG_" and in any character in the set.
253+
# Matches "draft_a.docx", "draft_b.docx", "draft_c.docx"
254+
# Does not match "draft_d.docx"
255+
draft_[abc]
256+
257+
# Matches all files that start with "IMG_" and end in a single alphabetic character
258+
IMG_[a-z]
259+
```
260+
<!-- prettier-ignore-end -->
261+
262+
---
263+
264+
## Use Cases
265+
266+
### Ignoring Files by Extension
267+
268+
<!-- prettier-ignore -->
269+
=== "Ignore all .jpg files"
270+
```toml
271+
*.jpg
272+
```
273+
=== "Ignore all files EXCEPT .jpg files"
274+
```toml
275+
*
276+
!*.jpg
277+
```
278+
=== "Ignore all .jpg files in specific folders"
279+
```toml
280+
./Photos/Worst Vacation/*.jpg
281+
Music/Artwork Art/*.jpg
282+
```
283+
284+
<!-- prettier-ignore -->
285+
!!! tip "Ensuring Complete Extension Matches"
286+
For some filetypes, it may be nessisary to specify different casing and alternative spellings in order to match with all possible variations of an extension in your library.
287+
288+
```toml title="Ignore (Most) Possible JPEG File Extensions"
289+
# The JPEG Cinematic Universe
290+
*.jpg
291+
*.jpeg
292+
*.jfif
293+
*.jpeg_large
294+
*.JPG
295+
*.JPEG
296+
*.JFIF
297+
*.JPEG_LARGE
298+
```
299+
300+
### Ignoring a Folder
301+
302+
<!-- prettier-ignore -->
303+
=== "Ignore all "Cache" folders"
304+
```toml
305+
# Matches any folder called "Cache" no matter where it is in your library.
306+
cache/
307+
```
308+
=== "Ignore a "Downloads" folder"
309+
```toml
310+
# "Downloads" must be a folder on the same level as your ".TagStudio" folder.
311+
# Does not match with folders name "Downloads" elsewhere in your library
312+
# Does not match with a file called "Downloads"
313+
/Downloads/
314+
```
315+
=== "Ignore .jpg files in specific folders"
316+
```toml
317+
Photos/Worst Vacation/*.jpg
318+
/Music/Artwork Art/*.jpg
319+
```
320+
321+
[^1]: The term "Git" is a licensed trademark of "The Git Project", a member of the Software Freedom Conservancy. Git is released under the [GNU General Public License version 2.0](https://opensource.org/license/GPL-2.0), an open source license. TagStudio is not associated with the Git Project, only including systems based on some therein.

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ nav:
4343
- library/tag_categories.md
4444
- library/tag_color.md
4545
- Utilities:
46+
- utilities/ignore.md
4647
- utilities/macro.md
4748
- Updates:
4849
- updates/changelog.md

nix/package/default.nix

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
qt6,
77
stdenv,
88
wrapGAppsHook,
9+
wcmatch,
910

1011
pillow-jxl-plugin,
1112
pyside6,

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ dependencies = [
3030
"toml~=0.10",
3131
"typing_extensions~=4.13",
3232
"ujson~=5.10",
33+
"wcmatch==10.*",
3334
]
3435

3536
[project.optional-dependencies]

src/tagstudio/core/constants.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
TS_FOLDER_NAME: str = ".TagStudio"
1010
BACKUP_FOLDER_NAME: str = "backups"
1111
COLLAGE_FOLDER_NAME: str = "collages"
12+
IGNORE_NAME: str = ".ts_ignore"
1213
THUMB_CACHE_NAME: str = "thumbs"
1314

1415
FONT_SAMPLE_TEXT: str = (

0 commit comments

Comments
 (0)