fix directory parsing in data_pipeline

[stefpi]

In data_pipeline function should_process_file directory names in included_dirs and excluded_dirs uses .strip(“./“). The strip function in python treats “./“ as [“.”, “/“] and removes each occurrence of either character from the start or end of the string until neither are found.

This means that a string such as "./.git/" will become “git” when .strip(“./“) is applied. This leads to no documents being processed from a directory if “git” is present anywhere in the file path. Some people like me use “~/git/” to store their git projects, which means all my documents were being excluded although they aren’t in the “.git” directory.

I changed the parsing to use .removeprefix("./").removesuffix("/") which means any folder of the structure “././“ will become “.”.

[gemini-code-assist]

Summary of Changes

Hello @stefpi, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical bug in the data_pipeline's directory parsing logic. Previously, the use of strip("./") for cleaning directory names led to incorrect matching, particularly for paths containing substrings like "git" (e.g., ~/git/<project>). The fix updates the parsing to use split("./")[1].rstrip("/"), ensuring that directory names are accurately extracted and processed for inclusion and exclusion rules, thereby preventing legitimate files from being inadvertently ignored.

Highlights

• Directory Parsing Fix: Corrected an issue in data_pipeline's should_process_file where strip("./") incorrectly processed directory names, leading to unintended file exclusions.
• Improved String Manipulation: Replaced strip("./") with split("./")[1].rstrip("/") to accurately parse directory names, preventing ".git/" from becoming "git" and causing false exclusions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly identifies and fixes a bug in how directory paths are parsed, where strip('./') was too aggressive and could incorrectly filter paths. However, the proposed solution using split('./')[1] is not robust and can lead to an IndexError if a directory name doesn't contain './', which is a likely scenario for user-provided directories. I've left comments with suggestions for a safer implementation that avoids this new issue while still fixing the original problem.