Commit cc3ca22
authored
to 3.0: supporting data branch diff as file with stage. (#23189)
### **User description**
## What type of PR is this?
- [ ] API-change
- [ ] BUG
- [x] Improvement
- [ ] Documentation
- [ ] Feature
- [ ] Test and CI
- [ ] Code Refactoring
## Which issue(s) this PR fixes:
issue ##21979
## What this PR does / why we need it:
```
create stage stage02 url = 's3://.../' credentials = {"aws_key_id"='...',"aws_secret_key"='...',"AWS_REGION"='...','PROVIDER'='...', 'ENDPOINT'='...'};
data branch diff t2 against t1 output 'stage://stage02';
```
___
### **PR Type**
Enhancement
___
### **Description**
- Add stage path decoding support for data branch diff output
- Refactor file path handling to support both local and stage paths
- Implement pipe-based file writing for improved performance
- Add context cancellation checks in diff operations
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Data Branch Diff"] -->|"Output to stage or local"| B["tryDecodeStagePath"]
B -->|"Stage path detected"| C["prepareFSForDiffAsFile"]
B -->|"Local path"| D["Standard file handling"]
C -->|"Mutable FS"| E["Direct mutator write"]
C -->|"Immutable FS"| F["newSingleWriteAppender"]
F -->|"Pipe-based writing"| G["File output"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Refactoring</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>authenticate.go</strong><dd><code>Extract stage path
decoding into helper function</code>
</dd></summary>
<hr>
pkg/frontend/authenticate.go
<ul><li>Extract stage path decoding logic into new
<code>tryDecodeStagePath</code> function<br> <li> Refactor
<code>doCheckFilePath</code> to use the new helper function<br> <li>
Improve error handling and return value clarity</ul>
</details>
</td>
<td><a
href="https://github.com/matrixorigin/matrixone/pull/23189/files#diff-849f201c351210bd95807e99d1538e2602a5244b256c35e58467afe304c509e6">+38/-15</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>data_branch.go</strong><dd><code>Add stage path support
and pipe-based file writing</code>
</dd></summary>
<hr>
pkg/frontend/data_branch.go
<ul><li>Add <code>io</code> package import for pipe-based file
writing<br> <li> Disable validation function to allow stage path
processing<br> <li> Increase channel buffer size from 10 to 100 for
better throughput<br> <li> Fix context parameter in
<code>diffOnBase</code> call from <code>execCtx.reqCtx</code> to
<code>ctx</code><br> <li> Refactor <code>prepareFSForDiffAsFile</code>
to support stage paths with proper <br>cleanup handling<br> <li>
Implement <code>newSingleWriteAppender</code> function using
<code>io.Pipe</code> for efficient <br>streaming writes<br> <li> Update
<code>writeCSV</code> to handle both stage and local file paths with
<br>appropriate cleanup<br> <li> Add context cancellation checks in
<code>diffDataHelper</code> to prevent resource <br>leaks</ul>
</details>
</td>
<td><a
href="https://github.com/matrixorigin/matrixone/pull/23189/files#diff-632e0d2d46c8bdef4144b4d6f532cf8df78cc583e4903672ae6675d8c09783b4">+154/-49</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___1 parent b716774 commit cc3ca22
File tree
3 files changed
+411
-127
lines changed- pkg/frontend
3 files changed
+411
-127
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3516 | 3516 | | |
3517 | 3517 | | |
3518 | 3518 | | |
3519 | | - | |
3520 | | - | |
3521 | | - | |
3522 | | - | |
3523 | | - | |
3524 | | - | |
| 3519 | + | |
| 3520 | + | |
| 3521 | + | |
| 3522 | + | |
| 3523 | + | |
| 3524 | + | |
| 3525 | + | |
| 3526 | + | |
3525 | 3527 | | |
3526 | | - | |
3527 | | - | |
3528 | 3528 | | |
3529 | 3529 | | |
3530 | | - | |
3531 | | - | |
3532 | | - | |
| 3530 | + | |
| 3531 | + | |
3533 | 3532 | | |
3534 | 3533 | | |
3535 | 3534 | | |
3536 | | - | |
3537 | | - | |
3538 | | - | |
| 3535 | + | |
| 3536 | + | |
3539 | 3537 | | |
| 3538 | + | |
| 3539 | + | |
3540 | 3540 | | |
3541 | | - | |
3542 | 3541 | | |
| 3542 | + | |
| 3543 | + | |
| 3544 | + | |
| 3545 | + | |
| 3546 | + | |
| 3547 | + | |
| 3548 | + | |
| 3549 | + | |
| 3550 | + | |
| 3551 | + | |
| 3552 | + | |
| 3553 | + | |
| 3554 | + | |
| 3555 | + | |
| 3556 | + | |
| 3557 | + | |
| 3558 | + | |
| 3559 | + | |
| 3560 | + | |
| 3561 | + | |
| 3562 | + | |
| 3563 | + | |
| 3564 | + | |
| 3565 | + | |
3543 | 3566 | | |
3544 | 3567 | | |
3545 | 3568 | | |
| |||
0 commit comments