Skip to content

Commit 39cc8ca

Browse files
mckenzieartsmckenziearts
authored
Feature/migrate channels to psql (#399)
1 parent 693980c commit 39cc8ca

22 files changed

+1592
-2
lines changed

‎app-modules/database-migration/README.md‎

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
# Database Migration Module
2+
3+
This module provides comprehensive MySQL to PostgreSQL migration capabilities with SSH tunnel support, specifically designed for migrating data from remote MySQL databases to local PostgreSQL databases securely.
4+
5+
## Features
6+
7+
### SSH Tunnel Management
8+
- **SSH Tunnel Creation**: Secure tunnels to remote MySQL databases
9+
- **Status Monitoring**: Check tunnel connectivity and status
10+
- **Auto-activation**: Optional automatic tunnel activation on application boot
11+
- **Background Processing**: Queue-based tunnel creation for better performance
12+
13+
### Database Migration
14+
- **Full Database Migration**: Migrate all tables from MySQL to PostgreSQL
15+
- **Selective Migration**: Migrate specific tables only
16+
- **Data Transformation**: Automatic data type conversion (MySQL → PostgreSQL)
17+
- **Chunked Processing**: Process large datasets in configurable chunks
18+
- **Progress Tracking**: Real-time migration progress with detailed output
19+
- **Dry Run Mode**: Preview migration without actually transferring data
20+
- **Verification**: Compare record counts between source and target databases
21+
22+
### Developer Experience
23+
- **Artisan Commands**: Easy-to-use CLI commands for all operations
24+
- **Exception Handling**: Custom exceptions for better error management
25+
- **Facade Support**: Simple API access through Laravel facades
26+
- **Comprehensive Testing**: Full test coverage with PestPHP
27+
- **Detailed Logging**: Configurable logging for debugging and monitoring
28+
29+
## Installation
30+
31+
Since this module is integrated into the main Laravel.cm project, it's automatically available once the project is set up.
32+
33+
### Configuration
34+
35+
Publish the configuration file:
36+
37+
```bash
38+
php artisan vendor:publish --tag=ssh-tunnel-config
39+
```
40+
41+
Configure your environment variables in `.env`:
42+
43+
```env
44+
# SSH Tunnel Configuration
45+
SSH_TUNNEL_USER=your-ssh-user
46+
SSH_TUNNEL_HOSTNAME=your-server.com
47+
SSH_TUNNEL_IDENTITY_FILE=/path/to/your/private/key
48+
SSH_TUNNEL_LOCAL_PORT=3307
49+
SSH_TUNNEL_BIND_PORT=3306
50+
SSH_TUNNEL_BIND_ADDRESS=127.0.0.1
51+
SSH_TUNNEL_AUTO_ACTIVATE=false
52+
SSH_TUNNEL_LOGGING_ENABLED=true
53+
SSH_TUNNEL_LOGGING_CHANNEL=default
54+
55+
# Database Connections (already configured in config/database.php)
56+
# Secondary connection points to MySQL via SSH tunnel
57+
DB_HOST_SECOND=127.0.0.1
58+
DB_PORT_SECOND=3307
59+
DB_DATABASE_SECOND=your_mysql_database
60+
DB_USERNAME_SECOND=your_mysql_user
61+
DB_PASSWORD_SECOND=your_mysql_password
62+
```
63+
64+
## Usage
65+
66+
### SSH Tunnel Management
67+
68+
#### Artisan Commands
69+
```bash
70+
# Activate the tunnel
71+
php artisan ssh-tunnel:activate
72+
73+
# Check tunnel status
74+
php artisan ssh-tunnel:activate --check
75+
76+
# Destroy the tunnel
77+
php artisan ssh-tunnel:activate --destroy
78+
```
79+
80+
### Database Migration
81+
82+
#### Full Migration
83+
```bash
84+
# Migrate all tables (dry run first to preview)
85+
php artisan db:migrate-mysql-to-pgsql --dry-run
86+
87+
# Perform actual migration
88+
php artisan db:migrate-mysql-to-pgsql
89+
```
90+
91+
#### Selective Migration
92+
```bash
93+
# Migrate specific tables
94+
php artisan db:migrate-mysql-to-pgsql --tables=users --tables=articles
95+
96+
# Custom chunk size for large tables
97+
php artisan db:migrate-mysql-to-pgsql --chunk=500
98+
```
99+
100+
#### Migration Options
101+
- `--tables=table1,table2`: Migrate only specified tables
102+
- `--chunk=1000`: Number of records to process per chunk (default: 1000)
103+
- `--dry-run`: Preview migration without transferring data
104+
105+
### Programmatic Usage
106+
107+
#### SSH Tunnel Service
108+
```php
109+
use Laravelcm\DatabaseMigration\Services\SshTunnelService;
110+
111+
$tunnelService = app(SshTunnelService::class);
112+
113+
// Activate tunnel
114+
$status = $tunnelService->activate();
115+
116+
// Check if tunnel is active
117+
$isActive = $tunnelService->isActive();
118+
119+
// Destroy tunnel
120+
$destroyed = $tunnelService->destroy();
121+
```
122+
123+
#### Database Migration Service
124+
```php
125+
use Laravelcm\DatabaseMigration\Services\DatabaseMigrationService;
126+
127+
$migrationService = app(DatabaseMigrationService::class);
128+
129+
// Get all source tables
130+
$tables = $migrationService->getSourceTables();
131+
132+
// Migrate a specific table
133+
$migrationService->migrateTable('users', 1000, function($processed, $total) {
134+
echo "Processed {$processed}/{$total} records\n";
135+
});
136+
137+
// Verify migration
138+
$verification = $migrationService->verifyMigration(['users', 'articles']);
139+
140+
// Test connections
141+
$connectionStatus = $migrationService->testConnections();
142+
```
143+
144+
#### Using Facades
145+
```php
146+
use Laravelcm\DatabaseMigration\Facades\SshTunnel;
147+
148+
// Activate tunnel
149+
SshTunnel::activate();
150+
151+
// Check status
152+
$isActive = SshTunnel::isActive();
153+
154+
// Destroy tunnel
155+
SshTunnel::destroy();
156+
```
157+
158+
#### Background Jobs
159+
```php
160+
use Laravelcm\DatabaseMigration\Jobs\CreateSshTunnel;
161+
162+
// Dispatch job to create tunnel in background
163+
CreateSshTunnel::dispatch();
164+
```
165+
166+
## Data Transformation
167+
168+
The migration service automatically handles common MySQL to PostgreSQL data transformations:
169+
170+
- **Boolean Fields**: MySQL tinyint(1) → PostgreSQL boolean
171+
- **Empty Strings**: Empty strings → NULL (where appropriate)
172+
- **Invalid Timestamps**: MySQL '0000-00-00 00:00:00' → NULL
173+
- **Character Encoding**: Proper UTF-8 handling
174+
175+
## Testing
176+
177+
The module includes comprehensive tests using PestPHP with 16 test cases covering:
178+
179+
- SSH tunnel functionality
180+
- Database migration operations
181+
- Command-line interfaces
182+
- Error handling scenarios
183+
184+
Run the module tests:
185+
186+
```bash
187+
php artisan test app-modules/database-migration/tests
188+
```
189+
190+
### Test Coverage
191+
- **Unit Tests**: Service classes and core functionality
192+
- **Feature Tests**: Artisan commands and integration scenarios
193+
- **Mocking**: Proper isolation of external dependencies
194+
195+
## Configuration Options
196+
197+
All configuration options are available in `config/ssh-tunnel.php`:
198+
199+
### SSH Settings
200+
- `ssh.user`: SSH username
201+
- `ssh.hostname`: Remote server hostname
202+
- `ssh.identity_file`: Path to SSH private key
203+
- `ssh.local_port`: Local port for tunnel
204+
- `ssh.bind_port`: Remote port to bind to
205+
- `ssh.bind_address`: Bind address (usually 127.0.0.1)
206+
207+
### System Executables
208+
- `executables.ssh`: Path to SSH binary
209+
- `executables.ps`: Path to ps command
210+
- `executables.grep`: Path to grep command
211+
- `executables.awk`: Path to awk command
212+
213+
### Connection Settings
214+
- `connection.max_tries`: Maximum connection retry attempts
215+
- `connection.wait_microseconds`: Wait time between retries
216+
217+
### Logging
218+
- `logging.enabled`: Enable/disable logging
219+
- `logging.channel`: Laravel log channel to use
220+
221+
### Auto-activation
222+
- `auto_activate`: Automatically activate tunnel on app boot
223+
224+
## Error Handling
225+
226+
The module includes comprehensive error handling:
227+
228+
### Custom Exceptions
229+
- `SshTunnelException`: SSH tunnel-specific errors
230+
- Detailed error messages with troubleshooting information
231+
- Proper exception chaining and context
232+
233+
### Common Issues
234+
1. **SSH Key Issues**: Ensure proper key permissions (600)
235+
2. **Port Conflicts**: Check if local port is already in use
236+
3. **Network Connectivity**: Verify SSH access to remote server
237+
4. **Database Permissions**: Ensure proper MySQL user permissions
238+
239+
## Performance Considerations
240+
241+
- **Chunked Processing**: Large tables processed in configurable chunks
242+
- **Memory Management**: Efficient memory usage for large datasets
243+
- **Progress Tracking**: Real-time feedback for long-running operations
244+
- **Connection Pooling**: Optimized database connection handling
245+
246+
## Security
247+
248+
- **SSH Key Authentication**: Secure key-based authentication
249+
- **Encrypted Tunnels**: All data transfer through encrypted SSH tunnels
250+
- **No Password Storage**: No database passwords in tunnel configuration
251+
- **Audit Logging**: Comprehensive logging for security auditing
252+
253+
## Requirements
254+
255+
- PHP 8.4+
256+
- Laravel 11+
257+
- SSH client installed on the system
258+
- Valid SSH key for remote server access
259+
- PostgreSQL and MySQL PHP extensions
260+
- Sufficient memory for large dataset processing
261+
262+
## Migration Workflow
263+
264+
1. **Setup**: Configure SSH tunnel and database connections
265+
2. **Test**: Verify connections with `ssh-tunnel:activate --check`
266+
3. **Preview**: Run migration with `--dry-run` flag
267+
4. **Execute**: Perform actual migration
268+
5. **Verify**: Check data integrity and record counts
269+
6. **Cleanup**: Optionally destroy SSH tunnel

‎app-modules/database-migration/composer.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"name": "laravelcm/database-migration",
3+
"description": "Module laravelcm/database-migration for Laravel.cm - Migration tools from MySQL to PostgreSQL with SSH tunnel support",
4+
"type": "library",
5+
"version": "1.0.0",
6+
"license": "proprietary",
7+
"require": {
8+
"php": "^8.4"
9+
},
10+
"require-dev": {
11+
"pestphp/pest": "^3.8",
12+
"pestphp/pest-plugin-laravel": "^3.0"
13+
},
14+
"autoload": {
15+
"psr-4": {
16+
"Laravelcm\\DatabaseMigration\\": "src/",
17+
"Laravelcm\\DatabaseMigration\\Database\\Factories\\": "database/factories/",
18+
"Laravelcm\\DatabaseMigration\\Database\\Seeders\\": "database/seeders/"
19+
}
20+
},
21+
"minimum-stability": "stable",
22+
"extra": {
23+
"laravel": {
24+
"providers": [
25+
"Laravelcm\\DatabaseMigration\\Providers\\DatabaseMigrationServiceProvider"
26+
]
27+
}
28+
}
29+
}

‎app-modules/database-migration/config/ssh-tunnel.php‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
return [
6+
/*
7+
|--------------------------------------------------------------------------
8+
| SSH Tunnel Configuration
9+
|--------------------------------------------------------------------------
10+
|
11+
| Configuration pour la création et maintenance d'un tunnel SSH
12+
| pour l'accès sécurisé aux bases de données distantes.
13+
|
14+
*/
15+
16+
'verify_process' => env('SSH_TUNNEL_VERIFY_PROCESS', 'nc'),
17+
18+
/*
19+
|--------------------------------------------------------------------------
20+
| Executable Paths
21+
|--------------------------------------------------------------------------
22+
*/
23+
'executables' => [
24+
'nc' => env('SSH_TUNNEL_NC_PATH', '/usr/bin/nc'),
25+
'bash' => env('SSH_TUNNEL_BASH_PATH', '/usr/bin/bash'),
26+
'ssh' => env('SSH_TUNNEL_SSH_PATH', '/usr/bin/ssh'),
27+
'nohup' => env('SSH_TUNNEL_NOHUP_PATH', '/usr/bin/nohup'),
28+
],
29+
30+
/*
31+
|--------------------------------------------------------------------------
32+
| Local Configuration
33+
|--------------------------------------------------------------------------
34+
*/
35+
'local' => [
36+
'address' => env('SSH_TUNNEL_LOCAL_ADDRESS', '127.0.0.1'),
37+
'port' => env('SSH_TUNNEL_LOCAL_PORT', 3307),
38+
],
39+
40+
/*
41+
|--------------------------------------------------------------------------
42+
| Remote Configuration
43+
|--------------------------------------------------------------------------
44+
*/
45+
'remote' => [
46+
'bind_address' => env('SSH_TUNNEL_BIND_ADDRESS', '127.0.0.1'),
47+
'bind_port' => env('SSH_TUNNEL_BIND_PORT', 3306),
48+
],
49+
50+
/*
51+
|--------------------------------------------------------------------------
52+
| SSH Connection
53+
|--------------------------------------------------------------------------
54+
*/
55+
'ssh' => [
56+
'user' => env('SSH_TUNNEL_USER'),
57+
'hostname' => env('SSH_TUNNEL_HOSTNAME'),
58+
'port' => env('SSH_TUNNEL_PORT', 22),
59+
'identity_file' => env('SSH_TUNNEL_IDENTITY_FILE', '~/.ssh/id_rsa'),
60+
'options' => env('SSH_TUNNEL_SSH_OPTIONS', '-o StrictHostKeyChecking=no'),
61+
'verbosity' => env('SSH_TUNNEL_VERBOSITY', ''),
62+
],
63+
64+
/*
65+
|--------------------------------------------------------------------------
66+
| Connection Settings
67+
|--------------------------------------------------------------------------
68+
*/
69+
'connection' => [
70+
'wait_microseconds' => env('SSH_TUNNEL_WAIT', 1000000), // 1 second
71+
'max_tries' => env('SSH_TUNNEL_MAX_TRIES', 3),
72+
'timeout_seconds' => env('SSH_TUNNEL_TIMEOUT', 30),
73+
],
74+
75+
/*
76+
|--------------------------------------------------------------------------
77+
| Logging
78+
|--------------------------------------------------------------------------
79+
*/
80+
'logging' => [
81+
'enabled' => env('SSH_TUNNEL_LOGGING', true),
82+
'nohup_log' => env('SSH_TUNNEL_NOHUP_LOG', '/dev/null'),
83+
'channel' => env('SSH_TUNNEL_LOG_CHANNEL', 'single'),
84+
],
85+
86+
/*
87+
|--------------------------------------------------------------------------
88+
| Auto-activation
89+
|--------------------------------------------------------------------------
90+
*/
91+
'auto_activate' => env('SSH_TUNNEL_AUTO_ACTIVATE', false),
92+
];

0 commit comments

Comments
 (0)