Backport enhanced analytics support and comprehensive documentation from v3.0

nyetwurk · nyetwurk · commit 834f25554aeb · 2025-08-22T19:08:47.000-07:00
diff --git a/README.md b/README.md
@@ -2,6 +2,34 @@
 
 Kafka publisher for use with Solana's [plugin framework](https://docs.solana.com/developing/plugins/geyser-plugins).
 
+## What This Plugin Does
+
+Streams **three types** of Solana data to Kafka:
+- **Account Updates**: Balance changes, data modifications, owner changes
+- **Transactions**: Complete transaction details, instructions, fees, success/failure
+- **Slot Status**: Network progress, confirmation status, health metrics
+
+## Quick Start
+
+**Want to see data flowing immediately?** Use this minimal config:
+
+```json
+{
+  "libpath": "target/release/libsolana_accountsdb_plugin_kafka.so",
+  "kafka": {
+    "bootstrap.servers": "localhost:9092"
+  },
+  "filters": [{
+    "update_account_topic": "solana.testnet.account_updates",
+    "slot_status_topic": "solana.testnet.slot_status",
+    "transaction_topic": "solana.testnet.transactions",
+    "publish_all_accounts": true
+  }]
+}
+```
+
+This will publish **all** account updates, transactions, and slot status to Kafka. Perfect for testing and development.
+
 ## Installation
 
 ### Binary releases
@@ -40,6 +68,10 @@ Config is specified via the plugin's JSON config file.
 
 ### Example Config
 
+**⚠️ WARNING: This example config will NOT publish most data by default!**
+
+The following config is a minimal example that demonstrates the structure, but with `publish_all_accounts: false` and no `program_filters`, you'll only see slot status updates. For testing, consider using the scenarios below.
+
 ```json
 {
   "libpath": "target/release/libsolana_accountsdb_plugin_kafka.so",
@@ -64,12 +96,31 @@ Config is specified via the plugin's JSON config file.
     "wrap_messages": false
   }]
 }
+
+**For Testing/Development (Recommended):**
+```json
+{
+  "libpath": "target/release/libsolana_accountsdb_plugin_kafka.so",
+  "kafka": {
+    "bootstrap.servers": "localhost:9092"
+  },
+  "filters": [{
+    "update_account_topic": "solana.testnet.account_updates",
+    "slot_status_topic": "solana.testnet.slot_status",
+    "transaction_topic": "solana.testnet.transactions",
+    "publish_all_accounts": true
+  }]
+}
+```
 ```
 
 ### Reference
 
 - `libpath`: Path to Kafka plugin
-- `kafka`: [`librdkafka` config options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).
+- `kafka`: [`librdkafka` config options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). Common options include:
+  - `statistics.interval.ms`: Enables Prometheus metrics collection (set to 1000ms or higher)
+  - `queue.buffering.max.messages`: Controls producer buffer size
+  - `queue.buffering.max.kbytes`: Controls producer buffer size in KB
 - `shutdown_timeout_ms`: Time the plugin is given to flush out all messages to Kafka upon exit request.
 - `prometheus`: Optional port to provide metrics in Prometheus format.
 - `filters`: Vec of filters with next fields:
@@ -94,9 +145,57 @@ The message types are keyed as follows:
 
 ### Filtering
 
-If `program_ignores` are specified, then these addresses will be filtered out of the account updates
-and transaction notifications.  More specifically, account update messages for these accounts will not be emitted,
-and transaction notifications for any transaction involving these accounts will not be emitted.
+**⚠️ IMPORTANT: Understanding how filtering works is crucial for getting data flowing to Kafka!**
+
+The plugin uses a **whitelist approach** for filtering. By default, most events are filtered out unless you explicitly configure what you want to see.
+
+#### How Filtering Works
+
+1. **Account Updates**: Only published if the account's owner program is in `program_filters` OR the account address is in `account_filters`
+2. **Transactions**: Only published if they involve accounts from programs in `program_filters` OR specific accounts in `account_filters`
+3. **Slot Status**: Always published (not affected by filters)
+4. **Program Ignores**: Blacklist of programs/accounts to exclude (applied after whitelist filtering)
+
+#### Common Filtering Scenarios
+
+**Scenario 1: See Everything (Recommended for testing)**
+```json
+{
+  "publish_all_accounts": true,
+  "program_filters": [],
+  "account_filters": []
+}
+```
+
+**Scenario 2: See Specific Programs Only**
+```json
+{
+  "publish_all_accounts": false,
+  "program_filters": [
+    "11111111111111111111111111111111",  // System Program
+    "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"  // Token Program
+  ]
+}
+```
+
+**Scenario 3: See Specific Accounts Only**
+```json
+{
+  "publish_all_accounts": false,
+  "account_filters": [
+    "YourAccountAddressHere111111111111111111111111"
+  ]
+}
+```
+
+#### Troubleshooting: No Data in Kafka?
+
+If you're not seeing messages in Kafka despite successful slot processing:
+
+1. **Check your filters**: Make sure you have either `publish_all_accounts: true` or specific `program_filters`/`account_filters`
+2. **Verify topics**: Ensure your topic names are correct and Kafka is running
+3. **Check program ignores**: Make sure you're not accidentally filtering out everything with overly restrictive `program_ignores`
+4. **Test with minimal config**: Start with `publish_all_accounts: true` to verify the plugin is working
 
 ### Message Wrapping
 
@@ -109,6 +208,92 @@ Note that if `wrap_messages` is true, in order to avoid key collision, the messa
 which is dependent on the type of the message being wrapped.  Account update message keys are prefixed with
 65 (A), slot status keys with 83 (S), and transaction keys with 84 (T).
 
+## Enhanced Analytics Support
+
+The plugin now provides significantly richer analytics data for blockchain monitoring and analysis, enhancing all supported Solana transaction formats.
+
+### Transaction Types
+
+The plugin supports multiple Solana transaction formats:
+
+- **Legacy Transactions**: Traditional Solana message format
+- **V0 Transactions**: Versioned transactions with address lookup tables (LUTs)
+
+All transaction types are enhanced with comprehensive analytics metadata.
+
+### Enhanced Analytics Features
+
+All transactions provide additional analytics data that can be used for:
+
+#### Performance Monitoring
+- **Compute Units**: Actual compute units consumed by transactions
+- **Pricing Information**: Compute unit pricing from ComputeBudget instructions
+- **Cost Analysis**: Transaction fees and compute costs
+
+#### Error Analysis & Debugging
+- **Error Detection**: Reliable error status from transaction metadata
+- **Success Tracking**: Transaction success/failure status
+- **Error Details**: Structured error information without log parsing
+
+#### Address Intelligence
+- **Address Lookup Tables**: Support for V0 LUT transactions
+- **Loaded Address Details**: Index and writable status for loaded addresses
+- **Account Metadata**: Enhanced account information and versioning
+
+#### Slot & Network Analytics
+- **Confirmation Status**: Smart confirmation counting based on slot status
+- **Status Descriptions**: Human-readable slot status descriptions
+- **Progress Tracking**: Slot progression monitoring
+
+### Message Schema Enhancements
+
+The protobuf schema has been enhanced with analytics fields:
+
+#### UpdateAccountEvent
+- `data_version`: Account data version for change tracking
+- `is_startup`: Whether this is a startup account update
+- `account_age`: Approximate account age in slots
+
+#### SlotStatusEvent
+- `is_confirmed`: Whether slot is confirmed by supermajority
+- `confirmation_count`: Confirmation level (0-2 based on status)
+- `status_description`: Human-readable status description
+
+#### TransactionEvent
+- `compute_units_consumed`: Actual compute units used
+- `compute_units_price`: Compute unit pricing in micro-lamports
+- `total_cost`: Total transaction cost (fee + compute)
+- `instruction_count`: Number of instructions in transaction
+- `account_count`: Number of accounts involved
+- `execution_time_ns`: Execution time in nanoseconds
+- `is_successful`: Transaction success status
+- `execution_logs`: Detailed execution logs
+- `error_details`: Detailed error information
+- `confirmation_count`: Number of confirmations
+
+#### LoadedAddresses
+- `writable_info`: Detailed writable address information
+- `readonly_info`: Detailed readonly address information
+
+### Configuration for Analytics Features
+
+Analytics features are enabled by default and require no additional configuration. The plugin automatically:
+
+- Detects transaction format (Legacy or V0)
+- Extracts available metadata fields
+- Provides enhanced analytics where data is available
+- Maintains backwards compatibility with existing consumers
+
+### Use Cases
+
+Analytics enhancements enable new use cases:
+
+- **Performance Monitoring**: Track compute unit usage and costs
+- **Error Analysis**: Monitor transaction failures and success rates
+- **Network Analytics**: Analyze slot confirmation patterns
+- **Address Intelligence**: Monitor address lookup table usage
+- **Cost Optimization**: Analyze transaction pricing and efficiency
+
 ## Buffering
 
 The Kafka producer acts strictly non-blocking to allow the Solana validator to sync without much induced lag.
@@ -122,3 +307,61 @@ The buffer size can be controlled using `librdkafka` config options, including:
 
 - `queue.buffering.max.messages`: Maximum number of messages allowed on the producer queue.
 - `queue.buffering.max.kbytes`: Maximum total message size sum allowed on the producer queue.
+
+## Troubleshooting
+
+### Common Issues
+
+#### 1. No Data in Kafka Topics
+
+**Symptoms**: Solana validator shows slot processing but no messages appear in Kafka topics.
+
+**Causes & Solutions**:
+- **Filtering too restrictive**: Set `publish_all_accounts: true` or add specific `program_filters`
+- **Wrong topic names**: Verify your topic names match exactly
+- **Kafka connection issues**: Check if Kafka is running and accessible
+- **Plugin not loaded**: Verify the plugin path in `libpath` is correct
+
+**Quick Test**: Use the Quick Start config above to verify the plugin works.
+
+#### 2. Only Slot Status Messages Appear
+
+**Cause**: This is expected behavior with the default example config! Slot status is always published, but account updates and transactions require explicit filter configuration.
+
+**Solution**: Add `publish_all_accounts: true` or configure `program_filters`.
+
+#### 3. Plugin Fails to Load
+
+**Common Causes**:
+- **Version mismatch**: Ensure Solana and plugin are built with identical Rust/Solana versions
+- **Library path**: Check `libpath` points to the correct `.so` or `.dylib` file
+- **Permissions**: Ensure the plugin file is readable by the Solana process
+
+#### 4. High Memory Usage
+
+**Cause**: Large Kafka producer buffers can consume significant memory.
+
+**Solution**: Adjust buffer settings:
+```json
+{
+  "kafka": {
+    "queue.buffering.max.messages": "10000",
+    "queue.buffering.max.kbytes": "1048576"
+  }
+}
+```
+
+### Debugging Tips
+
+1. **Start Simple**: Begin with `publish_all_accounts: true` to verify basic functionality
+2. **Check Topics**: Use Kafdrop or `kafka-console-consumer` to verify topics exist
+3. **Monitor Metrics**: Enable Prometheus metrics to see message counts and errors
+4. **Verify Filters**: Double-check your filter configuration matches your expectations
+
+### Getting Help
+
+If you're still having issues:
+1. Check this troubleshooting section
+2. Review the filtering documentation above
+3. Try the Quick Start configuration
+4. Open an issue with your config and error details
diff --git a/proto/event.proto b/proto/event.proto
@@ -35,6 +35,11 @@ message UpdateAccountEvent {
 
   // First signature of the transaction caused this account modification
   optional bytes txn_signature = 9;
+
+  // Additional analytics fields
+  uint32 data_version = 10;                   // Account data version for change tracking
+  bool is_startup = 11;                       // Whether this is a startup account update
+  uint64 account_age = 12;                    // Account age in slots since creation
 }
 
 message SlotStatusEvent {
@@ -43,6 +48,11 @@ message SlotStatusEvent {
   uint64 parent = 2;
 
   SlotStatus status = 3;
+
+  // Additional analytics fields
+  bool is_confirmed = 4;                      // Whether slot is confirmed
+  uint32 confirmation_count = 5;              // Number of confirmations
+  string status_description = 6;              // Human-readable status description
 }
 
 enum SlotStatus {
@@ -174,6 +184,9 @@ message TransactionStatusMeta {
   repeated TransactionTokenBalance pre_token_balances = 8;
   repeated TransactionTokenBalance post_token_balances = 9;
   repeated Reward rewards = 10;
+  uint32 compute_units_consumed = 11;        // Compute units used by transaction
+  repeated string error_logs = 13;           // Detailed error logs for debugging
+  bool is_successful = 14;                   // Transaction success status (inverse of is_status_err)
 }
 
 // based on solana_accountsdb_plugin_interface::accountsdb_plugin_interface::ReplicaTransactionInfo
@@ -184,6 +197,17 @@ message TransactionEvent {
   TransactionStatusMeta transaction_status_meta = 4;
   uint64 slot = 5;
   uint64 index = 6;
+
+  // Enhanced analytics fields
+  uint32 compute_units_consumed = 7;         // Compute units used by transaction
+  uint64 total_cost = 8;                     // Total transaction cost (fee + compute)
+  uint32 instruction_count = 9;              // Number of instructions in transaction
+  uint32 account_count = 10;                 // Number of accounts involved
+  uint64 execution_time_ns = 11;             // Execution time in nanoseconds
+  bool is_successful = 12;                   // Transaction success status
+  repeated string execution_logs = 13;       // Detailed execution logs
+  repeated string error_details = 14;        // Detailed error information
+  uint32 confirmation_count = 15;            // Number of confirmations
 }
 
 message MessageWrapper {
diff --git a/src/plugin.rs b/src/plugin.rs
