Skip to content

Commit 1322ba5

Browse files
jherrera-jumpjherrera-jump
committed
gui: overview tile stats
1 parent 152a613 commit 1322ba5

File tree

5 files changed

+260
-41
lines changed

5 files changed

+260
-41
lines changed

‎book/api/websocket.md‎

Lines changed: 161 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -575,11 +575,60 @@ the middle.
575575
Information about the tile topology of Firedancer. This is a list of
576576
tiles in the system.
577577

578+
In Firedancer, available tiles are
579+
580+
- netlnk: Helps the net tile perform network-stack related functions.
581+
Separated from the net tile for security/performance reasons.
582+
- net: Handles all ingress/egress network traffic.
583+
- metric: Serves system-wide metrics from shared memory to an HTTP
584+
Prometheus endpoint.
585+
- ipecho: Obtains shred version from cluster.
586+
- gossvf: "Gossip verify". Performs preliminary validation of untrusted
587+
gossip messages arriving from the network.
588+
- gossip: Handles trusted, parsed messages from gossvf tile.
589+
- shred: Parses, verifies, and reconstructs untrusted shred payloads
590+
from the network.
591+
- repair: Consumes parsed shreds from shred tile, issues repair requests
592+
for any missing shreds, and reconstructs the block.
593+
- replay: Consumes block shreds from repair and schedules execution and
594+
validation of block transactions.
595+
- exec: Executes replay transactions.
596+
- tower: Maintains consensus-related data structures and helps replay
597+
vote.
598+
- send: Sends vote transactions originating from this validator into our
599+
own TPU as well as to other leaders' TPU.
600+
- quic: Implements QUIC network protocol for receiving transactions.
601+
- verify: Verifies transaction signatures and performs preliminary
602+
deduplication.
603+
- dedup: Performs second deduplication pass for verified transactions.
604+
- resolv: Resolves transaction address lookup tables (i.e., tables of
605+
account addresses which transactions use but must be retrieved from the
606+
accounts database).
607+
- pack: Functions as transaction scheduler.
608+
- bank: Helps pack execute scheduled transactions.
609+
- poh: Generates block "ticks" and manages leader status.
610+
- sign: Generates signatures for various tiles which require them (e.g.
611+
repair, gossip).
612+
- rpc: Supports a subset of the Solana RPC API.
613+
- gui: Serves the GUI, which includes the WebSocket API described in
614+
this document.
615+
616+
##### short-lived
617+
618+
- snapct: Manages the snapshot loading state machine.
619+
- snapld: Loads snapshots from the network or from the file system.
620+
- snapdc: Decompresses snapshot data.
621+
- snapin: Inserts snapshot data into the accounts database.
622+
- genesi: Handles cluster bootstrapping if validator is booting a new
623+
cluster. If booting into an existing cluster, fetches cluster info (e.g.
624+
genesis hash).
625+
578626
**`Tile`**
579-
| Field | Type | Description
580-
|---------|---------|------------
581-
| kind | `string` | What kind of tile it is. One of `net`, `sock`, `quic`, `verify`, `dedup`, `pack`, `bank`, `poh`, `shred`, `store`, `sign`, `plugin`, or `http`.
582-
| kind_id | `number` | The index of the tile in its kind. For example, if there are four `verify` tiles they have `kind_id` values of 0, 1, 2, and 3 respectively.
627+
| Field | Type | Description |
628+
|---------|---------|-------------|
629+
| kind | `string` | What kind of tile it is. In Firedancer, one of the above tiles. In Frankendancer, might be one of `net`, `sock`, `quic`, `verify`, `dedup`, `pack`, `bank`, `poh`, `shred`, `store`, `sign`, `plugin`, or `http` |
630+
| kind_id | `number` | The index of the tile in its kind. For example, if there are four `verify` tiles they have `kind_id` values of 0, 1, 2, and 3 respectively |
631+
| pid | `number` | The process id of the tile |
583632

584633
::: details Example
585634

@@ -984,6 +1033,114 @@ first connect by the `summary.tiles` message.
9841033

9851034
:::
9861035

1036+
#### `summary.live_tile_metrics`
1037+
| frequency | type | example |
1038+
|------------------|---------------|---------|
1039+
| *Once* + *10ms* | `TileMetrics` | below |
1040+
1041+
Live tile metrics is a live feed of various metrics related to tile
1042+
health and resource utilization.
1043+
1044+
The timers field is a matrix of percentages, where entry on row
1045+
`i`, column `j` is the percentage of time tile `i` spent in `regimes[j]`
1046+
over the previous 10 millisecond sampling window. A value of `-1`
1047+
indicates no sample was taken in the window, typically because the tile
1048+
was context switched out by the kernel or it is hung.
1049+
1050+
The regimes array contains the processing states that a tile can exist
1051+
in. Tile regimes are the cartesian product of the following two state
1052+
vectors:
1053+
1054+
State vector 1:
1055+
1056+
- running: means that at the time the run loop executed, there was no
1057+
upstream message I/O for the tile to handle.
1058+
- processing: means that at the time the run loop executed, there was one or
1059+
more messages for the tile to consume.
1060+
- stalled: means that at the time the run loop executed, a downstream
1061+
consumer of the messages produced by this tile is slow or stalled, and
1062+
the message link for that consumer has filled up. This state causes the
1063+
tile to stop processing upstream messages.
1064+
1065+
State Vector 2:
1066+
1067+
- maintenance: the portion of the run loop that executes infrequent,
1068+
potentially CPU heavy tasks
1069+
- routine: the portion of the run loop that executes regularly,
1070+
regardless of the presence of incoming messages
1071+
- handling: the portion of the run loop that executes as a side effect
1072+
of an incoming message from an upstream producer tile
1073+
1074+
```json
1075+
[
1076+
"running_maintenance",
1077+
"processing_maintenance",
1078+
"stalled_maintenance",
1079+
"running_routine",
1080+
"processing_routine",
1081+
"stalled_routine",
1082+
"running_handling",
1083+
"processing_handling",
1084+
// "stalled_handling" is an impossible state, and is therefore excluded
1085+
]
1086+
```
1087+
1088+
The tiles indicies `i` appear in the same order here that they are
1089+
reported when you first connect by the `summary.tiles` message.
1090+
1091+
::: details Example
1092+
1093+
```json
1094+
{
1095+
"topic": "summary",
1096+
"key": "live_tile_metrics",
1097+
"value": {
1098+
"timers": [
1099+
[10.1, 0, 0, 15.3, 17, 58, 0, 0],
1100+
[10, 0, 0, 15, 17, 58, 0, 0],
1101+
...
1102+
],
1103+
"in_backp": [
1104+
0,
1105+
0,
1106+
...
1107+
],
1108+
"backp_msgs": [
1109+
0,
1110+
10,
1111+
...
1112+
],
1113+
"alive": [
1114+
1,
1115+
1,
1116+
...
1117+
],
1118+
"nvcsw": [
1119+
0,
1120+
1234,
1121+
...
1122+
],
1123+
"nivcsw": [
1124+
0,
1125+
3,
1126+
...
1127+
]
1128+
}
1129+
}
1130+
```
1131+
1132+
:::
1133+
1134+
**`TileMetrics`**
1135+
| Field | Type | Description |
1136+
|------------|--------------|-------------|
1137+
| timers | `number[][]` | `timers[i][j]` is the percentage of time from the last 10ms tile `i` spent in regime `regimes[j]` |
1138+
| in_backp | `boolean[]` | `in_backp[i]` is `true` if tile `i` is currently backpressured and `false` otherwise. See description of regimes above for more context |
1139+
| backp_msgs | `number[]` | `backp_msgs[i]` is the number of times since startup that tile `i` has had to wait for one of more consumers to catch up to resume publishing |
1140+
| alive | `boolean[]` | `alive[i]` is `true` if tile `i` has updated its heartbeat timer any time in the last 10ms and `false` otherwise |
1141+
| nvcsw | `number[]` | `nvcsw[i]` is the number of voluntary context switches the occurred for tile `i` since startup |
1142+
| nivcsw | `number[]` | `nivcsw[i]` is the number of involuntary context switches the occurred for tile `i` since startup |
1143+
9871144
### block_engine
9881145
Block engines are providers of additional transactions to the validator,
9891146
which are configurable by the operator. The validator may not be

‎src/disco/gui/fd_gui.c‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,7 @@ fd_gui_ws_open( fd_gui_t * gui,
302302
fd_gui_printf_completed_slot,
303303
fd_gui_printf_estimated_slot,
304304
fd_gui_printf_live_tile_timers,
305+
fd_gui_printf_live_tile_metrics,
305306
fd_gui_printf_catch_up_history,
306307
};
307308

@@ -345,14 +346,20 @@ fd_gui_tile_timers_snap( fd_gui_t * gui ) {
345346
}
346347
volatile ulong const * tile_metrics = fd_metrics_tile( tile->metrics );
347348

348-
cur[ i ].caughtup_housekeeping_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_CAUGHT_UP_HOUSEKEEPING ) ];
349-
cur[ i ].processing_housekeeping_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_PROCESSING_HOUSEKEEPING ) ];
350-
cur[ i ].backpressure_housekeeping_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_BACKPRESSURE_HOUSEKEEPING ) ];
351-
cur[ i ].caughtup_prefrag_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_CAUGHT_UP_PREFRAG ) ];
352-
cur[ i ].processing_prefrag_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_PROCESSING_PREFRAG ) ];
353-
cur[ i ].backpressure_prefrag_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_BACKPRESSURE_PREFRAG ) ];
354-
cur[ i ].caughtup_postfrag_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_CAUGHT_UP_POSTFRAG ) ];
355-
cur[ i ].processing_postfrag_ticks = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_PROCESSING_POSTFRAG ) ];
349+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_CAUGHT_UP_HOUSEKEEPING_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_CAUGHT_UP_HOUSEKEEPING ) ];
350+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_PROCESSING_HOUSEKEEPING_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_PROCESSING_HOUSEKEEPING ) ];
351+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_BACKPRESSURE_HOUSEKEEPING_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_BACKPRESSURE_HOUSEKEEPING ) ];
352+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_CAUGHT_UP_PREFRAG_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_CAUGHT_UP_PREFRAG ) ];
353+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_PROCESSING_PREFRAG_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_PROCESSING_PREFRAG ) ];
354+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_BACKPRESSURE_PREFRAG_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_BACKPRESSURE_PREFRAG ) ];
355+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_CAUGHT_UP_POSTFRAG_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_CAUGHT_UP_POSTFRAG ) ];
356+
cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_PROCESSING_POSTFRAG_IDX ] = tile_metrics[ MIDX( COUNTER, TILE, REGIME_DURATION_NANOS_PROCESSING_POSTFRAG ) ];
357+
358+
cur[ i ].in_backp = (int)tile_metrics[ MIDX(GAUGE, TILE, IN_BACKPRESSURE) ];
359+
cur[ i ].heartbeat = tile_metrics[ FD_METRICS_GAUGE_TILE_HEARTBEAT_OFF ];
360+
cur[ i ].backp_cnt = tile_metrics[ MIDX( COUNTER, TILE, BACKPRESSURE_COUNT ) ];
361+
cur[ i ].nvcsw = tile_metrics[ MIDX( COUNTER, TILE, CONTEXT_SWITCH_VOLUNTARY_COUNT ) ];
362+
cur[ i ].nivcsw = tile_metrics[ MIDX( COUNTER, TILE, CONTEXT_SWITCH_INVOLUNTARY_COUNT ) ];
356363
}
357364
}
358365

@@ -930,6 +937,9 @@ fd_gui_poll( fd_gui_t * gui, long now ) {
930937
fd_gui_printf_live_tile_timers( gui );
931938
fd_http_server_ws_broadcast( gui->http );
932939

940+
fd_gui_printf_live_tile_metrics( gui );
941+
fd_http_server_ws_broadcast( gui->http );
942+
933943
fd_gui_scheduler_counts_snap( gui, now );
934944

935945
if( FD_LIKELY( gui->summary.is_full_client && gui->shreds.staged_next_broadcast<gui->shreds.staged_tail ) ) {

‎src/disco/gui/fd_gui.h‎

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -233,16 +233,12 @@ struct fd_gui_validator_info {
233233
#define FD_GUI_SLOT_RANKING_TYPE_DESC (1)
234234

235235
struct fd_gui_tile_timers {
236-
ulong caughtup_housekeeping_ticks;
237-
ulong processing_housekeeping_ticks;
238-
ulong backpressure_housekeeping_ticks;
239-
240-
ulong caughtup_prefrag_ticks;
241-
ulong processing_prefrag_ticks;
242-
ulong backpressure_prefrag_ticks;
243-
244-
ulong caughtup_postfrag_ticks;
245-
ulong processing_postfrag_ticks;
236+
ulong timers[ FD_METRICS_ENUM_TILE_REGIME_CNT ];
237+
int in_backp;
238+
ulong heartbeat;
239+
ulong backp_cnt;
240+
ulong nvcsw;
241+
ulong nivcsw;
246242
};
247243

248244
typedef struct fd_gui_tile_timers fd_gui_tile_timers_t;

‎src/disco/gui/fd_gui_printf.c‎

Lines changed: 72 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
#include "../../waltz/http/fd_http_server_private.h"
55
#include "../../ballet/utf8/fd_utf8.h"
66
#include "../../disco/fd_txn_m.h"
7+
#include "../../disco/metrics/fd_metrics.h"
78

89
#ifdef __has_include
910
#if __has_include("../../app/fdctl/version.h")
@@ -491,6 +492,7 @@ fd_gui_printf_tiles( fd_gui_t * gui ) {
491492
jsonp_open_object( gui->http, NULL );
492493
jsonp_string( gui->http, "kind", tile->name );
493494
jsonp_ulong( gui->http, "kind_id", tile->kind_id );
495+
jsonp_ulong( gui->http, "pid", fd_metrics_tile( tile->metrics )[ MIDX( GAUGE, TILE, PID ) ] );
494496
jsonp_close_object( gui->http );
495497
}
496498
jsonp_close_array( gui->http );
@@ -761,23 +763,11 @@ fd_gui_printf_tile_timers( fd_gui_t * gui,
761763
continue;
762764
}
763765

764-
ulong cur_total = (cur[ i ].caughtup_housekeeping_ticks
765-
+ cur[ i ].processing_housekeeping_ticks
766-
+ cur[ i ].backpressure_housekeeping_ticks
767-
+ cur[ i ].caughtup_prefrag_ticks
768-
+ cur[ i ].processing_prefrag_ticks
769-
+ cur[ i ].backpressure_prefrag_ticks
770-
+ cur[ i ].caughtup_postfrag_ticks
771-
+ cur[ i ].processing_postfrag_ticks);
772-
773-
ulong prev_total = (prev[ i ].caughtup_housekeeping_ticks
774-
+ prev[ i ].processing_housekeeping_ticks
775-
+ prev[ i ].backpressure_housekeeping_ticks
776-
+ prev[ i ].caughtup_prefrag_ticks
777-
+ prev[ i ].processing_prefrag_ticks
778-
+ prev[ i ].backpressure_prefrag_ticks
779-
+ prev[ i ].caughtup_postfrag_ticks
780-
+ prev[ i ].processing_postfrag_ticks);
766+
ulong cur_total = 0UL;
767+
for( ulong j=0UL; j<FD_METRICS_ENUM_TILE_REGIME_CNT; j++ ) cur_total += cur[ i ].timers[ j ];
768+
769+
ulong prev_total = 0UL;
770+
for( ulong j=0UL; j<FD_METRICS_ENUM_TILE_REGIME_CNT; j++ ) prev_total += prev[ i ].timers[ j ];
781771

782772
double idle_ratio;
783773
if( FD_UNLIKELY( cur_total==prev_total ) ) {
@@ -786,15 +776,67 @@ fd_gui_printf_tile_timers( fd_gui_t * gui,
786776
JSON. */
787777
idle_ratio = -1;
788778
} else {
789-
ulong idle_time = cur[ i ].caughtup_postfrag_ticks - prev[ i ].caughtup_postfrag_ticks;
790-
ulong backpressure_time = cur[ i ].backpressure_prefrag_ticks - prev[ i ].backpressure_prefrag_ticks;
779+
ulong idle_time = cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_CAUGHT_UP_POSTFRAG_IDX ] - prev[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_CAUGHT_UP_POSTFRAG_IDX ];
780+
ulong backpressure_time = cur[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_BACKPRESSURE_PREFRAG_IDX ] - prev[ i ].timers[ FD_METRICS_ENUM_TILE_REGIME_V_BACKPRESSURE_PREFRAG_IDX ];
791781
idle_ratio = (double)(idle_time+backpressure_time) / (double)(cur_total - prev_total);
792782
}
793783

794784
jsonp_double( gui->http, NULL, idle_ratio );
795785
}
796786
}
797787

788+
static void
789+
fd_gui_printf_tile_metrics( fd_gui_t * gui,
790+
fd_gui_tile_timers_t const * prev,
791+
fd_gui_tile_timers_t const * cur ) {
792+
jsonp_open_array( gui->http, "timers" );
793+
for( ulong i=0UL; i<gui->topo->tile_cnt; i++ ) {
794+
fd_topo_tile_t const * tile = &gui->topo->tiles[ i ];
795+
796+
if( FD_UNLIKELY( !strncmp( tile->name, "bench", 5UL ) ) ) {
797+
/* bench tiles not reported */
798+
continue;
799+
}
800+
801+
ulong cur_total = 0UL;
802+
for( ulong j=0UL; j<FD_METRICS_ENUM_TILE_REGIME_CNT; j++ ) cur_total += cur[ i ].timers[ j ];
803+
804+
ulong prev_total = 0UL;
805+
for( ulong j=0UL; j<FD_METRICS_ENUM_TILE_REGIME_CNT; j++ ) prev_total += prev[ i ].timers[ j ];
806+
807+
if( FD_UNLIKELY( cur_total==prev_total ) ) {
808+
jsonp_open_array( gui->http, NULL );
809+
for( ulong j=0UL; j<FD_METRICS_ENUM_TILE_REGIME_CNT; j++ ) jsonp_double( gui->http, NULL, -1.0 );
810+
jsonp_close_array( gui->http );
811+
} else {
812+
jsonp_open_array( gui->http, NULL );
813+
for (ulong j = 0UL; j < FD_METRICS_ENUM_TILE_REGIME_CNT; j++) {
814+
double percent = ((double)(cur[ i ].timers[ j ] - prev[ i ].timers[ j ]) / (double)(cur_total-prev_total)) * 100.0;
815+
double percent_trunc = (double)((long)(percent * 100.0)) / 100.0;
816+
jsonp_double( gui->http, NULL, percent_trunc );
817+
}
818+
jsonp_close_array( gui->http );
819+
}
820+
}
821+
jsonp_close_array( gui->http );
822+
823+
jsonp_open_array( gui->http, "in_backp" );
824+
for( ulong i=0UL; i<gui->topo->tile_cnt; i++ ) jsonp_bool( gui->http, NULL, cur[ i ].in_backp );
825+
jsonp_close_array( gui->http );
826+
jsonp_open_array( gui->http, "backp_msgs" );
827+
for( ulong i=0UL; i<gui->topo->tile_cnt; i++ ) jsonp_ulong( gui->http, NULL, cur[ i ].backp_cnt );
828+
jsonp_close_array( gui->http );
829+
jsonp_open_array( gui->http, "alive" );
830+
for( ulong i=0UL; i<gui->topo->tile_cnt; i++ ) jsonp_ulong( gui->http, NULL, cur[ i ].heartbeat>prev[ i ].heartbeat );
831+
jsonp_close_array( gui->http );
832+
jsonp_open_array( gui->http, "nvcsw" );
833+
for( ulong i=0UL; i<gui->topo->tile_cnt; i++ ) jsonp_ulong( gui->http, NULL, cur[ i ].nvcsw );
834+
jsonp_close_array( gui->http );
835+
jsonp_open_array( gui->http, "nivcsw" );
836+
for( ulong i=0UL; i<gui->topo->tile_cnt; i++ ) jsonp_ulong( gui->http, NULL, cur[ i ].nivcsw );
837+
jsonp_close_array( gui->http );
838+
}
839+
798840
void
799841
fd_gui_printf_live_tile_timers( fd_gui_t * gui ) {
800842
jsonp_open_envelope( gui->http, "summary", "live_tile_timers" );
@@ -806,6 +848,17 @@ fd_gui_printf_live_tile_timers( fd_gui_t * gui ) {
806848
jsonp_close_envelope( gui->http );
807849
}
808850

851+
void
852+
fd_gui_printf_live_tile_metrics( fd_gui_t * gui ) {
853+
fd_gui_tile_timers_t * cur = gui->summary.tile_timers_snap[ (gui->summary.tile_timers_snap_idx+(FD_GUI_TILE_TIMER_SNAP_CNT-1UL))%FD_GUI_TILE_TIMER_SNAP_CNT ];
854+
fd_gui_tile_timers_t * prev = gui->summary.tile_timers_snap[ (gui->summary.tile_timers_snap_idx+(FD_GUI_TILE_TIMER_SNAP_CNT-2UL))%FD_GUI_TILE_TIMER_SNAP_CNT ];
855+
jsonp_open_envelope( gui->http, "summary", "live_tile_metrics" );
856+
jsonp_open_object( gui->http, "value" );
857+
fd_gui_printf_tile_metrics( gui, prev, cur );
858+
jsonp_close_object( gui->http );
859+
jsonp_close_envelope( gui->http );
860+
}
861+
809862
void
810863
fd_gui_printf_estimated_tps( fd_gui_t * gui ) {
811864
ulong idx = (gui->summary.estimated_tps_history_idx+FD_GUI_TPS_HISTORY_SAMPLE_CNT-1UL) % FD_GUI_TPS_HISTORY_SAMPLE_CNT;

‎src/disco/gui/fd_gui_printf.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,9 @@ void
133133
fd_gui_printf_live_network_metrics( fd_gui_t * gui,
134134
fd_gui_network_stats_t const * cur );
135135

136+
void
137+
fd_gui_printf_live_tile_metrics( fd_gui_t * gui );
138+
136139
void
137140
fd_gui_printf_live_txn_waterfall( fd_gui_t * gui,
138141
fd_gui_txn_waterfall_t const * prev,

0 commit comments

Comments
 (0)