-
Notifications
You must be signed in to change notification settings - Fork 21.5k
Description
Hey everyone,
I’m running into a weird issue and would appreciate some insight.
Setup:
Geth 1.16.5 + Prysm 6.1.4
10-core CPU, 32 GB RAM, fast NVMe, Ubuntu 24.04.3
Internet via TCP-only VPN (150+ Mbps inside the tunnel)
Everything used to work flawlessly on Geth ≤ 1.16.3 and Prysm 6.1.3 - no sync issues at all.
After upgrading, however, Geth started lagging behind the network by exactly 1 epoch. The age= value floats between 6 min - 6 min 30 sec.
eth.syncing returned false - the node is fully synced. Performance-wise everything looks fine - blocks are processed quickly, network connection is stable, plenty of peers (≈ 60 for Geth, ≈ 70 for Prysm). Prysm is fully synced and uses the checkpoint from beaconstate.info.
Rolling back to Geth 1.16.4 immediately fixed the issue - the 1-epoch delay disappeared and the node was running perfectly again. But about 8 hours later, the lag came back for no apparent reason.
Then I tried upgrading back to 1.16.5, wiped all Geth data, and did a full resync from scratch - the node ran fine for about 1.5 days, but at epoch 404224 the same issue reappeared: Geth started falling behind by 6–6.5 minutes consistently.
Any idea what could be causing this?
_The moment the failure started (Geth UTC+2):_
nov 02 03:13:25 sprmcr geth[20759]: INFO [11-02|03:13:25.406] Imported new potential chain segment number=23,708,285 hash=666dec..9d9e95 blocks=1 txs=147 mgas=16.063 elapsed=114.793ms mgasps=139.926 triediffs=297.30MiB triedirty=82.47MiB
nov 02 03:13:25 sprmcr geth[20759]: INFO [11-02|03:13:25.603] Chain head was updated number=23,708,285 hash=666dec..9d9e95 root=54f0c9..ab79e9 elapsed=5.621885ms
nov 02 03:13:37 sprmcr geth[20759]: INFO [11-02|03:13:37.216] Imported new potential chain segment number=23,708,286 hash=f8ca54..dfe661 blocks=1 txs=179 mgas=25.146 elapsed=343.293ms mgasps=73.250 triediffs=299.25MiB triedirty=82.73MiB
nov 02 03:13:37 sprmcr geth[20759]: INFO [11-02|03:13:37.347] Chain head was updated number=23,708,286 hash=f8ca54..dfe661 root=9c455b..36dba3 elapsed=5.095856ms
nov 02 03:13:49 sprmcr geth[20759]: INFO [11-02|03:13:49.200] Imported new potential chain segment number=23,708,287 hash=a3506f..dd1063 blocks=1 txs=123 mgas=18.746 elapsed=223.701ms mgasps=83.801 triediffs=298.09MiB triedirty=84.10MiB
nov 02 03:13:49 sprmcr geth[20759]: INFO [11-02|03:13:49.334] Chain head was updated number=23,708,287 hash=a3506f..dd1063 root=6e160f..a4fc9e elapsed=6.080359ms
**nov 02 03:15:19 sprmcr geth[20759]: INFO [11-02|03:15:19.129] Imported new potential chain segment number=23,708,288 hash=a03033..23696c blocks=1 txs=148 mgas=15.621 elapsed=84.285ms mgasps=185.340 age=1m20s triediffs=299.04MiB triedirty=84.27MiB
nov 02 03:15:40 sprmcr geth[20759]: WARN [11-02|03:15:40.785] Ignoring already known beacon payload number=23,708,288 hash=a03033..23696c age=1m41s**
nov 02 03:15:55 sprmcr geth[20759]: INFO [11-02|03:15:55.497] Chain head was updated number=23,708,288 hash=a03033..23696c root=e472b3..8b63a8 elapsed=3.705243ms age=1m56s
nov 02 03:16:03 sprmcr geth[20759]: INFO [11-02|03:16:03.922] Imported new potential chain segment number=23,708,289 hash=2aa82e..c9681f blocks=1 txs=109 mgas=45.130 elapsed=246.487ms mgasps=183.093 age=1m52s triediffs=298.77MiB triedirty=86.26MiB
nov 02 03:16:04 sprmcr geth[20759]: INFO [11-02|03:16:04.421] Chain head was updated number=23,708,289 hash=2aa82e..c9681f root=58d108..3b3dd4 elapsed=3.289096ms age=1m53s
nov 02 03:16:12 sprmcr geth[20759]: INFO [11-02|03:16:12.248] Imported new potential chain segment number=23,708,290 hash=4aafee..7034cf blocks=1 txs=190 mgas=29.893 elapsed=249.033ms mgasps=120.035 age=1m49s triediffs=299.47MiB triedirty=87.07MiB
nov 02 03:16:12 sprmcr geth[20759]: INFO [11-02|03:16:12.405] Chain head was updated number=23,708,290 hash=4aafee..7034cf root=a92114..bac1d0 elapsed=17.28981ms age=1m49s
nov 02 03:16:23 sprmcr geth[20759]: INFO [11-02|03:16:23.239] Imported new potential chain segment number=23,708,291 hash=9fffe6..9de60c blocks=1 txs=71 mgas=7.219 elapsed=44.594ms mgasps=161.888 age=1m48s triediffs=299.41MiB triedirty=87.20MiB
nov 02 03:16:23 sprmcr geth[20759]: INFO [11-02|03:16:23.488] Chain head was updated number=23,708,291 hash=9fffe6..9de60c root=8f62db..4d03ab elapsed=3.199377ms age=1m48s
nov 02 03:16:38 sprmcr geth[20759]: INFO [11-02|03:16:38.067] Imported new potential chain segment number=23,708,292 hash=9d7c27..c40f88 blocks=1 txs=196 mgas=34.055 elapsed=253.980ms mgasps=134.085 age=1m51s triediffs=299.07MiB triedirty=88.84MiB
nov 02 03:16:38 sprmcr geth[20759]: INFO [11-02|03:16:38.246] Chain head was updated number=23,708,292 hash=9d7c27..c40f88 root=82f3ec..26c54b elapsed=18.268169ms age=1m51s
nov 02 03:16:46 sprmcr geth[20759]: INFO [11-02|03:16:46.700] Imported new potential chain segment number=23,708,293 hash=332b1f..ad45c0 blocks=1 txs=108 mgas=9.699 elapsed=45.740ms mgasps=212.043 age=1m47s triediffs=298.27MiB triedirty=89.29MiB
nov 02 03:16:46 sprmcr geth[20759]: INFO [11-02|03:16:46.958] Chain head was updated number=23,708,293 hash=332b1f..ad45c0 root=de2232..660009 elapsed=4.043487ms age=1m47s
nov 02 03:16:59 sprmcr geth[20759]: INFO [11-02|03:16:59.435] Imported new potential chain segment number=23,708,294 hash=7c927c..59ecbe blocks=1 txs=199 mgas=25.621 elapsed=114.144ms mgasps=224.462 age=1m48s triediffs=298.34MiB triedirty=89.95MiB
nov 02 03:17:00 sprmcr geth[20759]: INFO [11-02|03:17:00.074] Chain head was updated number=23,708,294 hash=7c927c..59ecbe root=75c7c5..853122 elapsed=13.578352ms age=1m49s
nov 02 03:17:12 sprmcr geth[20759]: INFO [11-02|03:17:12.427] Imported new potential chain segment number=23,708,295 hash=8a8e59..87050a blocks=1 txs=123 mgas=15.251 elapsed=209.287ms mgasps=72.870 age=1m49s triediffs=299.03MiB triedirty=90.25MiB
nov 02 03:17:12 sprmcr geth[20759]: INFO [11-02|03:17:12.551] Chain head was updated number=23,708,295 hash=8a8e59..87050a root=a28086..998d35 elapsed=8.381493ms age=1m49s
nov 02 03:17:24 sprmcr geth[20759]: INFO [11-02|03:17:24.191] Imported new potential chain segment number=23,708,296 hash=ecccd8..84a944 blocks=1 txs=148 mgas=18.593 elapsed=107.037ms mgasps=173.707 age=1m49s triediffs=299.20MiB triedirty=90.68MiB
nov 02 03:17:24 sprmcr geth[20759]: INFO [11-02|03:17:24.353] Chain head was updated number=23,708,296 hash=ecccd8..84a944 root=c53013..351962 elapsed=7.117142ms age=1m49s
nov 02 03:17:37 sprmcr geth[20759]: INFO [11-02|03:17:37.908] Imported new potential chain segment number=23,708,297 hash=89d4b6..88dd3f blocks=1 txs=170 mgas=39.835 elapsed=155.598ms mgasps=256.012 age=1m50s triediffs=298.95MiB triedirty=92.42MiB
nov 02 03:17:38 sprmcr geth[20759]: INFO [11-02|03:17:38.062] Chain head was updated number=23,708,297 hash=89d4b6..88dd3f root=90867e..6c92f8 elapsed=14.499938ms age=1m51s
The age= value stops increasing at 6-6.5 minutes and does not grow further. After restarting Geth, the lag starts increasing smoothly again, from 1 minute up to 6.5 minutes.