Skip to content

Geth 1.16.5 consistently lags 1 epoch behind #33079

@Dmitry784

Description

@Dmitry784

Hey everyone,
I’m running into a weird issue and would appreciate some insight.

Setup:
Geth 1.16.5 + Prysm 6.1.4
10-core CPU, 32 GB RAM, fast NVMe, Ubuntu 24.04.3
Internet via TCP-only VPN (150+ Mbps inside the tunnel)

Everything used to work flawlessly on Geth ≤ 1.16.3 and Prysm 6.1.3 - no sync issues at all.
After upgrading, however, Geth started lagging behind the network by exactly 1 epoch. The age= value floats between 6 min - 6 min 30 sec.

eth.syncing returned false - the node is fully synced. Performance-wise everything looks fine - blocks are processed quickly, network connection is stable, plenty of peers (≈ 60 for Geth, ≈ 70 for Prysm). Prysm is fully synced and uses the checkpoint from beaconstate.info.

Rolling back to Geth 1.16.4 immediately fixed the issue - the 1-epoch delay disappeared and the node was running perfectly again. But about 8 hours later, the lag came back for no apparent reason.

Then I tried upgrading back to 1.16.5, wiped all Geth data, and did a full resync from scratch - the node ran fine for about 1.5 days, but at epoch 404224 the same issue reappeared: Geth started falling behind by 6–6.5 minutes consistently.

Any idea what could be causing this?

_The moment the failure started (Geth UTC+2):_

nov 02 03:13:25 sprmcr geth[20759]: INFO [11-02|03:13:25.406] Imported new potential chain segment     number=23,708,285 hash=666dec..9d9e95 blocks=1         txs=147         mgas=16.063  elapsed=114.793ms     mgasps=139.926 triediffs=297.30MiB triedirty=82.47MiB
nov 02 03:13:25 sprmcr geth[20759]: INFO [11-02|03:13:25.603] Chain head was updated                   number=23,708,285 hash=666dec..9d9e95 root=54f0c9..ab79e9 elapsed=5.621885ms
nov 02 03:13:37 sprmcr geth[20759]: INFO [11-02|03:13:37.216] Imported new potential chain segment     number=23,708,286 hash=f8ca54..dfe661 blocks=1         txs=179         mgas=25.146  elapsed=343.293ms     mgasps=73.250  triediffs=299.25MiB triedirty=82.73MiB
nov 02 03:13:37 sprmcr geth[20759]: INFO [11-02|03:13:37.347] Chain head was updated                   number=23,708,286 hash=f8ca54..dfe661 root=9c455b..36dba3 elapsed=5.095856ms
nov 02 03:13:49 sprmcr geth[20759]: INFO [11-02|03:13:49.200] Imported new potential chain segment     number=23,708,287 hash=a3506f..dd1063 blocks=1         txs=123         mgas=18.746  elapsed=223.701ms     mgasps=83.801  triediffs=298.09MiB triedirty=84.10MiB
nov 02 03:13:49 sprmcr geth[20759]: INFO [11-02|03:13:49.334] Chain head was updated                   number=23,708,287 hash=a3506f..dd1063 root=6e160f..a4fc9e elapsed=6.080359ms
**nov 02 03:15:19 sprmcr geth[20759]: INFO [11-02|03:15:19.129] Imported new potential chain segment     number=23,708,288 hash=a03033..23696c blocks=1         txs=148         mgas=15.621  elapsed=84.285ms      mgasps=185.340 age=1m20s    triediffs=299.04MiB triedirty=84.27MiB
nov 02 03:15:40 sprmcr geth[20759]: WARN [11-02|03:15:40.785] Ignoring already known beacon payload    number=23,708,288 hash=a03033..23696c age=1m41s**
nov 02 03:15:55 sprmcr geth[20759]: INFO [11-02|03:15:55.497] Chain head was updated                   number=23,708,288 hash=a03033..23696c root=e472b3..8b63a8 elapsed=3.705243ms    age=1m56s
nov 02 03:16:03 sprmcr geth[20759]: INFO [11-02|03:16:03.922] Imported new potential chain segment     number=23,708,289 hash=2aa82e..c9681f blocks=1         txs=109         mgas=45.130  elapsed=246.487ms     mgasps=183.093 age=1m52s    triediffs=298.77MiB triedirty=86.26MiB
nov 02 03:16:04 sprmcr geth[20759]: INFO [11-02|03:16:04.421] Chain head was updated                   number=23,708,289 hash=2aa82e..c9681f root=58d108..3b3dd4 elapsed=3.289096ms    age=1m53s
nov 02 03:16:12 sprmcr geth[20759]: INFO [11-02|03:16:12.248] Imported new potential chain segment     number=23,708,290 hash=4aafee..7034cf blocks=1         txs=190         mgas=29.893  elapsed=249.033ms     mgasps=120.035 age=1m49s    triediffs=299.47MiB triedirty=87.07MiB
nov 02 03:16:12 sprmcr geth[20759]: INFO [11-02|03:16:12.405] Chain head was updated                   number=23,708,290 hash=4aafee..7034cf root=a92114..bac1d0 elapsed=17.28981ms    age=1m49s
nov 02 03:16:23 sprmcr geth[20759]: INFO [11-02|03:16:23.239] Imported new potential chain segment     number=23,708,291 hash=9fffe6..9de60c blocks=1         txs=71          mgas=7.219   elapsed=44.594ms      mgasps=161.888 age=1m48s    triediffs=299.41MiB triedirty=87.20MiB
nov 02 03:16:23 sprmcr geth[20759]: INFO [11-02|03:16:23.488] Chain head was updated                   number=23,708,291 hash=9fffe6..9de60c root=8f62db..4d03ab elapsed=3.199377ms    age=1m48s
nov 02 03:16:38 sprmcr geth[20759]: INFO [11-02|03:16:38.067] Imported new potential chain segment     number=23,708,292 hash=9d7c27..c40f88 blocks=1         txs=196         mgas=34.055  elapsed=253.980ms     mgasps=134.085 age=1m51s    triediffs=299.07MiB triedirty=88.84MiB
nov 02 03:16:38 sprmcr geth[20759]: INFO [11-02|03:16:38.246] Chain head was updated                   number=23,708,292 hash=9d7c27..c40f88 root=82f3ec..26c54b elapsed=18.268169ms   age=1m51s
nov 02 03:16:46 sprmcr geth[20759]: INFO [11-02|03:16:46.700] Imported new potential chain segment     number=23,708,293 hash=332b1f..ad45c0 blocks=1         txs=108         mgas=9.699   elapsed=45.740ms      mgasps=212.043 age=1m47s    triediffs=298.27MiB triedirty=89.29MiB
nov 02 03:16:46 sprmcr geth[20759]: INFO [11-02|03:16:46.958] Chain head was updated                   number=23,708,293 hash=332b1f..ad45c0 root=de2232..660009 elapsed=4.043487ms    age=1m47s
nov 02 03:16:59 sprmcr geth[20759]: INFO [11-02|03:16:59.435] Imported new potential chain segment     number=23,708,294 hash=7c927c..59ecbe blocks=1         txs=199         mgas=25.621  elapsed=114.144ms     mgasps=224.462 age=1m48s    triediffs=298.34MiB triedirty=89.95MiB
nov 02 03:17:00 sprmcr geth[20759]: INFO [11-02|03:17:00.074] Chain head was updated                   number=23,708,294 hash=7c927c..59ecbe root=75c7c5..853122 elapsed=13.578352ms   age=1m49s
nov 02 03:17:12 sprmcr geth[20759]: INFO [11-02|03:17:12.427] Imported new potential chain segment     number=23,708,295 hash=8a8e59..87050a blocks=1         txs=123         mgas=15.251  elapsed=209.287ms     mgasps=72.870  age=1m49s    triediffs=299.03MiB triedirty=90.25MiB
nov 02 03:17:12 sprmcr geth[20759]: INFO [11-02|03:17:12.551] Chain head was updated                   number=23,708,295 hash=8a8e59..87050a root=a28086..998d35 elapsed=8.381493ms    age=1m49s
nov 02 03:17:24 sprmcr geth[20759]: INFO [11-02|03:17:24.191] Imported new potential chain segment     number=23,708,296 hash=ecccd8..84a944 blocks=1         txs=148         mgas=18.593  elapsed=107.037ms     mgasps=173.707 age=1m49s    triediffs=299.20MiB triedirty=90.68MiB
nov 02 03:17:24 sprmcr geth[20759]: INFO [11-02|03:17:24.353] Chain head was updated                   number=23,708,296 hash=ecccd8..84a944 root=c53013..351962 elapsed=7.117142ms    age=1m49s
nov 02 03:17:37 sprmcr geth[20759]: INFO [11-02|03:17:37.908] Imported new potential chain segment     number=23,708,297 hash=89d4b6..88dd3f blocks=1         txs=170         mgas=39.835  elapsed=155.598ms     mgasps=256.012 age=1m50s    triediffs=298.95MiB triedirty=92.42MiB
nov 02 03:17:38 sprmcr geth[20759]: INFO [11-02|03:17:38.062] Chain head was updated                   number=23,708,297 hash=89d4b6..88dd3f root=90867e..6c92f8 elapsed=14.499938ms   age=1m51s

The age= value stops increasing at 6-6.5 minutes and does not grow further. After restarting Geth, the lag starts increasing smoothly again, from 1 minute up to 6.5 minutes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions