Info: Child (10136) said varnishd: fellow_storage.c:739: fellow_stream_f: Assertion `fs->checkpoint == fs->written' failed.
I have found that the chid is crashing whilst running with a very light traffic load for testing
I have 400G of 1G hugepages that has helped to extend the time before crash to > 10min
the backend store is a zfs pool
The crash log from stdout
2023-10-27T13:49:01.070748285+11:00 stderr F Child launched OK
2023-10-27T13:49:01.074062115+11:00 stderr F Info: Child (10136) said Child starts
2023-10-27T13:49:01.074083841+11:00 stderr F Info: Child (10136) said Loaded -E vext_cache/libvmod_slash.so,tikjufnt.so
2023-10-27T13:49:01.074110616+11:00 stderr F Info: Child (10136) said mmap(377957122048, MAP_HUGETLB | MAP_HUGE_1GB) succeeded
2023-10-27T13:49:01.074126792+11:00 stderr F Info: Child (10136) said fellow: metadata (bitmap) memory: 1499834360 bytes
2023-10-27T13:49:01.074134200+11:00 stderr F Info: Child (10136) said fellow: /cache/varnish: hdr 5 generation 662
2023-10-27T13:49:01.074149190+11:00 stderr F Info: Child (10136) said fellow: metadata (bitmap) memory: 466037744 bytes
2023-10-27T13:49:01.074171088+11:00 stderr F Info: Child (10136) said fellow: region[0 = pend] 2147483648/1048576000
2023-10-27T13:49:01.074187245+11:00 stderr F Info: Child (10136) said fellow: region[1 = active] 3221225472/1048576000
2023-10-27T13:49:01.074187245+11:00 stderr F Info: Child (10136) said fellow: region[2 = empty] 1073741824/1048576000
2023-10-27T13:49:01.074208985+11:00 stderr F Info: Child (10136) said mmap(42949672960, MAP_HUGETLB | MAP_HUGE_1GB) succeeded
2023-10-27T13:49:01.074223188+11:00 stderr F Info: Child (10136) said buddy: metadata (bitmap) memory: 85221920 bytes
2023-10-27T13:49:01.074237242+11:00 stderr F Info: Child (10136) said mmap(1073741824, MAP_HUGETLB | MAP_HUGE_1GB) succeeded
2023-10-27T13:49:01.074251640+11:00 stderr F Info: Child (10136) said buddy: metadata (bitmap) memory: 2133472 bytes
2023-10-27T13:49:19.988209302+11:00 stderr F Info: Child (10136) said varnishd: fellow_storage.c:739: fellow_stream_f: Assertion `fs->checkpoint == fs->written' failed.
2023-10-27T13:49:20.359912587+11:00 stderr F Error: Child (10136) died signal=6 (core dumped)
2023-10-27T13:49:20.360284607+11:00 stderr F Error: Child (10136) Panic at: Fri, 27 Oct 2023 02:49:20 GMT
2023-10-27T13:49:20.360284607+11:00 stderr F Wrong turn at cache/cache_main.c:327:
2023-10-27T13:49:20.360284607+11:00 stderr F Signal 6 (Aborted) received at 0x2798 si_code -6
2023-10-27T13:49:20.360284607+11:00 stderr F version = varnish-7.4.1 revision d5a5aa9cc879320840ca467ddbb7df0f99c9ba0f, vrt api = 18.0
2023-10-27T13:49:20.360284607+11:00 stderr F ident = Linux,5.4.259-1.el8.elrepo.x86_64,x86_64,-jnone,-sfellow,-sbuddy,-sbuddy,-Elibvmod_slash.so,-hcritbit,epoll
2023-10-27T13:49:20.360284607+11:00 stderr F now = 1797.431411 (mono), 1698374959.988224 (real)
2023-10-27T13:49:20.360284607+11:00 stderr F Backtrace:
2023-10-27T13:49:20.360284607+11:00 stderr F 0x444390: varnishd() [0x444390]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x4b8047: varnishd(VAS_Fail+0x17) [0x4b8047]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x43f69e: varnishd() [0x43f69e]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26d06ccf0: /lib64/libpthread.so.0(+0x12cf0) [0x7ff26d06ccf0]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26cce3acf: /lib64/libc.so.6(gsignal+0x10f) [0x7ff26cce3acf]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26ccb6ea5: /lib64/libc.so.6(abort+0x127) [0x7ff26ccb6ea5]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26ccb6d79: /lib64/libc.so.6(+0x21d79) [0x7ff26ccb6d79]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26ccdc426: /lib64/libc.so.6(+0x47426) [0x7ff26ccdc426]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26b396ffd: vext_cache/libvmod_slash.so,tikjufnt.so(+0xcffd) [0x7ff26b396ffd]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26b3d6a6d: vext_cache/libvmod_slash.so,tikjufnt.so(fellow_cache_obj_iter+0x4fd) [0x7ff26b3d6a6d]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26b39cce6: vext_cache/libvmod_slash.so,tikjufnt.so(+0x12ce6) [0x7ff26b39cce6]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x4279a6: varnishd(VDP_DeliverObj+0x66) [0x4279a6]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x46fb62: varnishd(V1D_Deliver+0x2f2) [0x46fb62]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x449dc1: varnishd() [0x449dc1]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x44bd20: varnishd(CNT_Request+0xc0) [0x44bd20]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x4716ff: varnishd() [0x4716ff]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x467093: varnishd() [0x467093]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x467684: varnishd() [0x467684]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26d0621ca: /lib64/libpthread.so.0(+0x81ca) [0x7ff26d0621ca]
2023-10-27T13:49:20.360284607+11:00 stderr F 0x7ff26cccee73: /lib64/libc.so.6(clone+0x43) [0x7ff26cccee73]
2023-10-27T13:49:20.360284607+11:00 stderr F errno = 32 (Broken pipe)