Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.152765592s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:56:24 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 6.15278518s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.690804346s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:56:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.690837419s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
3.358245773s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:55:53 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.358272153s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.594549877s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:55:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 6.594576417s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.916019059s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:55:43 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 4.916037645s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.790995491s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:55:41 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.79105288s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
3.245155485s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:55:28 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.245182986s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.84091519s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:55:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.840946688s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
4.879045092s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:55:05 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 4.879108352s
core
uplink
eu1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
3.372018268s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:54:46 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.372044318s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.011691225s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:54:41 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.011718566s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.197103829s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:54:27 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 6.197131281s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
3.47994799s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:54:26 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.479966586s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.710959225s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:54:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.710980896s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.178943194s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:54:04 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.178964635s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.991615458s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:53:59 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.991655905s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
3.406526521s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:53:27 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.406556939s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.755097117s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:53:19 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.75511414s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.888086472s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:53:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.888107743s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
4.943806579s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:53:14 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 4.943951135s
core
uplink
eu1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
3.582380325s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:52:29 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.582410492s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.386457483s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:52:27 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.386484504s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.116422245s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:52:07 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.116444217s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.79148603s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:52:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.791558719s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.998074361s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:52:05 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.99809962s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
3.27015753s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:52:02 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.270184421s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.038317957s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:51:44 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.038335871s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.236516618s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:51:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 6.236547316s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.06640466s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:51:20 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 6.066435088s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
3.394138423s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:51:04 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.394171876s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.858862756s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:50:58 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.858899697s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
4.874469275s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:50:46 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 4.874494544s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.995922165s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:50:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.995943635s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.88529203s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:50:34 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.885321816s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
3.220433169s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:50:33 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.220463406s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.049109144s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:50:29 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.049137427s
core
uplink
eu1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
3.26064166s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:50:12 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.260668322s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.169858569s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:50:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.169906499s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
8.328767233s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:49:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 8.328796349s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.1026024s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:49:30 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.102620685s
core
uplink
eu1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
3.444182809s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:49:28 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.444207956s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.054030548s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:49:26 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 6.054065685s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
7.501588993s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:49:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 7.501615094s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.429771626s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:49:04 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.429807263s
core
uplink
eu1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
3.615883076s trace:
https://jaeger.dev.storj.io/trace/73d974b074449dbf
FAIL (2025-12-24 01:48:58 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 49)
Duration: 3.615910519s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.165654586s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:48:47 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.165683702s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.410292854s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:48:31 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 6.410320648s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
4.919607054s trace:
https://jaeger.dev.storj.io/trace/75d8387cd2b2a1e4
FAIL (2025-12-24 01:48:17 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 4.919638094s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.747203338s trace:
https://jaeger.dev.storj.io/trace/201ecee6502814db
FAIL (2025-12-24 01:48:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191159121 (441h47m41.062242156s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191218845 (441h47m41.06230189s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191447190 (441h47m41.062530225s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191622874 (441h47m41.062705929s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191707686 (441h47m41.062790721s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.191981498 (441h47m41.063064533s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192015382 (441h47m41.063098427s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192262683 (441h47m41.063345718s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192543688 (441h47m41.063626733s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:41.192963147 (441h47m41.064046192s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Uploading piece...
Dec 24 01:16:42.024151032 (441h47m41.895234067s) [upload=387799]: Next returned nil segment
Dec 24 01:16:42.024165349 (441h47m41.895248394s) [upload=387799]: Waiting for error group managing segments...
Dec 24 01:16:42.414361975 (441h47m42.28544502s) [noise=true , address=185.167.204.206:4068 , piece_id=U75JRQ7YPLBFDP4HOA7ZBOR57Z6DRT5A4TAG63AD76FMBJVG3LOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414493967 (441h47m42.285577012s) [noise=true , address=185.167.204.206:4038 , piece_id=ALOUX6ZHPACYGHPW7X6IR2OERHPBD5CBFENP4BJJWL7CTR4RN62Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.414917273 (441h47m42.286000308s) [noise=true , address=185.167.204.206:4049 , piece_id=G5VPHOCSZELVFQPDEV5AXB2JNMEUEVXJUJ6EL3LKRYUL4RNUX2IA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415249786 (441h47m42.286332821s) [noise=true , address=185.167.204.206:4053 , piece_id=PCUI55ZJARJJDWLFAQWO3WG4GUH4SGTK6CW64GPM5GYXXXDDV6PQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415381217 (441h47m42.286464252s) [noise=true , address=185.167.204.205:4034 , piece_id=QYAEO7ZP2OMAQOWTHG7MSBM6NRTCPXGWWGTG22EVUPOEJCCVIQQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415738687 (441h47m42.286821722s) [noise=true , address=185.167.204.206:4050 , piece_id=I5H2MD3O7MDVV57BUO323AHIWPH3AOMOF2BAIVBIAOQFVOQINZQQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.415760910 (441h47m42.286843955s) [noise=true , address=185.167.204.205:4005 , piece_id=GSGJOPZZMO6UKS25ULTHJK64QJKFFO4PU57DWOOHUVPKFHN7JZMA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416162705 (441h47m42.28724574s) [noise=true , address=185.167.204.205:4008 , piece_id=FIIAIUFFLQO5NTHLLVKWLEOL5TCAXMTGX5QJC3PNDOWZTY2MKTOA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.416404205 (441h47m42.28748724s) [noise=true , address=185.167.204.205:4024 , piece_id=M5XIABNPCI5YJJTQWHV2USSLKJ5G5UXCPYMCFRFDLHYTOKVBJDRA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.418750403 (441h47m42.289833438s) [noise=true , address=185.167.204.205:4027 , piece_id=KLJCEI7JZ2SVORZKTP7EEN2BV7ZV3AZHGNUAB67B4M5QROOV4CJQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.420233638 (441h47m42.291316683s) [noise=true , address=185.167.204.205:4030 , piece_id=OAXPHBMR23B3BSWDQ5LAVMUC5UHLNJGLJCC7CKPKURZFRWUXHQIA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.421648243 (441h47m42.292731278s) [noise=true , address=185.167.204.205:4025 , piece_id=MG3VZ4TZ3LH3G2WXCZA72LM2AMG5YZX3G7E2IJTEI6YIEKEGLTHA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422131203 (441h47m42.293214258s) [noise=true , address=185.167.204.205:4004 , piece_id=PVZFHLNUADL5I35F35JXQHVUEVQ5MYQYWSDHBWHEXWNTKVCEHU2Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.422239188 (441h47m42.293322223s) [noise=true , address=185.167.204.205:4023 , piece_id=GOLRVDLLZTBN4EG4BSFV2GPKQ3V45S5QTNMX5IWC5AUXBX4PYH4A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.423642431 (441h47m42.294725476s) [noise=true , address=185.167.204.206:4065 , piece_id=Z65S7NQUAMXS5UHGE2CIJIN2WBICVA5ENMNOADRZT2G6A7QLMF5Q, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:42.977836857 (441h47m42.848919892s) [noise=true , address=81.171.14.70:4039 , piece_id=NS2OYQYNAKWVOKRMTF4JQGZJWV2EK76MT34SYGVN63TJT3JGCINA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.010653319 (441h47m42.881736354s) [noise=true , address=81.171.14.70:4034 , piece_id=QJATDCDBKBMWV2DTCCP3NG2UMIJKHRLN47ZI7PJCJQLAHN2CKABQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.052454703 (441h47m42.923537738s) [noise=true , address=81.171.14.70:4031 , piece_id=XFJKOSSEAKUCU7374YF53P24FRUJVJLBN26RKCVRPXISLWVR5OMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.057506165 (441h47m42.92858921s) [noise=true , address=81.171.14.70:4026 , piece_id=PXA2UGKHG3HRRGC4SSPIEHGZ4SFMITLQRXPBSXPFGSTJCDLY2OSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.060101427 (441h47m42.931184472s) [noise=true , address=81.171.14.70:4035 , piece_id=65W7PMAJZYEIVXWO35L3EOWFKW5FWJSGQVJCFQ4QJDJ4AFGBMLSQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.554537278 (441h47m43.425620333s) [noise=true , address=81.171.14.70:4025 , piece_id=DSDUDNL6BSZSRGUZ5HFXRVMP54JD3Y64DW6CHT33OYXFIXCDDXPQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.581555537 (441h47m43.452638572s) [noise=true , address=81.171.14.70:4037 , piece_id=PHDI3KVZZ7M375RKHGD7QRU5UMNGDDEA5RGXRHY5APVPASIFJ7SA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.734290976 (441h47m43.605374011s) [noise=true , address=89.149.201.147:4020 , piece_id=DFDY2YGGPHKNPJNA4RJC7W7FDWRHNEVFPZ33GRTWISELVL2DQDDA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:43.737614335 (441h47m43.60869738s) [noise=true , address=89.149.201.147:4014 , piece_id=6VZ4XKLGTVJFYGWYNGJSCSGT7I66627DVURRGLBHMNNCGWFP7YWQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.086804531 (441h47m43.957887566s) [noise=true , address=89.149.201.147:4011 , piece_id=NH6WNCY4XOJBNFUCSLHM5PKKKDKT7OLNLVJQRBRHL7M55ODYHSBA, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113695457 (441h47m43.984778502s) [noise=true , address=89.149.201.147:4013 , piece_id=XRFIPQ2SXU5IGR7AX7Y3XLQUOQ4WIC3T6XVJFUF4Z33AMCONMD6A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: <nil>
Dec 24 01:16:44.113725694 (441h47m43.98480873s) [seg_pos={0 0}, upload=387799]: Segment reached optimal threshold of 26 pieces.
Dec 24 01:16:44.114000067 (441h47m43.985083112s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114010186 (441h47m43.985093231s) [noise=true , address=89.149.201.147:4019 , piece_id=BCQCSVBRXPIJBZRA26HBUBANPDSWY2RNGI7S2QAPN43GCPSSWJMQ, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114382044 (441h47m43.9854651s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 24 01:16:44.114392084 (441h47m43.985475119s) [noise=true , address=89.149.201.147:4024 , piece_id=JXA37L75PMPH6GB5T7D3HIKKTFB7RDESD2JCMQ7NSWFKHSA5R45A, seg_pos={0 0}, upload=387799]: Long tail cancellation due to optimal threshold
Dec 24 01:16:44.114413755 (441h47m43.98549679s) [seg_pos={0 0}, upload=387799]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 24 01:16:44.114498266 (441h47m43.985581321s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 24 01:16:44.114504729 (441h47m43.985587764s) [upload=387799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 24 01:16:44.341827317 (441h47m44.212910352s) [upload=387799]: Done uploading segments...
Dec 24 01:16:44.341848366 (441h47m44.212931411s) [upload=387799]: Upload finished. err: <nil>
Duration: 5.747310703s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.903247901s trace:
https://jaeger.dev.storj.io/trace/4138223ba7f3c1a7
FAIL (2025-12-24 01:48:04 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 52)
Duration: 5.903277096s
core
uplink
us1
|