Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X........X......X..........................] duration: 33.240050658s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:21:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 33.240077299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..XXX........XXX................XX......X....X.X...X........] duration: 3.692635543s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:20:34 UTC) orbiter warning: context deadline exceeded Duration: 3.692682492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X.....X....XXX..X.X.X....XX........X.XX..XX......XX] duration: 30.003557146s trace: https://jaeger.dev.storj.io/trace/36e8031d295fc7dc FAIL (2025-12-31 10:20:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BWYR3KRDC65CEGYLKHWHLDKE5V7M4UNHZDS3JLBN5NWM4I26OVTQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: AYN4RJZLNOMUAY4QSJNI7QAT27S5TRG34S7GXRGC7GOKWEMAPTPA) context deadline exceeded ======LOGS====== Dec 31 10:19:58.819761028 (618h50m58.690844063s) [upload=542312]: Starting upload Dec 31 10:19:58.819830390 (618h50m58.690913426s) [upload=542312]: Done starting upload Dec 31 10:19:58.819865107 (618h50m58.690948162s) [upload=542312]: Uploading segments... Dec 31 10:19:58.820004773 (618h50m58.691087808s) [upload=542312]: Got next segment. Inline: false Dec 31 10:19:58.820030602 (618h50m58.691113647s) [upload=542312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:19:58.820034559 (618h50m58.691117594s) [upload=542312]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:19:59.078475023 (618h50m58.949558058s) [seg_pos={0 0}, upload=542312]: Begin upload segment... Dec 31 10:19:59.078517473 (618h50m58.949600528s) [seg_pos={0 0}, upload=542312]: Done begin upload segment. Dec 31 10:19:59.078554604 (618h50m58.949637649s) [noise=true , address=145.79.227.13:4803 , piece_id=TKDEKP2KMM67AZJ4UIOVCPDYAJGV2YCMRAXVAV63H64ASUJB5ZDQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.078561006 (618h50m58.949644041s) [noise=true , address=145.79.227.39:4015 , piece_id=CBUKLPUMRL2QHQT2OICMXJGPPBHG3FYHY3IXBPZFS22X3OUPNXDA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.078612595 (618h50m58.94969564s) [noise=true , address=145.79.227.16:4910 , piece_id=OYBCSV7FRUTBO6TQQKEZWS534JRMVGIE4NMXKNS3VTOPVYZPARHQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.079132475 (618h50m58.95021551s) [noise=true , address=145.79.227.46:4602 , piece_id=HZB3AB37PMNR3NLMSCKN3ZL3A7A33NLMT3RYDPEHKDTYD7SJIZBQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.079709593 (618h50m58.950792628s) [noise=true , address=45.91.116.13:4153 , piece_id=4NQ24M5CMKYN6SUAT7FMGBGDNOYI4Z6VY4B5AIRRYUS6MGSUBYRA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.080043840 (618h50m58.951126885s) [noise=true , address=145.79.227.43:4157 , piece_id=WTYZSI65XN2T4LHKNPR3WPTQNOKXIQB3M547MICYW55OMHJNYVZQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.080409946 (618h50m58.951492981s) [noise=true , address=45.91.116.12:4225 , piece_id=BWYR3KRDC65CEGYLKHWHLDKE5V7M4UNHZDS3JLBN5NWM4I26OVTQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.080701803 (618h50m58.951784838s) [noise=true , address=145.79.227.18:4007 , piece_id=HA3ELFWD3UKFLSSPGJ6I5RAZEAQS54PFEMAJMACHOL63JBLVQJGQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.081019908 (618h50m58.952102953s) [noise=true , address=45.91.116.10:4081 , piece_id=GK2ZYPBFP24JQZK24HX4DMNRB2Z55EODPO6XYFM52FQK7TBY4XUQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.081413728 (618h50m58.952496763s) [noise=true , address=145.79.227.48:4303 , piece_id=GYCGCZ4EWLSQOLSDBBJHLKDNISRSQRJMFAZ7N5M74QGLPVBHXGJA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.082862106 (618h50m58.953945151s) [noise=true , address=145.79.227.2:4115 , piece_id=46TLR2L4P63IE44WEON5NZS4M2YDRDXCISHIS7WL4CR3JWFEDXMA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.084208971 (618h50m58.955292006s) [noise=true , address=45.91.116.14:4192 , piece_id=AYN4RJZLNOMUAY4QSJNI7QAT27S5TRG34S7GXRGC7GOKWEMAPTPA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.085782287 (618h50m58.956865322s) [noise=true , address=45.91.116.8:4047 , piece_id=JT72P3DFK6KHUPXI6SRJLWQQJYKPDSBF3HFSAOW6K7SDSIF3PVHQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.087234543 (618h50m58.958317588s) [noise=true , address=145.79.227.52:4445 , piece_id=D5AD4UDM6SVXVHIQRMCTXALZMDKWQME42HIZDZL2GRTN7FNNB4QQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.088363963 (618h50m58.959446998s) [noise=true , address=145.79.227.10:4401 , piece_id=TIZSAHYZJNP4OZSJQ4ZXXVPBGO46ONQ7VSNADVO33J6UVOJTRBXA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.088933777 (618h50m58.960016812s) [noise=true , address=145.79.227.49:4337 , piece_id=GKC5X2PJN2HZTRI6QXB6TTJMFJQE5JRKY4AY3USVH2NK4Z5GG6ZA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:20:00.701199701 (618h51m0.572282746s) [upload=542312]: Next returned nil segment Dec 31 10:20:00.701215781 (618h51m0.572298826s) [upload=542312]: Waiting for error group managing segments... Dec 31 10:20:01.068054299 (618h51m0.939137344s) [noise=true , address=145.79.227.52:4445 , piece_id=D5AD4UDM6SVXVHIQRMCTXALZMDKWQME42HIZDZL2GRTN7FNNB4QQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.070404394 (618h51m0.941487429s) [noise=true , address=145.79.227.48:4303 , piece_id=GYCGCZ4EWLSQOLSDBBJHLKDNISRSQRJMFAZ7N5M74QGLPVBHXGJA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.071460184 (618h51m0.942543229s) [noise=true , address=145.79.227.18:4007 , piece_id=HA3ELFWD3UKFLSSPGJ6I5RAZEAQS54PFEMAJMACHOL63JBLVQJGQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.073317681 (618h51m0.944400726s) [noise=true , address=145.79.227.2:4115 , piece_id=46TLR2L4P63IE44WEON5NZS4M2YDRDXCISHIS7WL4CR3JWFEDXMA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.078096982 (618h51m0.949180017s) [noise=true , address=145.79.227.16:4910 , piece_id=OYBCSV7FRUTBO6TQQKEZWS534JRMVGIE4NMXKNS3VTOPVYZPARHQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.084142844 (618h51m0.955225889s) [noise=true , address=145.79.227.46:4602 , piece_id=HZB3AB37PMNR3NLMSCKN3ZL3A7A33NLMT3RYDPEHKDTYD7SJIZBQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.092998256 (618h51m0.964081301s) [noise=true , address=145.79.227.39:4015 , piece_id=CBUKLPUMRL2QHQT2OICMXJGPPBHG3FYHY3IXBPZFS22X3OUPNXDA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.094065108 (618h51m0.965148143s) [noise=true , address=145.79.227.13:4803 , piece_id=TKDEKP2KMM67AZJ4UIOVCPDYAJGV2YCMRAXVAV63H64ASUJB5ZDQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.145771735 (618h51m1.01685478s) [noise=true , address=145.79.227.10:4401 , piece_id=TIZSAHYZJNP4OZSJQ4ZXXVPBGO46ONQ7VSNADVO33J6UVOJTRBXA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.171574517 (618h51m1.042657552s) [noise=true , address=145.79.227.43:4157 , piece_id=WTYZSI65XN2T4LHKNPR3WPTQNOKXIQB3M547MICYW55OMHJNYVZQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.284412900 (618h51m1.155495945s) [noise=true , address=145.79.227.49:4337 , piece_id=GKC5X2PJN2HZTRI6QXB6TTJMFJQE5JRKY4AY3USVH2NK4Z5GG6ZA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.422901513 (618h51m1.293984558s) [noise=true , address=45.91.116.12:4225 , piece_id=BWYR3KRDC65CEGYLKHWHLDKE5V7M4UNHZDS3JLBN5NWM4I26OVTQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.432415329 (618h51m1.303498374s) [noise=true , address=45.91.116.14:4192 , piece_id=AYN4RJZLNOMUAY4QSJNI7QAT27S5TRG34S7GXRGC7GOKWEMAPTPA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.432445657 (618h51m1.303528702s) [seg_pos={0 0}, upload=542312]: Segment reached optimal threshold of 13 pieces. Dec 31 10:20:01.432748964 (618h51m1.303832009s) [noise=true , address=45.91.116.8:4047 , piece_id=JT72P3DFK6KHUPXI6SRJLWQQJYKPDSBF3HFSAOW6K7SDSIF3PVHQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 10:20:01.432760426 (618h51m1.303843471s) [noise=true , address=45.91.116.8:4047 , piece_id=JT72P3DFK6KHUPXI6SRJLWQQJYKPDSBF3HFSAOW6K7SDSIF3PVHQ, seg_pos={0 0}, upload=542312]: Long tail cancellation due to optimal threshold Dec 31 10:20:01.433069074 (618h51m1.304152109s) [noise=true , address=45.91.116.10:4081 , piece_id=GK2ZYPBFP24JQZK24HX4DMNRB2Z55EODPO6XYFM52FQK7TBY4XUQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:59798->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:59798->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59798->45.91.116.10:4081: use of closed network connection Dec 31 10:20:01.433079604 (618h51m1.304162649s) [noise=true , address=45.91.116.10:4081 , piece_id=GK2ZYPBFP24JQZK24HX4DMNRB2Z55EODPO6XYFM52FQK7TBY4XUQ, seg_pos={0 0}, upload=542312]: Long tail cancellation due to optimal threshold Dec 31 10:20:01.433170958 (618h51m1.304254003s) [noise=true , address=45.91.116.13:4153 , piece_id=4NQ24M5CMKYN6SUAT7FMGBGDNOYI4Z6VY4B5AIRRYUS6MGSUBYRA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 10:20:01.433218809 (618h51m1.304301854s) [noise=true , address=45.91.116.13:4153 , piece_id=4NQ24M5CMKYN6SUAT7FMGBGDNOYI4Z6VY4B5AIRRYUS6MGSUBYRA, seg_pos={0 0}, upload=542312]: Long tail cancellation due to optimal threshold Dec 31 10:20:01.433259757 (618h51m1.304342802s) [seg_pos={0 0}, upload=542312]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:20:01.433389514 (618h51m1.304472559s) [upload=542312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:20:01.433398521 (618h51m1.304481566s) [upload=542312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:20:01.726496276 (618h51m1.597579361s) [upload=542312]: Done uploading segments... Dec 31 10:20:01.726526283 (618h51m1.597609328s) [upload=542312]: Upload finished. err: <nil> Duration: 30.003579627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.................X........X......X.........................] duration: 40.66943261s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 40.669450535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...XXX........XXX................XX......X....X.X...X.......] duration: 3.763576874s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:20:01 UTC) orbiter warning: context deadline exceeded Duration: 3.763603083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....XXX........XXX................XX......X....X.X...X......] duration: 3.456373854s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:19:42 UTC) orbiter warning: context deadline exceeded Duration: 3.456403191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.................X........X......X........................] duration: 28.136245498s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 28.136273451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........X.....X....XXX..X.X.X....XX........X.XX..XX......X] duration: 30.003308288s trace: https://jaeger.dev.storj.io/trace/826d0457756b1e8 FAIL (2025-12-31 10:19:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 10:19:02.668824720 (618h50m2.539907775s) [upload=542301]: Starting upload Dec 31 10:19:02.668983041 (618h50m2.540066086s) [upload=542301]: Done starting upload Dec 31 10:19:02.669027395 (618h50m2.54011043s) [upload=542301]: Uploading segments... Dec 31 10:19:02.669316456 (618h50m2.540399522s) [upload=542301]: Got next segment. Inline: false Dec 31 10:19:02.669373314 (618h50m2.540456369s) [upload=542301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:19:02.669380107 (618h50m2.540463152s) [upload=542301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:19:02.918962742 (618h50m2.790045788s) [seg_pos={0 0}, upload=542301]: Begin upload segment... Dec 31 10:19:02.919036252 (618h50m2.790119297s) [seg_pos={0 0}, upload=542301]: Done begin upload segment. Dec 31 10:19:02.919100293 (618h50m2.790183339s) [noise=true , address=77.90.31.17:4770 , piece_id=ZIFXM5R6GSYSCG7TXFFFEQFPLN6WHORPNULUMTCDRO2JZWSED4XQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.919243798 (618h50m2.790326843s) [noise=true , address=145.79.227.19:4078 , piece_id=FJAGRQRTZ7L6KM37XDOXD6C6CGTJIN4GLBU4JC5R6BL4FHKCB4QA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.919909806 (618h50m2.790992851s) [noise=true , address=145.79.227.31:4798 , piece_id=TKMKUF65NFQOOTYZAW6ABHBF2U7IQTAZMOND3SEBL42P6OW6HODA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.920090069 (618h50m2.791173104s) [noise=true , address=145.79.227.40:4051 , piece_id=RMB6AQF4YKPKDCDJVAUYBKBBPRXT767KZ2ALJ2BQFHPOCC3J4FXQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.920461386 (618h50m2.791544421s) [noise=true , address=77.90.31.11:4555 , piece_id=NSE66OWBA3CBWSHBTZTPXXXNBASC2UYEL7Y4ENY6NPFMDK3WJMXA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.920573830 (618h50m2.791656875s) [noise=true , address=145.79.227.20:4438 , piece_id=ZUNUXPI6G25QGCOSTLHFA2PD4MBA4CUITHAOV5ETWN55JN34KDHA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921038885 (618h50m2.79212193s) [noise=true , address=145.79.227.17:4971 , piece_id=25H3ZYHRRZFBUPIF4B44AOQ5BMT7BLILSBESOB2IALXYAVXEC6OA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921239067 (618h50m2.792322132s) [noise=true , address=145.79.227.18:4007 , piece_id=DHHN7O5GGCOCUX4RGR4MT4R572K6N7CLE3DMOE5LNX7O5ISL3PKQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921482981 (618h50m2.792566026s) [noise=true , address=145.79.227.52:4447 , piece_id=OSYRG3C77FN3OEYFACAWA7GKE6NGEKIOI43F3I3SXLDDXWXMGAFA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921877191 (618h50m2.792960226s) [noise=true , address=145.79.227.41:4086 , piece_id=G76WDVBUHMRB2MPGN6XLFSX7UUU4FGNP6SUN5RMSEOKL6WRE2LJQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921922017 (618h50m2.793005053s) [noise=true , address=77.90.31.18:4805 , piece_id=6RCLHIZDZDA74BSEB3PJC3VDF4MDLV5NSN24LAVF2OWPIQS4XDAQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.922412351 (618h50m2.793495396s) [noise=true , address=77.90.31.16:4736 , piece_id=SYH636EFJAWJM327IVXI25TEPC2CUBZ3DMHUS2PALLTGEMFZJEVA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.923331862 (618h50m2.794414907s) [noise=true , address=45.91.116.9:4262 , piece_id=JANELRJQ4ELLBM7F5E7IHCZBGZFRGQHGZZNT3VACSCAFRN42KBRA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.924332477 (618h50m2.795415522s) [noise=true , address=77.90.31.14:4661 , piece_id=4N3QF2SV4SXYPCVRKDUU5MIWNB5XCQLJ2JDI6YAXMX33KULHRVVA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.925226188 (618h50m2.796309233s) [noise=true , address=77.90.31.15:4700 , piece_id=S5OBGPYVRLHBVZH6O6R2NTR2QOXC6RK67CJ6JUGIMAF2CYO2AHLQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.926041160 (618h50m2.797124205s) [noise=true , address=145.79.227.7:4293 , piece_id=2744XVEUVN5DAMD3RV4Q6FT7X6WM3SBHX3SZV2EA3BPDRDHNBPRQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:04.621137329 (618h50m4.492220364s) [upload=542301]: Next returned nil segment Dec 31 10:19:04.621163348 (618h50m4.492246393s) [upload=542301]: Waiting for error group managing segments... Dec 31 10:19:04.905177926 (618h50m4.776260961s) [noise=true , address=145.79.227.31:4798 , piece_id=TKMKUF65NFQOOTYZAW6ABHBF2U7IQTAZMOND3SEBL42P6OW6HODA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.909692923 (618h50m4.780775958s) [noise=true , address=145.79.227.20:4438 , piece_id=ZUNUXPI6G25QGCOSTLHFA2PD4MBA4CUITHAOV5ETWN55JN34KDHA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.918998833 (618h50m4.790081878s) [noise=true , address=145.79.227.19:4078 , piece_id=FJAGRQRTZ7L6KM37XDOXD6C6CGTJIN4GLBU4JC5R6BL4FHKCB4QA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.932095681 (618h50m4.803178716s) [noise=true , address=145.79.227.41:4086 , piece_id=G76WDVBUHMRB2MPGN6XLFSX7UUU4FGNP6SUN5RMSEOKL6WRE2LJQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.934577077 (618h50m4.805660112s) [noise=true , address=145.79.227.7:4293 , piece_id=2744XVEUVN5DAMD3RV4Q6FT7X6WM3SBHX3SZV2EA3BPDRDHNBPRQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.936571775 (618h50m4.80765482s) [noise=true , address=145.79.227.52:4447 , piece_id=OSYRG3C77FN3OEYFACAWA7GKE6NGEKIOI43F3I3SXLDDXWXMGAFA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.989521850 (618h50m4.860604885s) [noise=true , address=145.79.227.17:4971 , piece_id=25H3ZYHRRZFBUPIF4B44AOQ5BMT7BLILSBESOB2IALXYAVXEC6OA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.078679741 (618h50m4.949762786s) [noise=true , address=145.79.227.18:4007 , piece_id=DHHN7O5GGCOCUX4RGR4MT4R572K6N7CLE3DMOE5LNX7O5ISL3PKQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.159365899 (618h50m5.030448944s) [noise=true , address=77.90.31.17:4770 , piece_id=ZIFXM5R6GSYSCG7TXFFFEQFPLN6WHORPNULUMTCDRO2JZWSED4XQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.209984214 (618h50m5.081067259s) [noise=true , address=77.90.31.11:4555 , piece_id=NSE66OWBA3CBWSHBTZTPXXXNBASC2UYEL7Y4ENY6NPFMDK3WJMXA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.340698936 (618h50m5.211781971s) [noise=true , address=45.91.116.9:4262 , piece_id=JANELRJQ4ELLBM7F5E7IHCZBGZFRGQHGZZNT3VACSCAFRN42KBRA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.753148975 (618h50m5.624232021s) [noise=true , address=77.90.31.16:4736 , piece_id=SYH636EFJAWJM327IVXI25TEPC2CUBZ3DMHUS2PALLTGEMFZJEVA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.753904654 (618h50m5.624987699s) [noise=true , address=77.90.31.14:4661 , piece_id=4N3QF2SV4SXYPCVRKDUU5MIWNB5XCQLJ2JDI6YAXMX33KULHRVVA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.753917409 (618h50m5.625000444s) [seg_pos={0 0}, upload=542301]: Segment reached optimal threshold of 13 pieces. Dec 31 10:19:05.754157135 (618h50m5.62524017s) [noise=true , address=77.90.31.15:4700 , piece_id=S5OBGPYVRLHBVZH6O6R2NTR2QOXC6RK67CJ6JUGIMAF2CYO2AHLQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 31 10:19:05.754166132 (618h50m5.625249177s) [noise=true , address=77.90.31.15:4700 , piece_id=S5OBGPYVRLHBVZH6O6R2NTR2QOXC6RK67CJ6JUGIMAF2CYO2AHLQ, seg_pos={0 0}, upload=542301]: Long tail cancellation due to optimal threshold Dec 31 10:19:05.754172955 (618h50m5.625256s) [noise=true , address=77.90.31.18:4805 , piece_id=6RCLHIZDZDA74BSEB3PJC3VDF4MDLV5NSN24LAVF2OWPIQS4XDAQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 10:19:05.754183394 (618h50m5.625266429s) [noise=true , address=77.90.31.18:4805 , piece_id=6RCLHIZDZDA74BSEB3PJC3VDF4MDLV5NSN24LAVF2OWPIQS4XDAQ, seg_pos={0 0}, upload=542301]: Long tail cancellation due to optimal threshold Dec 31 10:19:05.754238329 (618h50m5.625321375s) [noise=true , address=145.79.227.40:4051 , piece_id=RMB6AQF4YKPKDCDJVAUYBKBBPRXT767KZ2ALJ2BQFHPOCC3J4FXQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 10:19:05.754243219 (618h50m5.625326264s) [noise=true , address=145.79.227.40:4051 , piece_id=RMB6AQF4YKPKDCDJVAUYBKBBPRXT767KZ2ALJ2BQFHPOCC3J4FXQ, seg_pos={0 0}, upload=542301]: Long tail cancellation due to optimal threshold Dec 31 10:19:05.754258298 (618h50m5.625341343s) [seg_pos={0 0}, upload=542301]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:19:05.754339061 (618h50m5.625422106s) [upload=542301]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:19:05.754345774 (618h50m5.625428809s) [upload=542301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:19:06.054101826 (618h50m5.925184871s) [upload=542301]: Done uploading segments... Dec 31 10:19:06.054126382 (618h50m5.925209427s) [upload=542301]: Upload finished. err: <nil> Duration: 30.003335971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....XXX........XXX................XX......X....X.X...X.....] duration: 3.569440472s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:18:47 UTC) orbiter warning: context deadline exceeded Duration: 3.569466712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.................X........X......X.......................] duration: 19.779008294s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 19.7790372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.................X........X......X......................] duration: 32.515594833s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:18:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 32.515621935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......XXX........XXX................XX......X....X.X...X....] duration: 3.678475329s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:18:20 UTC) orbiter warning: context deadline exceeded Duration: 3.678503974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..........X.....X....XXX..X.X.X....XX........X.XX..XX......] duration: 5.959309764s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.959338268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..........X.....X....XXX..X.X.X....XX........X.XX..XX.....] duration: 5.924134578s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.924179733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.................X........X......X.....................] duration: 15.332238035s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:17:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 15.332255998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X......XXX........XXX................XX......X....X.X...X...] duration: 3.710066054s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:17:09 UTC) orbiter warning: context deadline exceeded Duration: 3.710123604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X..........X.....X....XXX..X.X.X....XX........X.XX..XX....] duration: 5.681979605s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.682012006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.................X........X......X....................] duration: 33.932640109s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:17:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 33.932670147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X......XXX........XXX................XX......X....X.X...X..] duration: 4.202846596s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:16:53 UTC) orbiter warning: context deadline exceeded Duration: 4.202872145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X..........X.....X....XXX..X.X.X....XX........X.XX..XX...] duration: 5.702254692s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:16:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.702286583s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X......XXX........XXX................XX......X....X.X...X.] duration: 3.497614653s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:16:22 UTC) orbiter warning: context deadline exceeded Duration: 3.497642165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X.................X........X......X...................] duration: 45.286104129s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 45.286120291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X..........X.....X....XXX..X.X.X....XX........X.XX..XX..] duration: 5.630412464s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.630440979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....X.................X........X......X..................] duration: 33.876648613s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:15:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 33.876677908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X..........X.....X....XXX..X.X.X....XX........X.XX..XX.] duration: 5.671951s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:15:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.671981368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X......XXX........XXX................XX......X....X.X...X] duration: 5.000943818s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:15:11 UTC) orbiter warning: context deadline exceeded Duration: 5.000968244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X..........X.....X....XXX..X.X.X....XX........X.XX..XX] duration: 30.002359169s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:14:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 30.002381781s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...X......XXX........XXX................XX......X....X.X...] duration: 3.427532114s trace: https://jaeger.dev.storj.io/trace/5ec8b2b3a43d709b FAIL (2025-12-31 10:14:25 UTC) orbiter warning: context deadline exceeded Duration: 3.427637013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X....X..........X.....X....XXX..X.X.X....XX........X.XX..X] duration: 30.002944764s trace: https://jaeger.dev.storj.io/trace/3d0ffc4fe2427a69 FAIL (2025-12-31 10:14:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: M7VGF7REHHXFI76CFRRTUWMTO5O3KFWB56ECTMZC6FQDGLO2KMGA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: FOWQBHIBH6WIPL425TTSBB7RNIGJQBLLRX54YYV6SWIJ2G2KRB5A) context deadline exceeded ======LOGS====== Dec 31 10:13:41.649585687 (618h44m41.520668722s) [upload=542222]: Starting upload Dec 31 10:13:41.649644207 (618h44m41.520727242s) [upload=542222]: Done starting upload Dec 31 10:13:41.649751793 (618h44m41.520834838s) [upload=542222]: Uploading segments... Dec 31 10:13:41.649851282 (618h44m41.520934317s) [upload=542222]: Got next segment. Inline: false Dec 31 10:13:41.649900977 (618h44m41.520984012s) [upload=542222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:13:41.649906046 (618h44m41.520989082s) [upload=542222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:13:41.895945554 (618h44m41.767028589s) [seg_pos={0 0}, upload=542222]: Begin upload segment... Dec 31 10:13:41.896002483 (618h44m41.767085518s) [seg_pos={0 0}, upload=542222]: Done begin upload segment. Dec 31 10:13:41.896048681 (618h44m41.767131736s) [noise=true , address=145.79.227.11:4729 , piece_id=YCJSQTNXHXV3P672VERNO4XXEDK5FUN4PJNULKKV3WMBDPNVPI6Q, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.896086052 (618h44m41.767169097s) [noise=true , address=45.91.116.13:4154 , piece_id=ATOQNN5NN3HPTOWKYV2IIAGFGWR6NJ5TPRQXHFDLTQQMCK6TAXWQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.896683089 (618h44m41.767766135s) [noise=true , address=145.79.227.41:4085 , piece_id=ZC6ZDMKZP6VBB6RDXTL4D7YDPUG67TOFVU7DPHQDCLP7RMHPQ7EQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.897226524 (618h44m41.768309559s) [noise=true , address=45.91.116.12:4227 , piece_id=4DYGMSP2CURTTAWOV6NGHZ6W4OZIXXJDJTHJXNA2TEISJCQMMWVQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.897543677 (618h44m41.768626723s) [noise=true , address=145.79.227.14:4840 , piece_id=WA4Y5TXRRUUXLLPASLK2K5KCJHTSVJ2XHJASCKQJLZZ7FFYKYIFQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.897892041 (618h44m41.768975076s) [noise=true , address=77.90.31.12:4589 , piece_id=FOWQBHIBH6WIPL425TTSBB7RNIGJQBLLRX54YYV6SWIJ2G2KRB5A, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.897984928 (618h44m41.769067983s) [noise=true , address=145.79.227.27:4691 , piece_id=CT3DAFVV5WEQE6YOTJDDSB32WCR5SW6QTFWBKKQUMS5SHWSSSAGQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.898194487 (618h44m41.769277532s) [noise=true , address=145.79.227.52:4446 , piece_id=NNTTYZUKXS7EZU6YBRLORDZKRDQQ7L2I7HI6RP553N6L4BWRJMNA, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.898538793 (618h44m41.769621828s) [noise=true , address=145.79.227.51:4411 , piece_id=YYX3D3MAFZGDT2JHSX4MRRTLQ6PGPHSQXUN3RQGMPRYFXQJMI43Q, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.898864172 (618h44m41.769947217s) [noise=true , address=145.79.227.6:4257 , piece_id=H2JOISOG3GKFAN4L262EY7MWCHZZFD67YRD5F2WTWVZOBHTQOJ4Q, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.899167460 (618h44m41.770250525s) [noise=true , address=145.79.227.34:4906 , piece_id=OXYCJRXN3WJREK2A4PQUPIVYEEWBZRXQFSJGVINSPRAMEFT2OKMQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.900556956 (618h44m41.771639991s) [noise=true , address=45.91.116.8:4045 , piece_id=M7VGF7REHHXFI76CFRRTUWMTO5O3KFWB56ECTMZC6FQDGLO2KMGA, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.901332693 (618h44m41.772415728s) [noise=true , address=145.79.227.15:4874 , piece_id=OKS3POGZTVPPZFX3CT2FCYRJKH7U77B3LDD7R6AMGX2LC5TRXTNA, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.903014365 (618h44m41.774097411s) [noise=true , address=145.79.227.38:4978 , piece_id=MUXX5UTFENVK635JRJFSCPG2C6YGARYC67YKZGATJIARIGC264CQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.903948413 (618h44m41.775031448s) [noise=true , address=145.79.227.9:4366 , piece_id=67OGS4KR7CZ2XDFLAN4WHNZE6C3UZYVEN64ZL6H2NLX4YYALQEGA, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:41.905182123 (618h44m41.776265158s) [noise=true , address=145.79.227.22:4511 , piece_id=IFBFOTDFRZMOK4G4XEKP3VINOXK3IL3ZB57TGW4M6M6JOAP562TQ, seg_pos={0 0}, upload=542222]: Uploading piece... Dec 31 10:13:43.503042848 (618h44m43.374125883s) [upload=542222]: Next returned nil segment Dec 31 10:13:43.503063067 (618h44m43.374146102s) [upload=542222]: Waiting for error group managing segments... Dec 31 10:13:43.885838992 (618h44m43.756922037s) [noise=true , address=145.79.227.41:4085 , piece_id=ZC6ZDMKZP6VBB6RDXTL4D7YDPUG67TOFVU7DPHQDCLP7RMHPQ7EQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.886900053 (618h44m43.757983088s) [noise=true , address=145.79.227.11:4729 , piece_id=YCJSQTNXHXV3P672VERNO4XXEDK5FUN4PJNULKKV3WMBDPNVPI6Q, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.887243447 (618h44m43.758326492s) [noise=true , address=145.79.227.14:4840 , piece_id=WA4Y5TXRRUUXLLPASLK2K5KCJHTSVJ2XHJASCKQJLZZ7FFYKYIFQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.891859045 (618h44m43.76294208s) [noise=true , address=145.79.227.22:4511 , piece_id=IFBFOTDFRZMOK4G4XEKP3VINOXK3IL3ZB57TGW4M6M6JOAP562TQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.893385792 (618h44m43.764468827s) [noise=true , address=145.79.227.51:4411 , piece_id=YYX3D3MAFZGDT2JHSX4MRRTLQ6PGPHSQXUN3RQGMPRYFXQJMI43Q, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.898447200 (618h44m43.769530255s) [noise=true , address=145.79.227.27:4691 , piece_id=CT3DAFVV5WEQE6YOTJDDSB32WCR5SW6QTFWBKKQUMS5SHWSSSAGQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.911103400 (618h44m43.782186435s) [noise=true , address=145.79.227.52:4446 , piece_id=NNTTYZUKXS7EZU6YBRLORDZKRDQQ7L2I7HI6RP553N6L4BWRJMNA, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.914634603 (618h44m43.785717638s) [noise=true , address=145.79.227.15:4874 , piece_id=OKS3POGZTVPPZFX3CT2FCYRJKH7U77B3LDD7R6AMGX2LC5TRXTNA, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.918000343 (618h44m43.789083378s) [noise=true , address=145.79.227.38:4978 , piece_id=MUXX5UTFENVK635JRJFSCPG2C6YGARYC67YKZGATJIARIGC264CQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.922887458 (618h44m43.793970493s) [noise=true , address=145.79.227.6:4257 , piece_id=H2JOISOG3GKFAN4L262EY7MWCHZZFD67YRD5F2WTWVZOBHTQOJ4Q, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:43.983426945 (618h44m43.85450999s) [noise=true , address=145.79.227.9:4366 , piece_id=67OGS4KR7CZ2XDFLAN4WHNZE6C3UZYVEN64ZL6H2NLX4YYALQEGA, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:44.081599881 (618h44m43.952682926s) [noise=true , address=77.90.31.12:4589 , piece_id=FOWQBHIBH6WIPL425TTSBB7RNIGJQBLLRX54YYV6SWIJ2G2KRB5A, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:44.201203085 (618h44m44.07228613s) [noise=true , address=45.91.116.8:4045 , piece_id=M7VGF7REHHXFI76CFRRTUWMTO5O3KFWB56ECTMZC6FQDGLO2KMGA, seg_pos={0 0}, upload=542222]: Done uploading piece. err: <nil> Dec 31 10:13:44.201229344 (618h44m44.072312399s) [seg_pos={0 0}, upload=542222]: Segment reached optimal threshold of 13 pieces. Dec 31 10:13:44.201476695 (618h44m44.07255974s) [noise=true , address=45.91.116.12:4227 , piece_id=4DYGMSP2CURTTAWOV6NGHZ6W4OZIXXJDJTHJXNA2TEISJCQMMWVQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 10:13:44.201487396 (618h44m44.072570451s) [noise=true , address=45.91.116.12:4227 , piece_id=4DYGMSP2CURTTAWOV6NGHZ6W4OZIXXJDJTHJXNA2TEISJCQMMWVQ, seg_pos={0 0}, upload=542222]: Long tail cancellation due to optimal threshold Dec 31 10:13:44.201645286 (618h44m44.072728331s) [noise=true , address=45.91.116.13:4154 , piece_id=ATOQNN5NN3HPTOWKYV2IIAGFGWR6NJ5TPRQXHFDLTQQMCK6TAXWQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 10:13:44.201653271 (618h44m44.072736306s) [noise=true , address=45.91.116.13:4154 , piece_id=ATOQNN5NN3HPTOWKYV2IIAGFGWR6NJ5TPRQXHFDLTQQMCK6TAXWQ, seg_pos={0 0}, upload=542222]: Long tail cancellation due to optimal threshold Dec 31 10:13:44.201860256 (618h44m44.072943301s) [noise=true , address=145.79.227.34:4906 , piece_id=OXYCJRXN3WJREK2A4PQUPIVYEEWBZRXQFSJGVINSPRAMEFT2OKMQ, seg_pos={0 0}, upload=542222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 10:13:44.201908999 (618h44m44.072992034s) [noise=true , address=145.79.227.34:4906 , piece_id=OXYCJRXN3WJREK2A4PQUPIVYEEWBZRXQFSJGVINSPRAMEFT2OKMQ, seg_pos={0 0}, upload=542222]: Long tail cancellation due to optimal threshold Dec 31 10:13:44.201940329 (618h44m44.073023374s) [seg_pos={0 0}, upload=542222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:13:44.202122135 (618h44m44.07320518s) [upload=542222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:13:44.202131162 (618h44m44.073214197s) [upload=542222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:13:44.494028329 (618h44m44.365111374s) [upload=542222]: Done uploading segments... Dec 31 10:13:44.494052034 (618h44m44.365135069s) [upload=542222]: Upload finished. err: <nil> Duration: 30.002958399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.....X.................X........X......X.................] duration: 26.506371922s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:14:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 26.506400146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X...X......XXX........XXX................XX......X....X.X..] duration: 3.562310546s trace: https://jaeger.dev.storj.io/trace/5ec8b2b3a43d709b FAIL (2025-12-31 10:13:54 UTC) orbiter warning: context deadline exceeded Duration: 3.56233425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X....X..........X.....X....XXX..X.X.X....XX........X.XX..] duration: 5.793144576s trace: https://jaeger.dev.storj.io/trace/18c50d4fa04e53a3 FAIL (2025-12-31 10:13:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q) context deadline exceeded ======LOGS====== Dec 31 10:11:48.372645618 (618h42m48.243728663s) [upload=542192]: Starting upload Dec 31 10:11:48.372742493 (618h42m48.243825548s) [upload=542192]: Done starting upload Dec 31 10:11:48.372859154 (618h42m48.243942199s) [upload=542192]: Uploading segments... Dec 31 10:11:48.372976579 (618h42m48.244059624s) [upload=542192]: Got next segment. Inline: false Dec 31 10:11:48.373027265 (618h42m48.2441103s) [upload=542192]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:11:48.373039898 (618h42m48.244122943s) [upload=542192]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:11:48.620004308 (618h42m48.491087343s) [seg_pos={0 0}, upload=542192]: Begin upload segment... Dec 31 10:11:48.620063180 (618h42m48.491146215s) [seg_pos={0 0}, upload=542192]: Done begin upload segment. Dec 31 10:11:48.620097445 (618h42m48.491180491s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.620187005 (618h42m48.49127005s) [noise=true , address=145.79.227.51:4409 , piece_id=LDVTUBSZL4UEM6Q3GCZGPM2TSDCHDDCYHLUHFIYC6GQYDAOQ6ZXQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.620951862 (618h42m48.492034907s) [noise=true , address=145.79.227.11:4730 , piece_id=V566QJNBVMZQLTESD4GLMSL3XGNAGIOTQDNE6YVQ54QCJLMVWDUA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.621413070 (618h42m48.492496105s) [noise=true , address=45.91.116.5:4300 , piece_id=26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622076413 (618h42m48.493159448s) [noise=true , address=77.90.31.19:4842 , piece_id=U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622253310 (618h42m48.493336355s) [noise=true , address=145.79.227.16:4912 , piece_id=OT24IWGLVH56ELDLPIH62VJGBVZDLVRM22YMGV7LUBFFIB6NHJVA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622766467 (618h42m48.493849512s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622938685 (618h42m48.49402172s) [noise=true , address=45.91.116.8:4045 , piece_id=DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623232826 (618h42m48.494315871s) [noise=true , address=145.79.227.31:4797 , piece_id=NI52CYV3GCTMRLGN2I4DVD6HXDX2JTTH6XTOEZDPULLX3MN2NYPA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623722097 (618h42m48.494805132s) [noise=true , address=145.79.227.8:4330 , piece_id=CL3DYBRMGTLTQUQSBCVTKJLKBQ5QIABQPTCNJ6ASZSSEFLHUYIXA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623772793 (618h42m48.494855838s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.624264830 (618h42m48.495347875s) [noise=true , address=145.79.227.36:4519 , piece_id=QKWVR2BYBJR4IHWA7BH42EYJGAEZB74HOICGXMJDK3DEYBYIGK6A, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.624498666 (618h42m48.495581711s) [noise=true , address=77.90.31.14:4661 , piece_id=YZSBCT53RCKRAHYMLUUZMEC3ZVMP72PODVYQNJAM2MG2352RP5NA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.625329859 (618h42m48.496412894s) [noise=true , address=145.79.227.29:4941 , piece_id=HOMXJPZ33HJUVMKFWOMPE3ZAWBQFOXBTRKTCV23ZY7O4ROXH5OAQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.626056282 (618h42m48.497139327s) [noise=true , address=145.79.227.2:4115 , piece_id=2UBA4XIQWI2KKHWHZOCNSY63UEBBA4TD4V6UZDTBV7Z2VN2NRFFA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.627411412 (618h42m48.498494457s) [noise=true , address=145.79.227.22:4511 , piece_id=6XMQZRW4LCY4MYTSD4KHIT5FHR5ZATCJQ5XIIYDSXHKE5DNZSV3Q, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:50.325992785 (618h42m50.19707583s) [upload=542192]: Next returned nil segment Dec 31 10:11:50.326034615 (618h42m50.19711766s) [upload=542192]: Waiting for error group managing segments... Dec 31 10:11:50.607738333 (618h42m50.478821368s) [noise=true , address=145.79.227.16:4912 , piece_id=OT24IWGLVH56ELDLPIH62VJGBVZDLVRM22YMGV7LUBFFIB6NHJVA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.610891767 (618h42m50.481974813s) [noise=true , address=145.79.227.11:4730 , piece_id=V566QJNBVMZQLTESD4GLMSL3XGNAGIOTQDNE6YVQ54QCJLMVWDUA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.613786389 (618h42m50.484869524s) [noise=true , address=145.79.227.36:4519 , piece_id=QKWVR2BYBJR4IHWA7BH42EYJGAEZB74HOICGXMJDK3DEYBYIGK6A, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.618059055 (618h42m50.4891421s) [noise=true , address=145.79.227.29:4941 , piece_id=HOMXJPZ33HJUVMKFWOMPE3ZAWBQFOXBTRKTCV23ZY7O4ROXH5OAQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.621762727 (618h42m50.492845772s) [noise=true , address=145.79.227.31:4797 , piece_id=NI52CYV3GCTMRLGN2I4DVD6HXDX2JTTH6XTOEZDPULLX3MN2NYPA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.622536400 (618h42m50.493619435s) [noise=true , address=145.79.227.22:4511 , piece_id=6XMQZRW4LCY4MYTSD4KHIT5FHR5ZATCJQ5XIIYDSXHKE5DNZSV3Q, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.635169286 (618h42m50.506252341s) [noise=true , address=145.79.227.2:4115 , piece_id=2UBA4XIQWI2KKHWHZOCNSY63UEBBA4TD4V6UZDTBV7Z2VN2NRFFA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.695934673 (618h42m50.567017708s) [noise=true , address=145.79.227.51:4409 , piece_id=LDVTUBSZL4UEM6Q3GCZGPM2TSDCHDDCYHLUHFIYC6GQYDAOQ6ZXQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.698959032 (618h42m50.570042077s) [noise=true , address=145.79.227.8:4330 , piece_id=CL3DYBRMGTLTQUQSBCVTKJLKBQ5QIABQPTCNJ6ASZSSEFLHUYIXA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.785127350 (618h42m50.656210385s) [noise=true , address=77.90.31.14:4661 , piece_id=YZSBCT53RCKRAHYMLUUZMEC3ZVMP72PODVYQNJAM2MG2352RP5NA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.912280288 (618h42m50.783363323s) [noise=true , address=77.90.31.19:4842 , piece_id=U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.947858104 (618h42m50.818941149s) [noise=true , address=45.91.116.5:4300 , piece_id=26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.996380439 (618h42m50.867463474s) [noise=true , address=45.91.116.8:4045 , piece_id=DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.996405926 (618h42m50.867488961s) [seg_pos={0 0}, upload=542192]: Segment reached optimal threshold of 13 pieces. Dec 31 10:11:50.996654199 (618h42m50.867737254s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 10:11:50.996664399 (618h42m50.867747444s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996695398 (618h42m50.867778443s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 10:11:50.996704395 (618h42m50.86778744s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996920927 (618h42m50.868003962s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection Dec 31 10:11:50.996927680 (618h42m50.868010715s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996943240 (618h42m50.868026285s) [seg_pos={0 0}, upload=542192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:11:50.997104356 (618h42m50.868187401s) [upload=542192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:11:50.997111239 (618h42m50.868194284s) [upload=542192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:11:51.285774760 (618h42m51.156857825s) [upload=542192]: Done uploading segments... Dec 31 10:11:51.285830787 (618h42m51.156913832s) [upload=542192]: Upload finished. err: <nil> Duration: 5.793171447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.....X.................X........X......X................] duration: 14.480012081s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:12:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 14.480053359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X...X......XXX........XXX................XX......X....X.X.] duration: 3.39649163s trace: https://jaeger.dev.storj.io/trace/5ec8b2b3a43d709b FAIL (2025-12-31 10:12:49 UTC) orbiter warning: context deadline exceeded Duration: 3.396524282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X....X..........X.....X....XXX..X.X.X....XX........X.XX.] duration: 5.716942551s trace: https://jaeger.dev.storj.io/trace/18c50d4fa04e53a3 FAIL (2025-12-31 10:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q) context deadline exceeded ======LOGS====== Dec 31 10:11:48.372645618 (618h42m48.243728663s) [upload=542192]: Starting upload Dec 31 10:11:48.372742493 (618h42m48.243825548s) [upload=542192]: Done starting upload Dec 31 10:11:48.372859154 (618h42m48.243942199s) [upload=542192]: Uploading segments... Dec 31 10:11:48.372976579 (618h42m48.244059624s) [upload=542192]: Got next segment. Inline: false Dec 31 10:11:48.373027265 (618h42m48.2441103s) [upload=542192]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:11:48.373039898 (618h42m48.244122943s) [upload=542192]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:11:48.620004308 (618h42m48.491087343s) [seg_pos={0 0}, upload=542192]: Begin upload segment... Dec 31 10:11:48.620063180 (618h42m48.491146215s) [seg_pos={0 0}, upload=542192]: Done begin upload segment. Dec 31 10:11:48.620097445 (618h42m48.491180491s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.620187005 (618h42m48.49127005s) [noise=true , address=145.79.227.51:4409 , piece_id=LDVTUBSZL4UEM6Q3GCZGPM2TSDCHDDCYHLUHFIYC6GQYDAOQ6ZXQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.620951862 (618h42m48.492034907s) [noise=true , address=145.79.227.11:4730 , piece_id=V566QJNBVMZQLTESD4GLMSL3XGNAGIOTQDNE6YVQ54QCJLMVWDUA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.621413070 (618h42m48.492496105s) [noise=true , address=45.91.116.5:4300 , piece_id=26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622076413 (618h42m48.493159448s) [noise=true , address=77.90.31.19:4842 , piece_id=U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622253310 (618h42m48.493336355s) [noise=true , address=145.79.227.16:4912 , piece_id=OT24IWGLVH56ELDLPIH62VJGBVZDLVRM22YMGV7LUBFFIB6NHJVA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622766467 (618h42m48.493849512s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622938685 (618h42m48.49402172s) [noise=true , address=45.91.116.8:4045 , piece_id=DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623232826 (618h42m48.494315871s) [noise=true , address=145.79.227.31:4797 , piece_id=NI52CYV3GCTMRLGN2I4DVD6HXDX2JTTH6XTOEZDPULLX3MN2NYPA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623722097 (618h42m48.494805132s) [noise=true , address=145.79.227.8:4330 , piece_id=CL3DYBRMGTLTQUQSBCVTKJLKBQ5QIABQPTCNJ6ASZSSEFLHUYIXA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623772793 (618h42m48.494855838s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.624264830 (618h42m48.495347875s) [noise=true , address=145.79.227.36:4519 , piece_id=QKWVR2BYBJR4IHWA7BH42EYJGAEZB74HOICGXMJDK3DEYBYIGK6A, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.624498666 (618h42m48.495581711s) [noise=true , address=77.90.31.14:4661 , piece_id=YZSBCT53RCKRAHYMLUUZMEC3ZVMP72PODVYQNJAM2MG2352RP5NA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.625329859 (618h42m48.496412894s) [noise=true , address=145.79.227.29:4941 , piece_id=HOMXJPZ33HJUVMKFWOMPE3ZAWBQFOXBTRKTCV23ZY7O4ROXH5OAQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.626056282 (618h42m48.497139327s) [noise=true , address=145.79.227.2:4115 , piece_id=2UBA4XIQWI2KKHWHZOCNSY63UEBBA4TD4V6UZDTBV7Z2VN2NRFFA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.627411412 (618h42m48.498494457s) [noise=true , address=145.79.227.22:4511 , piece_id=6XMQZRW4LCY4MYTSD4KHIT5FHR5ZATCJQ5XIIYDSXHKE5DNZSV3Q, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:50.325992785 (618h42m50.19707583s) [upload=542192]: Next returned nil segment Dec 31 10:11:50.326034615 (618h42m50.19711766s) [upload=542192]: Waiting for error group managing segments... Dec 31 10:11:50.607738333 (618h42m50.478821368s) [noise=true , address=145.79.227.16:4912 , piece_id=OT24IWGLVH56ELDLPIH62VJGBVZDLVRM22YMGV7LUBFFIB6NHJVA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.610891767 (618h42m50.481974813s) [noise=true , address=145.79.227.11:4730 , piece_id=V566QJNBVMZQLTESD4GLMSL3XGNAGIOTQDNE6YVQ54QCJLMVWDUA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.613786389 (618h42m50.484869524s) [noise=true , address=145.79.227.36:4519 , piece_id=QKWVR2BYBJR4IHWA7BH42EYJGAEZB74HOICGXMJDK3DEYBYIGK6A, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.618059055 (618h42m50.4891421s) [noise=true , address=145.79.227.29:4941 , piece_id=HOMXJPZ33HJUVMKFWOMPE3ZAWBQFOXBTRKTCV23ZY7O4ROXH5OAQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.621762727 (618h42m50.492845772s) [noise=true , address=145.79.227.31:4797 , piece_id=NI52CYV3GCTMRLGN2I4DVD6HXDX2JTTH6XTOEZDPULLX3MN2NYPA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.622536400 (618h42m50.493619435s) [noise=true , address=145.79.227.22:4511 , piece_id=6XMQZRW4LCY4MYTSD4KHIT5FHR5ZATCJQ5XIIYDSXHKE5DNZSV3Q, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.635169286 (618h42m50.506252341s) [noise=true , address=145.79.227.2:4115 , piece_id=2UBA4XIQWI2KKHWHZOCNSY63UEBBA4TD4V6UZDTBV7Z2VN2NRFFA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.695934673 (618h42m50.567017708s) [noise=true , address=145.79.227.51:4409 , piece_id=LDVTUBSZL4UEM6Q3GCZGPM2TSDCHDDCYHLUHFIYC6GQYDAOQ6ZXQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.698959032 (618h42m50.570042077s) [noise=true , address=145.79.227.8:4330 , piece_id=CL3DYBRMGTLTQUQSBCVTKJLKBQ5QIABQPTCNJ6ASZSSEFLHUYIXA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.785127350 (618h42m50.656210385s) [noise=true , address=77.90.31.14:4661 , piece_id=YZSBCT53RCKRAHYMLUUZMEC3ZVMP72PODVYQNJAM2MG2352RP5NA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.912280288 (618h42m50.783363323s) [noise=true , address=77.90.31.19:4842 , piece_id=U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.947858104 (618h42m50.818941149s) [noise=true , address=45.91.116.5:4300 , piece_id=26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.996380439 (618h42m50.867463474s) [noise=true , address=45.91.116.8:4045 , piece_id=DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.996405926 (618h42m50.867488961s) [seg_pos={0 0}, upload=542192]: Segment reached optimal threshold of 13 pieces. Dec 31 10:11:50.996654199 (618h42m50.867737254s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 10:11:50.996664399 (618h42m50.867747444s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996695398 (618h42m50.867778443s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 10:11:50.996704395 (618h42m50.86778744s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996920927 (618h42m50.868003962s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection Dec 31 10:11:50.996927680 (618h42m50.868010715s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996943240 (618h42m50.868026285s) [seg_pos={0 0}, upload=542192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:11:50.997104356 (618h42m50.868187401s) [upload=542192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:11:50.997111239 (618h42m50.868194284s) [upload=542192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:11:51.285774760 (618h42m51.156857825s) [upload=542192]: Done uploading segments... Dec 31 10:11:51.285830787 (618h42m51.156913832s) [upload=542192]: Upload finished. err: <nil> Duration: 5.716970796s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X...X......XXX........XXX................XX......X....X.X] duration: 5.001043738s trace: https://jaeger.dev.storj.io/trace/5ec8b2b3a43d709b FAIL (2025-12-31 10:12:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001059478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..X....X..........X.....X....XXX..X.X.X....XX........X.XX] duration: 30.003709966s trace: https://jaeger.dev.storj.io/trace/18c50d4fa04e53a3 FAIL (2025-12-31 10:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q) context deadline exceeded ======LOGS====== Dec 31 10:11:48.372645618 (618h42m48.243728663s) [upload=542192]: Starting upload Dec 31 10:11:48.372742493 (618h42m48.243825548s) [upload=542192]: Done starting upload Dec 31 10:11:48.372859154 (618h42m48.243942199s) [upload=542192]: Uploading segments... Dec 31 10:11:48.372976579 (618h42m48.244059624s) [upload=542192]: Got next segment. Inline: false Dec 31 10:11:48.373027265 (618h42m48.2441103s) [upload=542192]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:11:48.373039898 (618h42m48.244122943s) [upload=542192]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:11:48.620004308 (618h42m48.491087343s) [seg_pos={0 0}, upload=542192]: Begin upload segment... Dec 31 10:11:48.620063180 (618h42m48.491146215s) [seg_pos={0 0}, upload=542192]: Done begin upload segment. Dec 31 10:11:48.620097445 (618h42m48.491180491s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.620187005 (618h42m48.49127005s) [noise=true , address=145.79.227.51:4409 , piece_id=LDVTUBSZL4UEM6Q3GCZGPM2TSDCHDDCYHLUHFIYC6GQYDAOQ6ZXQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.620951862 (618h42m48.492034907s) [noise=true , address=145.79.227.11:4730 , piece_id=V566QJNBVMZQLTESD4GLMSL3XGNAGIOTQDNE6YVQ54QCJLMVWDUA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.621413070 (618h42m48.492496105s) [noise=true , address=45.91.116.5:4300 , piece_id=26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622076413 (618h42m48.493159448s) [noise=true , address=77.90.31.19:4842 , piece_id=U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622253310 (618h42m48.493336355s) [noise=true , address=145.79.227.16:4912 , piece_id=OT24IWGLVH56ELDLPIH62VJGBVZDLVRM22YMGV7LUBFFIB6NHJVA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622766467 (618h42m48.493849512s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.622938685 (618h42m48.49402172s) [noise=true , address=45.91.116.8:4045 , piece_id=DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623232826 (618h42m48.494315871s) [noise=true , address=145.79.227.31:4797 , piece_id=NI52CYV3GCTMRLGN2I4DVD6HXDX2JTTH6XTOEZDPULLX3MN2NYPA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623722097 (618h42m48.494805132s) [noise=true , address=145.79.227.8:4330 , piece_id=CL3DYBRMGTLTQUQSBCVTKJLKBQ5QIABQPTCNJ6ASZSSEFLHUYIXA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.623772793 (618h42m48.494855838s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.624264830 (618h42m48.495347875s) [noise=true , address=145.79.227.36:4519 , piece_id=QKWVR2BYBJR4IHWA7BH42EYJGAEZB74HOICGXMJDK3DEYBYIGK6A, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.624498666 (618h42m48.495581711s) [noise=true , address=77.90.31.14:4661 , piece_id=YZSBCT53RCKRAHYMLUUZMEC3ZVMP72PODVYQNJAM2MG2352RP5NA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.625329859 (618h42m48.496412894s) [noise=true , address=145.79.227.29:4941 , piece_id=HOMXJPZ33HJUVMKFWOMPE3ZAWBQFOXBTRKTCV23ZY7O4ROXH5OAQ, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.626056282 (618h42m48.497139327s) [noise=true , address=145.79.227.2:4115 , piece_id=2UBA4XIQWI2KKHWHZOCNSY63UEBBA4TD4V6UZDTBV7Z2VN2NRFFA, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:48.627411412 (618h42m48.498494457s) [noise=true , address=145.79.227.22:4511 , piece_id=6XMQZRW4LCY4MYTSD4KHIT5FHR5ZATCJQ5XIIYDSXHKE5DNZSV3Q, seg_pos={0 0}, upload=542192]: Uploading piece... Dec 31 10:11:50.325992785 (618h42m50.19707583s) [upload=542192]: Next returned nil segment Dec 31 10:11:50.326034615 (618h42m50.19711766s) [upload=542192]: Waiting for error group managing segments... Dec 31 10:11:50.607738333 (618h42m50.478821368s) [noise=true , address=145.79.227.16:4912 , piece_id=OT24IWGLVH56ELDLPIH62VJGBVZDLVRM22YMGV7LUBFFIB6NHJVA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.610891767 (618h42m50.481974813s) [noise=true , address=145.79.227.11:4730 , piece_id=V566QJNBVMZQLTESD4GLMSL3XGNAGIOTQDNE6YVQ54QCJLMVWDUA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.613786389 (618h42m50.484869524s) [noise=true , address=145.79.227.36:4519 , piece_id=QKWVR2BYBJR4IHWA7BH42EYJGAEZB74HOICGXMJDK3DEYBYIGK6A, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.618059055 (618h42m50.4891421s) [noise=true , address=145.79.227.29:4941 , piece_id=HOMXJPZ33HJUVMKFWOMPE3ZAWBQFOXBTRKTCV23ZY7O4ROXH5OAQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.621762727 (618h42m50.492845772s) [noise=true , address=145.79.227.31:4797 , piece_id=NI52CYV3GCTMRLGN2I4DVD6HXDX2JTTH6XTOEZDPULLX3MN2NYPA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.622536400 (618h42m50.493619435s) [noise=true , address=145.79.227.22:4511 , piece_id=6XMQZRW4LCY4MYTSD4KHIT5FHR5ZATCJQ5XIIYDSXHKE5DNZSV3Q, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.635169286 (618h42m50.506252341s) [noise=true , address=145.79.227.2:4115 , piece_id=2UBA4XIQWI2KKHWHZOCNSY63UEBBA4TD4V6UZDTBV7Z2VN2NRFFA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.695934673 (618h42m50.567017708s) [noise=true , address=145.79.227.51:4409 , piece_id=LDVTUBSZL4UEM6Q3GCZGPM2TSDCHDDCYHLUHFIYC6GQYDAOQ6ZXQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.698959032 (618h42m50.570042077s) [noise=true , address=145.79.227.8:4330 , piece_id=CL3DYBRMGTLTQUQSBCVTKJLKBQ5QIABQPTCNJ6ASZSSEFLHUYIXA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.785127350 (618h42m50.656210385s) [noise=true , address=77.90.31.14:4661 , piece_id=YZSBCT53RCKRAHYMLUUZMEC3ZVMP72PODVYQNJAM2MG2352RP5NA, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.912280288 (618h42m50.783363323s) [noise=true , address=77.90.31.19:4842 , piece_id=U2SHXRYW36XR3VYDUJVHB3MLR5MYFPHPW3M4DLVX2LH7X755BPYQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.947858104 (618h42m50.818941149s) [noise=true , address=45.91.116.5:4300 , piece_id=26AE6QYIL4ULL5VFSWYFKOW7TJHNRPN2T7ERVFVK2E5BO7RZERZQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.996380439 (618h42m50.867463474s) [noise=true , address=45.91.116.8:4045 , piece_id=DRTHJC62KMWCFLLUSL3DPMCPB2WMDVCXHD5SU6PF2SJLPEWGQB4Q, seg_pos={0 0}, upload=542192]: Done uploading piece. err: <nil> Dec 31 10:11:50.996405926 (618h42m50.867488961s) [seg_pos={0 0}, upload=542192]: Segment reached optimal threshold of 13 pieces. Dec 31 10:11:50.996654199 (618h42m50.867737254s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 10:11:50.996664399 (618h42m50.867747444s) [noise=true , address=45.91.116.13:4156 , piece_id=3OPYB4OK5OLS6SUOLOG524GELVUFDQ6KMHNSD2HZ5LQFHP3PQN2A, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996695398 (618h42m50.867778443s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 10:11:50.996704395 (618h42m50.86778744s) [noise=true , address=145.79.227.34:4906 , piece_id=DVWE6GOMJTJE4MVD25JPHNC623PCFTB4OHIY27KDL7H6FSKGDEGQ, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996920927 (618h42m50.868003962s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55882->77.90.31.10:4517: use of closed network connection Dec 31 10:11:50.996927680 (618h42m50.868010715s) [noise=true , address=77.90.31.10:4517 , piece_id=2RUM764ZCCH5EYYX2JN3MWVPXMIK4OXMUA53EJU6OA5OV4SWJKTQ, seg_pos={0 0}, upload=542192]: Long tail cancellation due to optimal threshold Dec 31 10:11:50.996943240 (618h42m50.868026285s) [seg_pos={0 0}, upload=542192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:11:50.997104356 (618h42m50.868187401s) [upload=542192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:11:50.997111239 (618h42m50.868194284s) [upload=542192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:11:51.285774760 (618h42m51.156857825s) [upload=542192]: Done uploading segments... Dec 31 10:11:51.285830787 (618h42m51.156913832s) [upload=542192]: Upload finished. err: <nil> Duration: 30.003733741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.....X.................X........X......X...............] duration: 23.85233758s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:11:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 23.8523681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...X...X......XXX........XXX................XX......X....X.] duration: 3.526363336s trace: https://jaeger.dev.storj.io/trace/5e8858832f68bf84 FAIL (2025-12-31 10:11:34 UTC) orbiter warning: context deadline exceeded Duration: 3.526390377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XX..X....X..........X.....X....XXX..X.X.X....XX........X.X] duration: 30.002837802s trace: https://jaeger.dev.storj.io/trace/51139cfb41ca69ff FAIL (2025-12-31 10:11:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 10:10:35.629115006 (618h41m35.500198041s) [upload=542176]: Starting upload Dec 31 10:10:35.629190760 (618h41m35.500273805s) [upload=542176]: Done starting upload Dec 31 10:10:35.629679280 (618h41m35.500762315s) [upload=542176]: Uploading segments... Dec 31 10:10:35.629727562 (618h41m35.500810607s) [upload=542176]: Got next segment. Inline: false Dec 31 10:10:35.629846739 (618h41m35.500929934s) [upload=542176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:10:35.629855776 (618h41m35.500938821s) [upload=542176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:10:35.876961313 (618h41m35.748044368s) [seg_pos={0 0}, upload=542176]: Begin upload segment... Dec 31 10:10:35.877026487 (618h41m35.748109522s) [seg_pos={0 0}, upload=542176]: Done begin upload segment. Dec 31 10:10:35.877097362 (618h41m35.748180397s) [noise=true , address=145.79.227.15:4873 , piece_id=YXJOTTA4CWYPJZYFVOFSOCRB7D342KOSCPLBHQ3ZQPAOKS3LHAQQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.877133511 (618h41m35.748216556s) [noise=true , address=145.79.227.37:4567 , piece_id=SIAFC57DT5XETI5WG63AV45BITQOSWV5C5GIAP7IZSGL6BFVT2QQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.877856328 (618h41m35.748939363s) [noise=true , address=145.79.227.43:4157 , piece_id=YHQN53ANULOQSBMQWKVWLGQ7IPWK6GRCLFVVRO7BAKR3HZS2KRRA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.877892677 (618h41m35.748975722s) [noise=true , address=145.79.227.52:4446 , piece_id=XXFYWWYZ5WY2RT557457C3QG2PCQAVA7AUB463A7QW5RRWXUOVYQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.878300072 (618h41m35.749383117s) [noise=true , address=145.79.227.50:4375 , piece_id=MT3JE2SSEZCZAP5JNGVFU2IMF2LOPYROS4EPNHXQ2TFNXXMR24QA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.878725713 (618h41m35.749808758s) [noise=true , address=45.91.116.13:4154 , piece_id=XQ6U5FO7H7IYN73HNMWQ564ZE6KUWYQRMQXOYICHXDBUSJGNM5WA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.878958375 (618h41m35.750041411s) [noise=true , address=145.79.227.18:4006 , piece_id=PJDAXPOWN4BQBPDDVNEGD3WSJDSLPWDOFRUKONDJFJREDE623G6Q, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.879437057 (618h41m35.750520092s) [noise=true , address=145.79.227.49:4338 , piece_id=FCTDOP2VPIMW2JIKXQZXBQQLYMUN7OW7GCE3ESHLD5K2SOJMCBUQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.883713911 (618h41m35.754796946s) [noise=true , address=145.79.227.51:4409 , piece_id=4MMNRZY3B6QLM33XANYIEAKPOXZUZ64SPACN3PBHXHZOIOFKXLZA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.885314279 (618h41m35.756397324s) [noise=true , address=77.90.31.16:4735 , piece_id=CJRL67ZIVBRTTH2YA23YAMH3TXZJO37F2H6W3RX44TFA64IBUJUA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.886841326 (618h41m35.757924372s) [noise=true , address=77.90.31.17:4773 , piece_id=HCEUM2OV6BICUKCDJMAGCZ4KHNULIFAZSVWEV2OWD3GVEB74UGLQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.888252273 (618h41m35.759335308s) [noise=true , address=45.91.116.12:4228 , piece_id=XT27AUBZQRHFTLVUMHZCP75SDRID4PFIETFYRBDFMTDLVYC7LRCQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.889399197 (618h41m35.760482242s) [noise=true , address=145.79.227.32:4834 , piece_id=OS2LAG2DKIQ5AWSSRUTBJXSZEPE6YD47H467NJLXKTXSR3B2XHJA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.890818369 (618h41m35.761901404s) [noise=true , address=145.79.227.22:4511 , piece_id=CEOXHLCIYVV4EP6NVLSBG2GRUAPHYSVOI75QPBAOSEV5YOHZ34KA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.892173279 (618h41m35.763256324s) [noise=true , address=45.91.116.8:4047 , piece_id=GXVDLB3QXVPRBHOFNNOCNWV642PQ4VKKI2I4QNOJPTPNIYFFFVWA, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:35.893423470 (618h41m35.764506515s) [noise=true , address=145.79.227.24:4583 , piece_id=UOGJIIR5JI56CV627NTLUJYZD32NV77ZVYONR7VY2TZEC77GPCZQ, seg_pos={0 0}, upload=542176]: Uploading piece... Dec 31 10:10:37.576645969 (618h41m37.447729004s) [upload=542176]: Next returned nil segment Dec 31 10:10:37.576661068 (618h41m37.447744113s) [upload=542176]: Waiting for error group managing segments... Dec 31 10:10:37.849660962 (618h41m37.720744007s) [noise=true , address=145.79.227.15:4873 , piece_id=YXJOTTA4CWYPJZYFVOFSOCRB7D342KOSCPLBHQ3ZQPAOKS3LHAQQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.863928750 (618h41m37.735011785s) [noise=true , address=145.79.227.52:4446 , piece_id=XXFYWWYZ5WY2RT557457C3QG2PCQAVA7AUB463A7QW5RRWXUOVYQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.881413494 (618h41m37.752496549s) [noise=true , address=145.79.227.37:4567 , piece_id=SIAFC57DT5XETI5WG63AV45BITQOSWV5C5GIAP7IZSGL6BFVT2QQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.885718451 (618h41m37.756801506s) [noise=true , address=145.79.227.24:4583 , piece_id=UOGJIIR5JI56CV627NTLUJYZD32NV77ZVYONR7VY2TZEC77GPCZQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.886989171 (618h41m37.758072226s) [noise=true , address=145.79.227.51:4409 , piece_id=4MMNRZY3B6QLM33XANYIEAKPOXZUZ64SPACN3PBHXHZOIOFKXLZA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.887674766 (618h41m37.758757811s) [noise=true , address=145.79.227.50:4375 , piece_id=MT3JE2SSEZCZAP5JNGVFU2IMF2LOPYROS4EPNHXQ2TFNXXMR24QA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.902235913 (618h41m37.773318958s) [noise=true , address=145.79.227.18:4006 , piece_id=PJDAXPOWN4BQBPDDVNEGD3WSJDSLPWDOFRUKONDJFJREDE623G6Q, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.903161244 (618h41m37.774244279s) [noise=true , address=145.79.227.43:4157 , piece_id=YHQN53ANULOQSBMQWKVWLGQ7IPWK6GRCLFVVRO7BAKR3HZS2KRRA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.907467825 (618h41m37.77855086s) [noise=true , address=145.79.227.22:4511 , piece_id=CEOXHLCIYVV4EP6NVLSBG2GRUAPHYSVOI75QPBAOSEV5YOHZ34KA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:37.969247964 (618h41m37.840331s) [noise=true , address=145.79.227.32:4834 , piece_id=OS2LAG2DKIQ5AWSSRUTBJXSZEPE6YD47H467NJLXKTXSR3B2XHJA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:38.065314039 (618h41m37.936397084s) [noise=true , address=77.90.31.16:4735 , piece_id=CJRL67ZIVBRTTH2YA23YAMH3TXZJO37F2H6W3RX44TFA64IBUJUA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:38.250011613 (618h41m38.121094658s) [noise=true , address=45.91.116.8:4047 , piece_id=GXVDLB3QXVPRBHOFNNOCNWV642PQ4VKKI2I4QNOJPTPNIYFFFVWA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:38.314286341 (618h41m38.185369386s) [noise=true , address=45.91.116.12:4228 , piece_id=XT27AUBZQRHFTLVUMHZCP75SDRID4PFIETFYRBDFMTDLVYC7LRCQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: <nil> Dec 31 10:10:38.314307741 (618h41m38.185390776s) [seg_pos={0 0}, upload=542176]: Segment reached optimal threshold of 13 pieces. Dec 31 10:10:38.314495930 (618h41m38.185578965s) [noise=true , address=145.79.227.49:4338 , piece_id=FCTDOP2VPIMW2JIKXQZXBQQLYMUN7OW7GCE3ESHLD5K2SOJMCBUQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 31 10:10:38.314503124 (618h41m38.185586169s) [noise=true , address=145.79.227.49:4338 , piece_id=FCTDOP2VPIMW2JIKXQZXBQQLYMUN7OW7GCE3ESHLD5K2SOJMCBUQ, seg_pos={0 0}, upload=542176]: Long tail cancellation due to optimal threshold Dec 31 10:10:38.314656706 (618h41m38.185739741s) [noise=true , address=45.91.116.13:4154 , piece_id=XQ6U5FO7H7IYN73HNMWQ564ZE6KUWYQRMQXOYICHXDBUSJGNM5WA, seg_pos={0 0}, upload=542176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 10:10:38.314663870 (618h41m38.185746905s) [noise=true , address=45.91.116.13:4154 , piece_id=XQ6U5FO7H7IYN73HNMWQ564ZE6KUWYQRMQXOYICHXDBUSJGNM5WA, seg_pos={0 0}, upload=542176]: Long tail cancellation due to optimal threshold Dec 31 10:10:38.314836279 (618h41m38.185919344s) [noise=true , address=77.90.31.17:4773 , piece_id=HCEUM2OV6BICUKCDJMAGCZ4KHNULIFAZSVWEV2OWD3GVEB74UGLQ, seg_pos={0 0}, upload=542176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 10:10:38.314847450 (618h41m38.185930485s) [noise=true , address=77.90.31.17:4773 , piece_id=HCEUM2OV6BICUKCDJMAGCZ4KHNULIFAZSVWEV2OWD3GVEB74UGLQ, seg_pos={0 0}, upload=542176]: Long tail cancellation due to optimal threshold Dec 31 10:10:38.314872017 (618h41m38.185955062s) [seg_pos={0 0}, upload=542176]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:10:38.314982968 (618h41m38.186066013s) [upload=542176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:10:38.314990492 (618h41m38.186073537s) [upload=542176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:10:38.610629745 (618h41m38.4817128s) [upload=542176]: Done uploading segments... Dec 31 10:10:38.610659172 (618h41m38.481742217s) [upload=542176]: Upload finished. err: <nil> Duration: 30.002850825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.....X.................X........X......X..............] duration: 31.63568146s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:11:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 31.635710255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX...X...X......XXX........XXX................XX......X....X] duration: 5.001453357s trace: https://jaeger.dev.storj.io/trace/5e8858832f68bf84 FAIL (2025-12-31 10:10:54 UTC) orbiter warning: context deadline exceeded Duration: 5.001478486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX...X...X......XXX........XXX................XX......X....] duration: 5.001593486s trace: https://jaeger.dev.storj.io/trace/12790fc05f1cb4cb FAIL (2025-12-31 10:10:08 UTC) orbiter warning: context deadline exceeded Duration: 5.00162741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.XX..X....X..........X.....X....XXX..X.X.X....XX........X.] duration: 5.72861573s trace: https://jaeger.dev.storj.io/trace/5cfcdbcbc74ba18c FAIL (2025-12-31 10:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 3V4HVGNFM7BV4YHECSGYEO6PRVG4UX4Q36VQMGLWXCXKTVECIVGA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 26M4AUYOOXWARU7GU3X3M23E5XEK3MNCPBYGH4I2DJMI3TKSHPWQ) context deadline exceeded ======LOGS====== Dec 31 10:08:33.122631847 (618h39m32.993714892s) [upload=542147]: Starting upload Dec 31 10:08:33.122689798 (618h39m32.993772843s) [upload=542147]: Done starting upload Dec 31 10:08:33.122894578 (618h39m32.993977623s) [upload=542147]: Uploading segments... Dec 31 10:08:33.123016720 (618h39m32.994099765s) [upload=542147]: Got next segment. Inline: false Dec 31 10:08:33.123060935 (618h39m32.99414397s) [upload=542147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:08:33.123067587 (618h39m32.994150623s) [upload=542147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:08:33.373668000 (618h39m33.244751045s) [seg_pos={0 0}, upload=542147]: Begin upload segment... Dec 31 10:08:33.373724767 (618h39m33.244807833s) [seg_pos={0 0}, upload=542147]: Done begin upload segment. Dec 31 10:08:33.373760636 (618h39m33.244843681s) [noise=true , address=45.91.116.11:4120 , piece_id=W6HUO3EML7AVTBIA57AFDM2BRHIWI4R4IQDLBQPEYSGRT6Z4GVAA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.373856078 (618h39m33.244939113s) [noise=true , address=45.91.116.8:4048 , piece_id=26M4AUYOOXWARU7GU3X3M23E5XEK3MNCPBYGH4I2DJMI3TKSHPWQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.374599553 (618h39m33.245682598s) [noise=true , address=145.79.227.9:4366 , piece_id=NB57KAH25HXALV4DAP3LJ4OGICAUD6M63H5LDQTTB2AEJ5DZGH4Q, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375084847 (618h39m33.246167882s) [noise=true , address=145.79.227.48:4301 , piece_id=4YZED3STH5VNLBTIGNWOB3V777L6ORCJ4LM44IPYZH5GIH5GIL4A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375125956 (618h39m33.246209082s) [noise=true , address=145.79.227.29:4941 , piece_id=4U55AYRHFPLFDK5XYNMZAZBVHIRFZNFRVLTXBTPJEOCV2OIZIAQQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375469249 (618h39m33.246552284s) [noise=true , address=145.79.227.46:4602 , piece_id=7IRVLS5NBUZO5IXANDIUNWQH7VFGNEGPMH3XKNCIQ32PQ5TBAWBQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375778077 (618h39m33.246861122s) [noise=true , address=145.79.227.35:4482 , piece_id=6BGFM5Y3OJXHUFPMCNCIUZ76OFTQ3Z3IGLLOXRDH6IQSULCIG53A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375855806 (618h39m33.246938851s) [noise=true , address=145.79.227.18:4007 , piece_id=26W6VUT3LMQKK4JBAF2CY57U7QTMCCDCRLQ2SZ3D276I6X6TYHPQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376106874 (618h39m33.247189909s) [noise=true , address=145.79.227.2:4115 , piece_id=5UUENII4XMPGXWQIOMZT26OYBIDURSLAPM5YGWXU6V7UMXCWE3RA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376420251 (618h39m33.247503296s) [noise=true , address=145.79.227.31:4797 , piece_id=3SNUZQQ5P2MNW36YAEVHAHBWEP4WSQNC6S37F65CHLN2H7BXIJ3A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376492538 (618h39m33.247575573s) [noise=true , address=145.79.227.17:4971 , piece_id=HAYSCYHCD2LSZKXHSU7WRMT5XP3T2RWVHHOEDKACN3PWXMGQJL5A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376812978 (618h39m33.247896013s) [noise=true , address=45.91.116.9:4262 , piece_id=3V4HVGNFM7BV4YHECSGYEO6PRVG4UX4Q36VQMGLWXCXKTVECIVGA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.377045311 (618h39m33.248128346s) [noise=true , address=45.91.116.5:4298 , piece_id=SVXWAIT3AION2UJ3XXRKXQDJ5O4KV4TDXBXTOZXQPSJK5D43AKDA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.377703745 (618h39m33.24878678s) [noise=true , address=145.79.227.42:4121 , piece_id=FX7DEOCONZEG7VECV2LKFA36XTY2YWSJKQZX5XIVVTPTFX2KH3ZA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.378395251 (618h39m33.249478286s) [noise=true , address=145.79.227.8:4330 , piece_id=AEQMOYE6T2VNWUPCNLT7J5IRERTH3WJS5AKQCQ7ACQWDB7Z4MPOQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.379716036 (618h39m33.250799071s) [noise=true , address=145.79.227.15:4876 , piece_id=Y57UZV7DJ2ACRFRUIXEXAIFUPMXMNZMPPLAC6V5L7RPODBV5HLTA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:35.066307819 (618h39m34.937390864s) [upload=542147]: Next returned nil segment Dec 31 10:08:35.066329842 (618h39m34.937412887s) [upload=542147]: Waiting for error group managing segments... Dec 31 10:08:35.347167731 (618h39m35.218250766s) [noise=true , address=145.79.227.9:4366 , piece_id=NB57KAH25HXALV4DAP3LJ4OGICAUD6M63H5LDQTTB2AEJ5DZGH4Q, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.361231180 (618h39m35.232314225s) [noise=true , address=145.79.227.8:4330 , piece_id=AEQMOYE6T2VNWUPCNLT7J5IRERTH3WJS5AKQCQ7ACQWDB7Z4MPOQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.367344751 (618h39m35.238427796s) [noise=true , address=145.79.227.2:4115 , piece_id=5UUENII4XMPGXWQIOMZT26OYBIDURSLAPM5YGWXU6V7UMXCWE3RA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.378154384 (618h39m35.249237419s) [noise=true , address=145.79.227.48:4301 , piece_id=4YZED3STH5VNLBTIGNWOB3V777L6ORCJ4LM44IPYZH5GIH5GIL4A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.378592318 (618h39m35.249675353s) [noise=true , address=145.79.227.29:4941 , piece_id=4U55AYRHFPLFDK5XYNMZAZBVHIRFZNFRVLTXBTPJEOCV2OIZIAQQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.384180950 (618h39m35.255263995s) [noise=true , address=145.79.227.17:4971 , piece_id=HAYSCYHCD2LSZKXHSU7WRMT5XP3T2RWVHHOEDKACN3PWXMGQJL5A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.386513692 (618h39m35.257596737s) [noise=true , address=145.79.227.18:4007 , piece_id=26W6VUT3LMQKK4JBAF2CY57U7QTMCCDCRLQ2SZ3D276I6X6TYHPQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.392891977 (618h39m35.263975022s) [noise=true , address=145.79.227.42:4121 , piece_id=FX7DEOCONZEG7VECV2LKFA36XTY2YWSJKQZX5XIVVTPTFX2KH3ZA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.449249902 (618h39m35.320332937s) [noise=true , address=145.79.227.31:4797 , piece_id=3SNUZQQ5P2MNW36YAEVHAHBWEP4WSQNC6S37F65CHLN2H7BXIJ3A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.453849911 (618h39m35.324932966s) [noise=true , address=145.79.227.46:4602 , piece_id=7IRVLS5NBUZO5IXANDIUNWQH7VFGNEGPMH3XKNCIQ32PQ5TBAWBQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.454207762 (618h39m35.325290807s) [noise=true , address=145.79.227.15:4876 , piece_id=Y57UZV7DJ2ACRFRUIXEXAIFUPMXMNZMPPLAC6V5L7RPODBV5HLTA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.690214404 (618h39m35.561297439s) [noise=true , address=45.91.116.8:4048 , piece_id=26M4AUYOOXWARU7GU3X3M23E5XEK3MNCPBYGH4I2DJMI3TKSHPWQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.746458401 (618h39m35.617541436s) [noise=true , address=45.91.116.9:4262 , piece_id=3V4HVGNFM7BV4YHECSGYEO6PRVG4UX4Q36VQMGLWXCXKTVECIVGA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.746498288 (618h39m35.617581323s) [seg_pos={0 0}, upload=542147]: Segment reached optimal threshold of 13 pieces. Dec 31 10:08:35.746724398 (618h39m35.617807443s) [noise=true , address=45.91.116.11:4120 , piece_id=W6HUO3EML7AVTBIA57AFDM2BRHIWI4R4IQDLBQPEYSGRT6Z4GVAA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 10:08:35.746735829 (618h39m35.617818874s) [noise=true , address=45.91.116.11:4120 , piece_id=W6HUO3EML7AVTBIA57AFDM2BRHIWI4R4IQDLBQPEYSGRT6Z4GVAA, seg_pos={0 0}, upload=542147]: Long tail cancellation due to optimal threshold Dec 31 10:08:35.746932745 (618h39m35.61801583s) [noise=true , address=145.79.227.35:4482 , piece_id=6BGFM5Y3OJXHUFPMCNCIUZ76OFTQ3Z3IGLLOXRDH6IQSULCIG53A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 10:08:35.746962622 (618h39m35.618045677s) [noise=true , address=145.79.227.35:4482 , piece_id=6BGFM5Y3OJXHUFPMCNCIUZ76OFTQ3Z3IGLLOXRDH6IQSULCIG53A, seg_pos={0 0}, upload=542147]: Long tail cancellation due to optimal threshold Dec 31 10:08:35.747082680 (618h39m35.618165735s) [noise=true , address=45.91.116.5:4298 , piece_id=SVXWAIT3AION2UJ3XXRKXQDJ5O4KV4TDXBXTOZXQPSJK5D43AKDA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 31 10:08:35.747093211 (618h39m35.618176266s) [noise=true , address=45.91.116.5:4298 , piece_id=SVXWAIT3AION2UJ3XXRKXQDJ5O4KV4TDXBXTOZXQPSJK5D43AKDA, seg_pos={0 0}, upload=542147]: Long tail cancellation due to optimal threshold Dec 31 10:08:35.747120121 (618h39m35.618203166s) [seg_pos={0 0}, upload=542147]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:08:35.747212628 (618h39m35.618295673s) [upload=542147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:08:35.747220332 (618h39m35.618303367s) [upload=542147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:08:36.046163938 (618h39m35.917246973s) [upload=542147]: Done uploading segments... Dec 31 10:08:36.046188234 (618h39m35.917271279s) [upload=542147]: Upload finished. err: <nil> Duration: 5.728636799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.....X.................X........X......X.............] duration: 20.495512795s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:09:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 20.495542681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX...X...X......XXX........XXX................XX......X...] duration: 3.592861518s trace: https://jaeger.dev.storj.io/trace/12790fc05f1cb4cb FAIL (2025-12-31 10:09:39 UTC) orbiter warning: context deadline exceeded Duration: 3.592891597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.XX..X....X..........X.....X....XXX..X.X.X....XX........X] duration: 30.006030322s trace: https://jaeger.dev.storj.io/trace/5cfcdbcbc74ba18c FAIL (2025-12-31 10:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 3V4HVGNFM7BV4YHECSGYEO6PRVG4UX4Q36VQMGLWXCXKTVECIVGA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 26M4AUYOOXWARU7GU3X3M23E5XEK3MNCPBYGH4I2DJMI3TKSHPWQ) context deadline exceeded ======LOGS====== Dec 31 10:08:33.122631847 (618h39m32.993714892s) [upload=542147]: Starting upload Dec 31 10:08:33.122689798 (618h39m32.993772843s) [upload=542147]: Done starting upload Dec 31 10:08:33.122894578 (618h39m32.993977623s) [upload=542147]: Uploading segments... Dec 31 10:08:33.123016720 (618h39m32.994099765s) [upload=542147]: Got next segment. Inline: false Dec 31 10:08:33.123060935 (618h39m32.99414397s) [upload=542147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:08:33.123067587 (618h39m32.994150623s) [upload=542147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:08:33.373668000 (618h39m33.244751045s) [seg_pos={0 0}, upload=542147]: Begin upload segment... Dec 31 10:08:33.373724767 (618h39m33.244807833s) [seg_pos={0 0}, upload=542147]: Done begin upload segment. Dec 31 10:08:33.373760636 (618h39m33.244843681s) [noise=true , address=45.91.116.11:4120 , piece_id=W6HUO3EML7AVTBIA57AFDM2BRHIWI4R4IQDLBQPEYSGRT6Z4GVAA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.373856078 (618h39m33.244939113s) [noise=true , address=45.91.116.8:4048 , piece_id=26M4AUYOOXWARU7GU3X3M23E5XEK3MNCPBYGH4I2DJMI3TKSHPWQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.374599553 (618h39m33.245682598s) [noise=true , address=145.79.227.9:4366 , piece_id=NB57KAH25HXALV4DAP3LJ4OGICAUD6M63H5LDQTTB2AEJ5DZGH4Q, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375084847 (618h39m33.246167882s) [noise=true , address=145.79.227.48:4301 , piece_id=4YZED3STH5VNLBTIGNWOB3V777L6ORCJ4LM44IPYZH5GIH5GIL4A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375125956 (618h39m33.246209082s) [noise=true , address=145.79.227.29:4941 , piece_id=4U55AYRHFPLFDK5XYNMZAZBVHIRFZNFRVLTXBTPJEOCV2OIZIAQQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375469249 (618h39m33.246552284s) [noise=true , address=145.79.227.46:4602 , piece_id=7IRVLS5NBUZO5IXANDIUNWQH7VFGNEGPMH3XKNCIQ32PQ5TBAWBQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375778077 (618h39m33.246861122s) [noise=true , address=145.79.227.35:4482 , piece_id=6BGFM5Y3OJXHUFPMCNCIUZ76OFTQ3Z3IGLLOXRDH6IQSULCIG53A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.375855806 (618h39m33.246938851s) [noise=true , address=145.79.227.18:4007 , piece_id=26W6VUT3LMQKK4JBAF2CY57U7QTMCCDCRLQ2SZ3D276I6X6TYHPQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376106874 (618h39m33.247189909s) [noise=true , address=145.79.227.2:4115 , piece_id=5UUENII4XMPGXWQIOMZT26OYBIDURSLAPM5YGWXU6V7UMXCWE3RA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376420251 (618h39m33.247503296s) [noise=true , address=145.79.227.31:4797 , piece_id=3SNUZQQ5P2MNW36YAEVHAHBWEP4WSQNC6S37F65CHLN2H7BXIJ3A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376492538 (618h39m33.247575573s) [noise=true , address=145.79.227.17:4971 , piece_id=HAYSCYHCD2LSZKXHSU7WRMT5XP3T2RWVHHOEDKACN3PWXMGQJL5A, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.376812978 (618h39m33.247896013s) [noise=true , address=45.91.116.9:4262 , piece_id=3V4HVGNFM7BV4YHECSGYEO6PRVG4UX4Q36VQMGLWXCXKTVECIVGA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.377045311 (618h39m33.248128346s) [noise=true , address=45.91.116.5:4298 , piece_id=SVXWAIT3AION2UJ3XXRKXQDJ5O4KV4TDXBXTOZXQPSJK5D43AKDA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.377703745 (618h39m33.24878678s) [noise=true , address=145.79.227.42:4121 , piece_id=FX7DEOCONZEG7VECV2LKFA36XTY2YWSJKQZX5XIVVTPTFX2KH3ZA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.378395251 (618h39m33.249478286s) [noise=true , address=145.79.227.8:4330 , piece_id=AEQMOYE6T2VNWUPCNLT7J5IRERTH3WJS5AKQCQ7ACQWDB7Z4MPOQ, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:33.379716036 (618h39m33.250799071s) [noise=true , address=145.79.227.15:4876 , piece_id=Y57UZV7DJ2ACRFRUIXEXAIFUPMXMNZMPPLAC6V5L7RPODBV5HLTA, seg_pos={0 0}, upload=542147]: Uploading piece... Dec 31 10:08:35.066307819 (618h39m34.937390864s) [upload=542147]: Next returned nil segment Dec 31 10:08:35.066329842 (618h39m34.937412887s) [upload=542147]: Waiting for error group managing segments... Dec 31 10:08:35.347167731 (618h39m35.218250766s) [noise=true , address=145.79.227.9:4366 , piece_id=NB57KAH25HXALV4DAP3LJ4OGICAUD6M63H5LDQTTB2AEJ5DZGH4Q, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.361231180 (618h39m35.232314225s) [noise=true , address=145.79.227.8:4330 , piece_id=AEQMOYE6T2VNWUPCNLT7J5IRERTH3WJS5AKQCQ7ACQWDB7Z4MPOQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.367344751 (618h39m35.238427796s) [noise=true , address=145.79.227.2:4115 , piece_id=5UUENII4XMPGXWQIOMZT26OYBIDURSLAPM5YGWXU6V7UMXCWE3RA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.378154384 (618h39m35.249237419s) [noise=true , address=145.79.227.48:4301 , piece_id=4YZED3STH5VNLBTIGNWOB3V777L6ORCJ4LM44IPYZH5GIH5GIL4A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.378592318 (618h39m35.249675353s) [noise=true , address=145.79.227.29:4941 , piece_id=4U55AYRHFPLFDK5XYNMZAZBVHIRFZNFRVLTXBTPJEOCV2OIZIAQQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.384180950 (618h39m35.255263995s) [noise=true , address=145.79.227.17:4971 , piece_id=HAYSCYHCD2LSZKXHSU7WRMT5XP3T2RWVHHOEDKACN3PWXMGQJL5A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.386513692 (618h39m35.257596737s) [noise=true , address=145.79.227.18:4007 , piece_id=26W6VUT3LMQKK4JBAF2CY57U7QTMCCDCRLQ2SZ3D276I6X6TYHPQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.392891977 (618h39m35.263975022s) [noise=true , address=145.79.227.42:4121 , piece_id=FX7DEOCONZEG7VECV2LKFA36XTY2YWSJKQZX5XIVVTPTFX2KH3ZA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.449249902 (618h39m35.320332937s) [noise=true , address=145.79.227.31:4797 , piece_id=3SNUZQQ5P2MNW36YAEVHAHBWEP4WSQNC6S37F65CHLN2H7BXIJ3A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.453849911 (618h39m35.324932966s) [noise=true , address=145.79.227.46:4602 , piece_id=7IRVLS5NBUZO5IXANDIUNWQH7VFGNEGPMH3XKNCIQ32PQ5TBAWBQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.454207762 (618h39m35.325290807s) [noise=true , address=145.79.227.15:4876 , piece_id=Y57UZV7DJ2ACRFRUIXEXAIFUPMXMNZMPPLAC6V5L7RPODBV5HLTA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.690214404 (618h39m35.561297439s) [noise=true , address=45.91.116.8:4048 , piece_id=26M4AUYOOXWARU7GU3X3M23E5XEK3MNCPBYGH4I2DJMI3TKSHPWQ, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.746458401 (618h39m35.617541436s) [noise=true , address=45.91.116.9:4262 , piece_id=3V4HVGNFM7BV4YHECSGYEO6PRVG4UX4Q36VQMGLWXCXKTVECIVGA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: <nil> Dec 31 10:08:35.746498288 (618h39m35.617581323s) [seg_pos={0 0}, upload=542147]: Segment reached optimal threshold of 13 pieces. Dec 31 10:08:35.746724398 (618h39m35.617807443s) [noise=true , address=45.91.116.11:4120 , piece_id=W6HUO3EML7AVTBIA57AFDM2BRHIWI4R4IQDLBQPEYSGRT6Z4GVAA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 10:08:35.746735829 (618h39m35.617818874s) [noise=true , address=45.91.116.11:4120 , piece_id=W6HUO3EML7AVTBIA57AFDM2BRHIWI4R4IQDLBQPEYSGRT6Z4GVAA, seg_pos={0 0}, upload=542147]: Long tail cancellation due to optimal threshold Dec 31 10:08:35.746932745 (618h39m35.61801583s) [noise=true , address=145.79.227.35:4482 , piece_id=6BGFM5Y3OJXHUFPMCNCIUZ76OFTQ3Z3IGLLOXRDH6IQSULCIG53A, seg_pos={0 0}, upload=542147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 10:08:35.746962622 (618h39m35.618045677s) [noise=true , address=145.79.227.35:4482 , piece_id=6BGFM5Y3OJXHUFPMCNCIUZ76OFTQ3Z3IGLLOXRDH6IQSULCIG53A, seg_pos={0 0}, upload=542147]: Long tail cancellation due to optimal threshold Dec 31 10:08:35.747082680 (618h39m35.618165735s) [noise=true , address=45.91.116.5:4298 , piece_id=SVXWAIT3AION2UJ3XXRKXQDJ5O4KV4TDXBXTOZXQPSJK5D43AKDA, seg_pos={0 0}, upload=542147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 31 10:08:35.747093211 (618h39m35.618176266s) [noise=true , address=45.91.116.5:4298 , piece_id=SVXWAIT3AION2UJ3XXRKXQDJ5O4KV4TDXBXTOZXQPSJK5D43AKDA, seg_pos={0 0}, upload=542147]: Long tail cancellation due to optimal threshold Dec 31 10:08:35.747120121 (618h39m35.618203166s) [seg_pos={0 0}, upload=542147]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:08:35.747212628 (618h39m35.618295673s) [upload=542147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:08:35.747220332 (618h39m35.618303367s) [upload=542147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:08:36.046163938 (618h39m35.917246973s) [upload=542147]: Done uploading segments... Dec 31 10:08:36.046188234 (618h39m35.917271279s) [upload=542147]: Upload finished. err: <nil> Duration: 30.006055869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.....X.................X........X......X............] duration: 33.585553866s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:08:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 33.585571669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX...X...X......XXX........XXX................XX......X..] duration: 3.565757116s trace: https://jaeger.dev.storj.io/trace/12790fc05f1cb4cb FAIL (2025-12-31 10:08:28 UTC) orbiter warning: context deadline exceeded Duration: 3.565784347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.....X.................X........X......X...........] duration: 35.972065083s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 35.972097897s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)