Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 18.812578343s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:28:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 18.812605404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X........................X......X..........] duration: 3.350157093s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:27:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.350183012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X......X....] duration: 5.898589711s trace: https://jaeger.dev.storj.io/trace/3afec173c6a0c14c FAIL (2025-12-30 02:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:25:06.765670430 (586h56m6.636753465s) [upload=514528]: Starting upload Dec 30 02:25:06.765726617 (586h56m6.636809662s) [upload=514528]: Done starting upload Dec 30 02:25:06.765874188 (586h56m6.636957233s) [upload=514528]: Uploading segments... Dec 30 02:25:06.766094707 (586h56m6.637177752s) [upload=514528]: Got next segment. Inline: false Dec 30 02:25:06.766130195 (586h56m6.63721324s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:25:06.766135655 (586h56m6.6372187s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:25:07.019166110 (586h56m6.890249145s) [seg_pos={0 0}, upload=514528]: Begin upload segment... Dec 30 02:25:07.019228738 (586h56m6.890311773s) [seg_pos={0 0}, upload=514528]: Done begin upload segment. Dec 30 02:25:07.019243758 (586h56m6.890326803s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019270248 (586h56m6.890353293s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019723060 (586h56m6.890806096s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020163258 (586h56m6.891246303s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020573690 (586h56m6.891656735s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020999821 (586h56m6.892082866s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.021436283 (586h56m6.892519318s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022130054 (586h56m6.893213099s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022585531 (586h56m6.893668576s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023034275 (586h56m6.894117321s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023548063 (586h56m6.894631108s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.025098496 (586h56m6.896181531s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.028194171 (586h56m6.899277216s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.029469810 (586h56m6.900552845s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.030671117 (586h56m6.901754162s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.031849090 (586h56m6.902932135s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:08.727720574 (586h56m8.598803619s) [upload=514528]: Next returned nil segment Dec 30 02:25:08.727744610 (586h56m8.598827645s) [upload=514528]: Waiting for error group managing segments... Dec 30 02:25:09.013917253 (586h56m8.885000288s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.014132563 (586h56m8.885215608s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.015650493 (586h56m8.886733538s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.017337295 (586h56m8.88842034s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023288547 (586h56m8.894371582s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023552550 (586h56m8.894635585s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.032926820 (586h56m8.904009855s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.036902200 (586h56m8.907985245s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.041189834 (586h56m8.912272879s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.242237985 (586h56m9.11332103s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.285422575 (586h56m9.15650561s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.330970757 (586h56m9.202053792s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390466027 (586h56m9.261549072s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390485394 (586h56m9.261568439s) [seg_pos={0 0}, upload=514528]: Segment reached optimal threshold of 13 pieces. Dec 30 02:25:09.390661429 (586h56m9.261744474s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390670226 (586h56m9.261753281s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390793360 (586h56m9.261876396s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390812587 (586h56m9.261895622s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390838246 (586h56m9.261921281s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390843746 (586h56m9.261926791s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390858594 (586h56m9.261941629s) [seg_pos={0 0}, upload=514528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:25:09.390935351 (586h56m9.262018386s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:25:09.390940861 (586h56m9.262023896s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:25:09.721725994 (586h56m9.592809039s) [upload=514528]: Done uploading segments... Dec 30 02:25:09.721744979 (586h56m9.592828014s) [upload=514528]: Upload finished. err: <nil> Duration: 5.898615209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X......X...] duration: 16.304747096s trace: https://jaeger.dev.storj.io/trace/3afec173c6a0c14c FAIL (2025-12-30 02:27:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:25:06.765670430 (586h56m6.636753465s) [upload=514528]: Starting upload Dec 30 02:25:06.765726617 (586h56m6.636809662s) [upload=514528]: Done starting upload Dec 30 02:25:06.765874188 (586h56m6.636957233s) [upload=514528]: Uploading segments... Dec 30 02:25:06.766094707 (586h56m6.637177752s) [upload=514528]: Got next segment. Inline: false Dec 30 02:25:06.766130195 (586h56m6.63721324s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:25:06.766135655 (586h56m6.6372187s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:25:07.019166110 (586h56m6.890249145s) [seg_pos={0 0}, upload=514528]: Begin upload segment... Dec 30 02:25:07.019228738 (586h56m6.890311773s) [seg_pos={0 0}, upload=514528]: Done begin upload segment. Dec 30 02:25:07.019243758 (586h56m6.890326803s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019270248 (586h56m6.890353293s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019723060 (586h56m6.890806096s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020163258 (586h56m6.891246303s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020573690 (586h56m6.891656735s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020999821 (586h56m6.892082866s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.021436283 (586h56m6.892519318s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022130054 (586h56m6.893213099s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022585531 (586h56m6.893668576s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023034275 (586h56m6.894117321s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023548063 (586h56m6.894631108s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.025098496 (586h56m6.896181531s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.028194171 (586h56m6.899277216s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.029469810 (586h56m6.900552845s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.030671117 (586h56m6.901754162s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.031849090 (586h56m6.902932135s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:08.727720574 (586h56m8.598803619s) [upload=514528]: Next returned nil segment Dec 30 02:25:08.727744610 (586h56m8.598827645s) [upload=514528]: Waiting for error group managing segments... Dec 30 02:25:09.013917253 (586h56m8.885000288s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.014132563 (586h56m8.885215608s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.015650493 (586h56m8.886733538s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.017337295 (586h56m8.88842034s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023288547 (586h56m8.894371582s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023552550 (586h56m8.894635585s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.032926820 (586h56m8.904009855s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.036902200 (586h56m8.907985245s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.041189834 (586h56m8.912272879s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.242237985 (586h56m9.11332103s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.285422575 (586h56m9.15650561s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.330970757 (586h56m9.202053792s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390466027 (586h56m9.261549072s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390485394 (586h56m9.261568439s) [seg_pos={0 0}, upload=514528]: Segment reached optimal threshold of 13 pieces. Dec 30 02:25:09.390661429 (586h56m9.261744474s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390670226 (586h56m9.261753281s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390793360 (586h56m9.261876396s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390812587 (586h56m9.261895622s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390838246 (586h56m9.261921281s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390843746 (586h56m9.261926791s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390858594 (586h56m9.261941629s) [seg_pos={0 0}, upload=514528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:25:09.390935351 (586h56m9.262018386s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:25:09.390940861 (586h56m9.262023896s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:25:09.721725994 (586h56m9.592809039s) [upload=514528]: Done uploading segments... Dec 30 02:25:09.721744979 (586h56m9.592828014s) [upload=514528]: Upload finished. err: <nil> Duration: 16.304772313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 15.350255076s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:27:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 15.350299832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X........................X......X.........] duration: 3.448359978s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:27:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448388112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X......X..] duration: 10.007212413s trace: https://jaeger.dev.storj.io/trace/3afec173c6a0c14c FAIL (2025-12-30 02:27:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:25:06.765670430 (586h56m6.636753465s) [upload=514528]: Starting upload Dec 30 02:25:06.765726617 (586h56m6.636809662s) [upload=514528]: Done starting upload Dec 30 02:25:06.765874188 (586h56m6.636957233s) [upload=514528]: Uploading segments... Dec 30 02:25:06.766094707 (586h56m6.637177752s) [upload=514528]: Got next segment. Inline: false Dec 30 02:25:06.766130195 (586h56m6.63721324s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:25:06.766135655 (586h56m6.6372187s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:25:07.019166110 (586h56m6.890249145s) [seg_pos={0 0}, upload=514528]: Begin upload segment... Dec 30 02:25:07.019228738 (586h56m6.890311773s) [seg_pos={0 0}, upload=514528]: Done begin upload segment. Dec 30 02:25:07.019243758 (586h56m6.890326803s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019270248 (586h56m6.890353293s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019723060 (586h56m6.890806096s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020163258 (586h56m6.891246303s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020573690 (586h56m6.891656735s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020999821 (586h56m6.892082866s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.021436283 (586h56m6.892519318s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022130054 (586h56m6.893213099s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022585531 (586h56m6.893668576s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023034275 (586h56m6.894117321s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023548063 (586h56m6.894631108s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.025098496 (586h56m6.896181531s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.028194171 (586h56m6.899277216s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.029469810 (586h56m6.900552845s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.030671117 (586h56m6.901754162s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.031849090 (586h56m6.902932135s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:08.727720574 (586h56m8.598803619s) [upload=514528]: Next returned nil segment Dec 30 02:25:08.727744610 (586h56m8.598827645s) [upload=514528]: Waiting for error group managing segments... Dec 30 02:25:09.013917253 (586h56m8.885000288s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.014132563 (586h56m8.885215608s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.015650493 (586h56m8.886733538s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.017337295 (586h56m8.88842034s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023288547 (586h56m8.894371582s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023552550 (586h56m8.894635585s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.032926820 (586h56m8.904009855s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.036902200 (586h56m8.907985245s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.041189834 (586h56m8.912272879s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.242237985 (586h56m9.11332103s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.285422575 (586h56m9.15650561s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.330970757 (586h56m9.202053792s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390466027 (586h56m9.261549072s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390485394 (586h56m9.261568439s) [seg_pos={0 0}, upload=514528]: Segment reached optimal threshold of 13 pieces. Dec 30 02:25:09.390661429 (586h56m9.261744474s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390670226 (586h56m9.261753281s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390793360 (586h56m9.261876396s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390812587 (586h56m9.261895622s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390838246 (586h56m9.261921281s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390843746 (586h56m9.261926791s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390858594 (586h56m9.261941629s) [seg_pos={0 0}, upload=514528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:25:09.390935351 (586h56m9.262018386s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:25:09.390940861 (586h56m9.262023896s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:25:09.721725994 (586h56m9.592809039s) [upload=514528]: Done uploading segments... Dec 30 02:25:09.721744979 (586h56m9.592828014s) [upload=514528]: Upload finished. err: <nil> Duration: 10.007238522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 31.505154704s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:26:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 31.505179701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X........................X......X........] duration: 3.340604051s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:26:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.340627215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X........................X......X.......] duration: 3.505539758s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:26:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505567851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 15.351883326s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 15.351910898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X......X.] duration: 5.906571659s trace: https://jaeger.dev.storj.io/trace/3afec173c6a0c14c FAIL (2025-12-30 02:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:25:06.765670430 (586h56m6.636753465s) [upload=514528]: Starting upload Dec 30 02:25:06.765726617 (586h56m6.636809662s) [upload=514528]: Done starting upload Dec 30 02:25:06.765874188 (586h56m6.636957233s) [upload=514528]: Uploading segments... Dec 30 02:25:06.766094707 (586h56m6.637177752s) [upload=514528]: Got next segment. Inline: false Dec 30 02:25:06.766130195 (586h56m6.63721324s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:25:06.766135655 (586h56m6.6372187s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:25:07.019166110 (586h56m6.890249145s) [seg_pos={0 0}, upload=514528]: Begin upload segment... Dec 30 02:25:07.019228738 (586h56m6.890311773s) [seg_pos={0 0}, upload=514528]: Done begin upload segment. Dec 30 02:25:07.019243758 (586h56m6.890326803s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019270248 (586h56m6.890353293s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019723060 (586h56m6.890806096s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020163258 (586h56m6.891246303s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020573690 (586h56m6.891656735s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020999821 (586h56m6.892082866s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.021436283 (586h56m6.892519318s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022130054 (586h56m6.893213099s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022585531 (586h56m6.893668576s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023034275 (586h56m6.894117321s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023548063 (586h56m6.894631108s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.025098496 (586h56m6.896181531s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.028194171 (586h56m6.899277216s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.029469810 (586h56m6.900552845s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.030671117 (586h56m6.901754162s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.031849090 (586h56m6.902932135s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:08.727720574 (586h56m8.598803619s) [upload=514528]: Next returned nil segment Dec 30 02:25:08.727744610 (586h56m8.598827645s) [upload=514528]: Waiting for error group managing segments... Dec 30 02:25:09.013917253 (586h56m8.885000288s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.014132563 (586h56m8.885215608s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.015650493 (586h56m8.886733538s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.017337295 (586h56m8.88842034s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023288547 (586h56m8.894371582s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023552550 (586h56m8.894635585s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.032926820 (586h56m8.904009855s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.036902200 (586h56m8.907985245s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.041189834 (586h56m8.912272879s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.242237985 (586h56m9.11332103s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.285422575 (586h56m9.15650561s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.330970757 (586h56m9.202053792s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390466027 (586h56m9.261549072s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390485394 (586h56m9.261568439s) [seg_pos={0 0}, upload=514528]: Segment reached optimal threshold of 13 pieces. Dec 30 02:25:09.390661429 (586h56m9.261744474s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390670226 (586h56m9.261753281s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390793360 (586h56m9.261876396s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390812587 (586h56m9.261895622s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390838246 (586h56m9.261921281s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390843746 (586h56m9.261926791s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390858594 (586h56m9.261941629s) [seg_pos={0 0}, upload=514528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:25:09.390935351 (586h56m9.262018386s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:25:09.390940861 (586h56m9.262023896s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:25:09.721725994 (586h56m9.592809039s) [upload=514528]: Done uploading segments... Dec 30 02:25:09.721744979 (586h56m9.592828014s) [upload=514528]: Upload finished. err: <nil> Duration: 5.906603761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X......X] duration: 30.003905955s trace: https://jaeger.dev.storj.io/trace/3afec173c6a0c14c FAIL (2025-12-30 02:25:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:25:06.765670430 (586h56m6.636753465s) [upload=514528]: Starting upload Dec 30 02:25:06.765726617 (586h56m6.636809662s) [upload=514528]: Done starting upload Dec 30 02:25:06.765874188 (586h56m6.636957233s) [upload=514528]: Uploading segments... Dec 30 02:25:06.766094707 (586h56m6.637177752s) [upload=514528]: Got next segment. Inline: false Dec 30 02:25:06.766130195 (586h56m6.63721324s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:25:06.766135655 (586h56m6.6372187s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:25:07.019166110 (586h56m6.890249145s) [seg_pos={0 0}, upload=514528]: Begin upload segment... Dec 30 02:25:07.019228738 (586h56m6.890311773s) [seg_pos={0 0}, upload=514528]: Done begin upload segment. Dec 30 02:25:07.019243758 (586h56m6.890326803s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019270248 (586h56m6.890353293s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.019723060 (586h56m6.890806096s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020163258 (586h56m6.891246303s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020573690 (586h56m6.891656735s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.020999821 (586h56m6.892082866s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.021436283 (586h56m6.892519318s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022130054 (586h56m6.893213099s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.022585531 (586h56m6.893668576s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023034275 (586h56m6.894117321s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.023548063 (586h56m6.894631108s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.025098496 (586h56m6.896181531s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.028194171 (586h56m6.899277216s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.029469810 (586h56m6.900552845s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.030671117 (586h56m6.901754162s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:07.031849090 (586h56m6.902932135s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Uploading piece... Dec 30 02:25:08.727720574 (586h56m8.598803619s) [upload=514528]: Next returned nil segment Dec 30 02:25:08.727744610 (586h56m8.598827645s) [upload=514528]: Waiting for error group managing segments... Dec 30 02:25:09.013917253 (586h56m8.885000288s) [noise=true , address=145.79.227.17:4972 , piece_id=OJZI4D4EAY65PXB36QH6USWHFTPAPZIWFKPOQHTTJUMFJQ4J6TUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.014132563 (586h56m8.885215608s) [noise=true , address=145.79.227.18:4006 , piece_id=4PQNKADPSRXNPOTGN6XLWSHBBQ7G5A46APBF4YNZXTXG5RCL5ZSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.015650493 (586h56m8.886733538s) [noise=true , address=145.79.227.7:4294 , piece_id=IUIOVHEXJVVCB6XBX5YDAEDXKMEUAFAFHGHD5UJJU4ZVV5KAXZNA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.017337295 (586h56m8.88842034s) [noise=true , address=145.79.227.45:4229 , piece_id=VSZ6EL2ODR23XIUXXUDVIX3ZQ6IIPY66RWMJJDY3SH5VYPGADXWQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023288547 (586h56m8.894371582s) [noise=true , address=145.79.227.5:4222 , piece_id=KNUCEB4J2GAR66QRV5GZ7NPJUJIGGE5OTGM72YGFD3H6PRI2C4QQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.023552550 (586h56m8.894635585s) [noise=true , address=145.79.227.16:4910 , piece_id=OBAHZTIJH2ZW5ZAIBKPLYCZG2SP6VTI64BVE7OBVUYE6ILD6FCUA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.032926820 (586h56m8.904009855s) [noise=true , address=145.79.227.27:4689 , piece_id=RFS7FU6JBAHAC2GC26LGS6ZS7UFFR4VX2FKIHWWY6MBHOBXMUDOQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.036902200 (586h56m8.907985245s) [noise=true , address=145.79.227.15:4873 , piece_id=O5A6GIKU5LK7XM5Z3NDKXYSZ6F3VIKNBAQNNZHRYM3T67IOXT3TQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.041189834 (586h56m8.912272879s) [noise=true , address=145.79.227.4:4186 , piece_id=3O2MTUERBBN7ITHYQZQ7OUIPVAQB7AEOKLKWCAV6QOGAOBDWADRQ, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.242237985 (586h56m9.11332103s) [noise=true , address=77.90.31.14:4661 , piece_id=SZZSN4VAUEQPDEPRVPEDZ6N4PJKPS4W42ZTOFN2N2M3632EE4QVA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.285422575 (586h56m9.15650561s) [noise=true , address=45.91.116.6:4974 , piece_id=A23S5BEJZVGHKWWN6XSO2WSU5ZJREHOGZRODTYG4ARXIDYSWOOYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.330970757 (586h56m9.202053792s) [noise=true , address=45.91.116.10:4081 , piece_id=3DONPAP54CLD3QZZPH3AHY4YCNXFKUBKGC4ZTGIXZV3232BRVL4A, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390466027 (586h56m9.261549072s) [noise=true , address=45.91.116.8:4046 , piece_id=YP7H56KOUME6AQHRMKBJ5LLRWDU225LPWGPKHW2BDNRMB3CD6L7Q, seg_pos={0 0}, upload=514528]: Done uploading piece. err: <nil> Dec 30 02:25:09.390485394 (586h56m9.261568439s) [seg_pos={0 0}, upload=514528]: Segment reached optimal threshold of 13 pieces. Dec 30 02:25:09.390661429 (586h56m9.261744474s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390670226 (586h56m9.261753281s) [noise=true , address=145.79.227.44:4193 , piece_id=4CD7IOUYODOSTI5VHER4QOGKHNVBZVBB7I4HAY2DMGXTW7ZM7TSA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390793360 (586h56m9.261876396s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390812587 (586h56m9.261895622s) [noise=true , address=77.90.31.12:4590 , piece_id=3I72BUB3YALWE7V7DQLG5AJIUV3A2U2MZVGORJPEEZ4CPN7S4NYA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390838246 (586h56m9.261921281s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 30 02:25:09.390843746 (586h56m9.261926791s) [noise=true , address=77.90.31.18:4805 , piece_id=EVZNGOACS5OQ7IKMAZBXC77KEAKGBCU2BKOCE2NTERNOCMSIJWGA, seg_pos={0 0}, upload=514528]: Long tail cancellation due to optimal threshold Dec 30 02:25:09.390858594 (586h56m9.261941629s) [seg_pos={0 0}, upload=514528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:25:09.390935351 (586h56m9.262018386s) [upload=514528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:25:09.390940861 (586h56m9.262023896s) [upload=514528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:25:09.721725994 (586h56m9.592809039s) [upload=514528]: Done uploading segments... Dec 30 02:25:09.721744979 (586h56m9.592828014s) [upload=514528]: Upload finished. err: <nil> Duration: 30.003930352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 32.646172519s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:25:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 32.646202217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X........................X......X......] duration: 3.528391762s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:25:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528407441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 30.050966948s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:24:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 30.050982346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X......] duration: 5.965412113s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:24:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 5.965438712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X........................X......X.....] duration: 3.648472376s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:24:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.648501851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 38.134911102s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:24:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 38.134933644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X.....] duration: 6.028301535s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:23:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 6.028326272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.041883129s trace: https://jaeger.dev.storj.io/trace/5f9e7eab87e2ea2b FAIL (2025-12-30 02:23:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, Piece ID: RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA) context deadline exceeded; piecestore: (Node ID: 145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, Piece ID: Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ) context deadline exceeded; piecestore: (Node ID: 13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, Piece ID: L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ) context deadline exceeded ======LOGS====== SYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136414718 (586h53m25.007497763s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.136690654 (586h53m25.007773689s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.137012757 (586h53m25.008095792s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138436869 (586h53m25.009519904s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138626319 (586h53m25.009709364s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.138831530 (586h53m25.009914575s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139032913 (586h53m25.010115948s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139226461 (586h53m25.010309496s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139415893 (586h53m25.010498928s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139666560 (586h53m25.010749595s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.139809011 (586h53m25.010892046s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140164367 (586h53m25.011247402s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:25.140480109 (586h53m25.011563154s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Uploading piece... Dec 30 02:22:26.588732794 (586h53m26.459815849s) [upload=514489]: Next returned nil segment Dec 30 02:22:26.588747371 (586h53m26.459830406s) [upload=514489]: Waiting for error group managing segments... Dec 30 02:22:27.003987751 (586h53m26.875070826s) [noise=true , address=145.79.227.48:4311 , piece_id=IWNFP4NMJQSIEE23RJVSEPSWNOEBBDCMUFDXHFTEIEHZ3IUTFJKA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.028878809 (586h53m26.899961844s) [noise=true , address=145.79.227.43:4192 , piece_id=S7EGMJW4LGBRTZ6XDQIQFP66R6IEJHOKDNYQSJGAR4OW7JSRCMEQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.032333247 (586h53m26.903416292s) [noise=true , address=145.79.227.17:4000 , piece_id=25IIANYVXQPUSOYKGHZIGYSRZNPPFZRNF2P6PTVC4JJOGYL2HLNQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.054942308 (586h53m26.926025343s) [noise=true , address=145.79.227.4:4220 , piece_id=YHPF7S7AKKZZ56D225WRCJMGEG5PUVQAXUJB53F3NAHNV6G5HPZQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.180902007 (586h53m27.051985042s) [noise=true , address=77.90.31.18:4813 , piece_id=RXQUTGWZIYJAXIXYHZEXQICWPUORADO6XDMW63V5QCIUIEH7SACA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.282860945 (586h53m27.15394398s) [noise=true , address=77.90.31.16:4740 , piece_id=Z2632BEPPUCLJY4NU7DKODNFERS6AKIHVDQYXMGOU722CVLL5JDQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.320983917 (586h53m27.192066962s) [noise=true , address=45.91.116.10:4095 , piece_id=RRHGHTFZE4A4LWNSWKB6MSLYNDKGLW6JVR43DJT2NOAIDFAISNZA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.417577408 (586h53m27.288660453s) [noise=true , address=45.91.116.13:4176 , piece_id=36TL5N4WI7FM3XOJP2LFNVNYBKUJXXGB5VXT764C2DFA5MVRQ2FA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.897199993 (586h53m27.768283029s) [noise=true , address=145.79.227.40:4053 , piece_id=7M2AVNOBPNSYCC2PZHQMEVP7S27BN2P4E5TZ36AWLR7IYK3YYPYA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.915763121 (586h53m27.786846166s) [noise=true , address=145.79.227.30:4771 , piece_id=6Z43CPMDQD32HK6ACLFXRLGRRNVJ6YWPUQY35AFP5NPQACB6BJ6Q, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.920813107 (586h53m27.791896142s) [noise=true , address=145.79.227.44:4210 , piece_id=Y47F4CVAA654FTX2GZSMQRVCEYMB46PIXQMFASDNTB6DE4T7VJRA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:27.949217483 (586h53m27.820300529s) [noise=true , address=145.79.227.40:4054 , piece_id=HJKDZMI3LJDRS5HLMIKOV5YANE2GDIOCJP6D5TT6VXQBSM3ZDPJA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.047549505 (586h53m27.91863255s) [noise=true , address=77.90.31.21:4908 , piece_id=L2IN5QPAEIXPDUNJIS3GJ7O2OYDDUNXUK6HKGRVSW4SL2IXNGJQQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079498461 (586h53m27.950581496s) [noise=true , address=145.79.227.34:4935 , piece_id=VBHQPM2QSCKUCBFLPVE2FXFJH7Q2E5BJTD7HO23JEROSCO2QXGWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: <nil> Dec 30 02:22:28.079522788 (586h53m27.950605833s) [seg_pos={0 0}, upload=514489]: Segment reached optimal threshold of 14 pieces. Dec 30 02:22:28.079715865 (586h53m27.9507989s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43866->64.49.13.169:4713: use of closed network connection Dec 30 02:22:28.079722357 (586h53m27.950805402s) [noise=true , address=64.49.13.169:4713 , piece_id=GX3XD47RFI7MVPRI2MZH3ESYPBLNEQBYRUZWDC5JBHYQMY6NO74A, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079769177 (586h53m27.950852212s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx): protocol: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47280->77.90.31.14:4686: use of closed network connection Dec 30 02:22:28.079782873 (586h53m27.950865908s) [noise=true , address=77.90.31.14:4686 , piece_id=T52H6UEFQRM5LZDNCS3NHLZXWDRSOEBATNE3KGAO2GKCKZVTUWCQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079870380 (586h53m27.950953425s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MySWgnTy1zEYb7DU3bbahW2eJpY8t9UZUU1P9mN2L9ZgwQYgL): protocol: expected piece hash; context canceled; EOF Dec 30 02:22:28.079876211 (586h53m27.950959246s) [noise=true , address=5.231.73.4:4007 , piece_id=CH6BEVO6H3NR7OVGMGKQRZ4FE5ZL7TSRIL3JXSWYLASYCWLEOTBQ, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.079959138 (586h53m27.951042173s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52938->145.79.227.19:4097: use of closed network connection Dec 30 02:22:28.079964448 (586h53m27.951047483s) [noise=true , address=145.79.227.19:4097 , piece_id=B3RRPGZTR3CS4BFEQ4ZTO2VK7PBSXKUOAFELMF4QZGOFB2HBHIWA, seg_pos={0 0}, upload=514489]: Long tail cancellation due to optimal threshold Dec 30 02:22:28.080063046 (586h53m27.951146081s) [seg_pos={0 0}, upload=514489]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:22:28.080147688 (586h53m27.951230733s) [upload=514489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:22:28.080151906 (586h53m27.951234941s) [upload=514489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:22:28.375145474 (586h53m28.246228519s) [upload=514489]: Done uploading segments... Dec 30 02:22:28.375166896 (586h53m28.246249931s) [upload=514489]: Upload finished. err: <nil> Duration: 1m0.041893559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X........................X......X....] duration: 3.47299669s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:23:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473025745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X....] duration: 6.020819878s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 6.020849314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X...] duration: 6.02213375s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:22:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 6.022165039s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X........................X......X...] duration: 3.618941725s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:22:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.618971281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X..] duration: 6.029002821s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:21:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 6.029028479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X.] duration: 27.262021671s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:21:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 27.262048393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X........................X......X..] duration: 5.001023028s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:21:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001049708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..X] duration: 30.002549712s trace: https://jaeger.dev.storj.io/trace/1838e9dff231ece7 FAIL (2025-12-30 02:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (586h51m27.482743992s) [upload=514456]: Done starting upload Dec 30 02:20:27.611805051 (586h51m27.482888086s) [upload=514456]: Uploading segments... Dec 30 02:20:27.611913367 (586h51m27.482996402s) [upload=514456]: Got next segment. Inline: false Dec 30 02:20:27.611965447 (586h51m27.483048492s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:20:27.611973492 (586h51m27.483056527s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:20:27.856577490 (586h51m27.727660535s) [seg_pos={0 0}, upload=514456]: Begin upload segment... Dec 30 02:20:27.856655039 (586h51m27.727738114s) [seg_pos={0 0}, upload=514456]: Done begin upload segment. Dec 30 02:20:27.856705585 (586h51m27.72778862s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.856869918 (586h51m27.727952993s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.857720617 (586h51m27.728803682s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858037611 (586h51m27.729120646s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858398038 (586h51m27.729481073s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858728546 (586h51m27.729811592s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.858980396 (586h51m27.730063431s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859128799 (586h51m27.730211844s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859487131 (586h51m27.730570176s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859550332 (586h51m27.730633377s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.859918412 (586h51m27.731001457s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860036758 (586h51m27.731119793s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860317843 (586h51m27.731400888s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.860672148 (586h51m27.731755193s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861228507 (586h51m27.732311542s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:27.861703291 (586h51m27.732786337s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Uploading piece... Dec 30 02:20:29.547464453 (586h51m29.418547498s) [upload=514456]: Next returned nil segment Dec 30 02:20:29.547504359 (586h51m29.418587404s) [upload=514456]: Waiting for error group managing segments... Dec 30 02:20:29.827901545 (586h51m29.69898459s) [noise=true , address=145.79.227.46:4601 , piece_id=LKZCFOHPNRZAUDCD42PC4YQKQUVDAEXXASLMDIPGYP6YCF3DXUTQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.830992350 (586h51m29.702075396s) [noise=true , address=145.79.227.52:4445 , piece_id=5QD3AOJV5PW6OG4NXXQRKW7V6O4HNORTLSPWMPYY5H7CI7R4BDLA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.836779921 (586h51m29.707862966s) [noise=true , address=145.79.227.51:4409 , piece_id=ZA5TC6NNFENUVX4S4HUWSLXBA6QJPKFQZU3MK6R6SVP265OF3ZEQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.848707322 (586h51m29.719790357s) [noise=true , address=145.79.227.41:4086 , piece_id=52XPRVPOSM56XOUDXWB7ZT5ZT4FFKSW552BR2KVQB6ZUNC33IW4Q, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.860054499 (586h51m29.731137544s) [noise=true , address=145.79.227.26:4655 , piece_id=AX5EK63GLB7D6ZYP4BLPCIHJ4LVGRKUNG7QN76EIIJEXGO3377SA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.870654173 (586h51m29.741737218s) [noise=true , address=145.79.227.10:4402 , piece_id=KQ5JDTZKO2W6DFDEQWWEMC4NQ2MA757WG4NJC4Q7OBKAOZDE23GA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:29.923845308 (586h51m29.794928343s) [noise=true , address=145.79.227.4:4185 , piece_id=HIYZVK4LMFR3UYETTQBHNY6UAHVVXR2K2M2YY2FSWIT6RNK3IKGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.069591364 (586h51m29.940674409s) [noise=true , address=45.91.116.12:4227 , piece_id=57FZBRKA4JGV7ZJMM3JP6HPAGGVPQJ2F7HYZ7CDQ5FYL4OWWWD4A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.146512365 (586h51m30.01759541s) [noise=true , address=45.91.116.9:4261 , piece_id=IV2DODK3FO65XFQ3LNZUGNHHFHNWIK6BIRBZAGGYWOZ7KKFUEZSA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.211986640 (586h51m30.083069675s) [noise=true , address=77.90.31.21:4878 , piece_id=HFE4FDPNQXVI5BEMJAGYFTIAOUQ26GVVPQFSKCZWOONOFPBQJS7A, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.230573943 (586h51m30.101656998s) [noise=true , address=45.91.116.13:4154 , piece_id=N7Q4I3DFCKBLSEL6UMVV3K5BJY3OAHEQGNC3BWCDKZ7CZHM2OZHA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.231494035 (586h51m30.10257707s) [noise=true , address=45.91.116.6:4975 , piece_id=TZ5ZWQFURXVQYXBKWNWHTYYPDQAWMJEJE2IDFJMA3ZY7LCGURSPQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236742880 (586h51m30.107825925s) [noise=true , address=45.91.116.10:4081 , piece_id=EI5Y5PM4Z6ARLRCUYIOSZAFDSYKYLPNBKKDUWBAMMPCHBPMO3VJQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: <nil> Dec 30 02:20:30.236768448 (586h51m30.107851483s) [seg_pos={0 0}, upload=514456]: Segment reached optimal threshold of 13 pieces. Dec 30 02:20:30.237037781 (586h51m30.108120816s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 02:20:30.237046888 (586h51m30.108129923s) [noise=true , address=45.91.116.8:4046 , piece_id=B5KH77ZUIM4ULKRSWWEXGE6DAPRRO5GCTYINAS5SXJXRJD5T4ACQ, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237153211 (586h51m30.108236246s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44844->45.91.116.4:4939: use of closed network connection Dec 30 02:20:30.237158441 (586h51m30.108241476s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ7N53BZ3Q7LUORMH4DKY624KKL7LZVV5OHRLWK3VXFQS2XI32EA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237170484 (586h51m30.108253529s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: read tcp 188.166.225.170:39998->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 02:20:30.237179751 (586h51m30.108262786s) [noise=true , address=45.91.116.7:4009 , piece_id=2T3B24TXJKTO7LJMQMR6ZGJLIKXP3HQLPYWRC6QPDAPW3KAOEMGA, seg_pos={0 0}, upload=514456]: Long tail cancellation due to optimal threshold Dec 30 02:20:30.237196614 (586h51m30.108279659s) [seg_pos={0 0}, upload=514456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:20:30.237310290 (586h51m30.108393335s) [upload=514456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:20:30.237315169 (586h51m30.108398204s) [upload=514456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:20:30.531526589 (586h51m30.402609624s) [upload=514456]: Done uploading segments... Dec 30 02:20:30.531547168 (586h51m30.402630203s) [upload=514456]: Upload finished. err: <nil> Duration: 30.002566093s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X........................X......X.] duration: 3.48509661s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:20:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48512318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X........................X......X] duration: 5.002374742s trace: https://jaeger.dev.storj.io/trace/7555748071abd338 FAIL (2025-12-30 02:20:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002401212s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX..] duration: 11.951447355s trace: https://jaeger.dev.storj.io/trace/37e31797b0061bf3 FAIL (2025-12-30 02:19:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ) context deadline exceeded ======LOGS====== Dec 30 02:18:03.288068866 (586h49m3.159151911s) [upload=514424]: Starting upload Dec 30 02:18:03.288169066 (586h49m3.159252112s) [upload=514424]: Done starting upload Dec 30 02:18:03.288322318 (586h49m3.159405373s) [upload=514424]: Uploading segments... Dec 30 02:18:03.288487964 (586h49m3.159571009s) [upload=514424]: Got next segment. Inline: false Dec 30 02:18:03.288571452 (586h49m3.159654507s) [upload=514424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:18:03.288580379 (586h49m3.159663424s) [upload=514424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:18:03.531253932 (586h49m3.402336977s) [seg_pos={0 0}, upload=514424]: Begin upload segment... Dec 30 02:18:03.531326290 (586h49m3.402409335s) [seg_pos={0 0}, upload=514424]: Done begin upload segment. Dec 30 02:18:03.531364372 (586h49m3.402447417s) [noise=true , address=145.79.227.51:4410 , piece_id=ACIVJ6L4XUPJHV6IZAVKRRSHOTPMNJKWK2JSSCPUW7KB2DKC7AOA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.531440958 (586h49m3.402523993s) [noise=true , address=145.79.227.19:4078 , piece_id=SYHXAA63GI7SIIB77JLONRWUPBGZ2ABJLRZRCLW3Z4JHTAPDD7MQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532201977 (586h49m3.403285012s) [noise=true , address=45.91.116.5:4299 , piece_id=6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532427166 (586h49m3.403510211s) [noise=true , address=145.79.227.16:4911 , piece_id=NRIA24QRHAMXGQXFV5FD6WT5KSZSWBJCW2MM3LCKOY42PXKL2UMA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532874448 (586h49m3.403957483s) [noise=true , address=45.91.116.12:4225 , piece_id=EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533208073 (586h49m3.404291118s) [noise=true , address=45.91.116.14:4191 , piece_id=BCJ5ROPBZW7PFQYOIUQJNRDDLZ4UHWE7LT63F7IVJU6KKREVQ35A, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533446597 (586h49m3.404529642s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533533303 (586h49m3.404616338s) [noise=true , address=77.90.31.18:4805 , piece_id=55OYHJT4XH5Y6HZRI7AWOVI4C5E6CXPZNL6K4GHAHBV2XT2IBBYA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533922614 (586h49m3.405005659s) [noise=true , address=145.79.227.36:4519 , piece_id=63CAL2PC7NLUHQXTRSN4UNUX477DYI2TGETWHFZGBXU67W3TFWYQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534182819 (586h49m3.405265864s) [noise=true , address=145.79.227.17:4971 , piece_id=AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534306886 (586h49m3.405389931s) [noise=true , address=145.79.227.3:4151 , piece_id=HHAWFW7WFN4KRLULZV2PI3JARDBXBYLIGLC2SQ26M2JQ7563NGAQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534645760 (586h49m3.405728795s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534811366 (586h49m3.405894411s) [noise=true , address=145.79.227.32:4834 , piece_id=7VME5HIQC7YW32BEPS6TIHMQS4KKEKH3DSOZW5LILVEMHFKSA4VA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.535659571 (586h49m3.406742607s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.536519608 (586h49m3.407602654s) [noise=true , address=145.79.227.26:4654 , piece_id=FLGXCZIPI6USZVJ6IWOXCXFS5X3FQK7MWAYVYGMAE6ESU55ASG4Q, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.537229591 (586h49m3.408312636s) [noise=true , address=145.79.227.38:4979 , piece_id=N4PGG5GSKZPRTMECGFLRO2V52TK5ULWHXS6BKSFGLSQUSWSI4JFA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:05.184357438 (586h49m5.055440483s) [upload=514424]: Next returned nil segment Dec 30 02:18:05.184373058 (586h49m5.055456093s) [upload=514424]: Waiting for error group managing segments... Dec 30 02:18:05.526931920 (586h49m5.398014965s) [noise=true , address=145.79.227.51:4410 , piece_id=ACIVJ6L4XUPJHV6IZAVKRRSHOTPMNJKWK2JSSCPUW7KB2DKC7AOA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.529517132 (586h49m5.400600167s) [noise=true , address=145.79.227.16:4911 , piece_id=NRIA24QRHAMXGQXFV5FD6WT5KSZSWBJCW2MM3LCKOY42PXKL2UMA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.544046158 (586h49m5.415129193s) [noise=true , address=145.79.227.26:4654 , piece_id=FLGXCZIPI6USZVJ6IWOXCXFS5X3FQK7MWAYVYGMAE6ESU55ASG4Q, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.545510566 (586h49m5.416593601s) [noise=true , address=145.79.227.3:4151 , piece_id=HHAWFW7WFN4KRLULZV2PI3JARDBXBYLIGLC2SQ26M2JQ7563NGAQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.545568056 (586h49m5.416651101s) [noise=true , address=145.79.227.19:4078 , piece_id=SYHXAA63GI7SIIB77JLONRWUPBGZ2ABJLRZRCLW3Z4JHTAPDD7MQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.547341713 (586h49m5.418424759s) [noise=true , address=145.79.227.17:4971 , piece_id=AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.551635028 (586h49m5.422718073s) [noise=true , address=145.79.227.38:4979 , piece_id=N4PGG5GSKZPRTMECGFLRO2V52TK5ULWHXS6BKSFGLSQUSWSI4JFA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.555804167 (586h49m5.426887222s) [noise=true , address=145.79.227.32:4834 , piece_id=7VME5HIQC7YW32BEPS6TIHMQS4KKEKH3DSOZW5LILVEMHFKSA4VA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.562253107 (586h49m5.433336152s) [noise=true , address=145.79.227.36:4519 , piece_id=63CAL2PC7NLUHQXTRSN4UNUX477DYI2TGETWHFZGBXU67W3TFWYQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.762458634 (586h49m5.633541679s) [noise=true , address=45.91.116.5:4299 , piece_id=6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.773374660 (586h49m5.644457695s) [noise=true , address=45.91.116.12:4225 , piece_id=EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.786080053 (586h49m5.657163088s) [noise=true , address=77.90.31.18:4805 , piece_id=55OYHJT4XH5Y6HZRI7AWOVI4C5E6CXPZNL6K4GHAHBV2XT2IBBYA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.886238001 (586h49m5.757321046s) [noise=true , address=45.91.116.14:4191 , piece_id=BCJ5ROPBZW7PFQYOIUQJNRDDLZ4UHWE7LT63F7IVJU6KKREVQ35A, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.886259142 (586h49m5.757342187s) [seg_pos={0 0}, upload=514424]: Segment reached optimal threshold of 13 pieces. Dec 30 02:18:05.886436720 (586h49m5.757519765s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886442631 (586h49m5.757525666s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886511101 (586h49m5.757594146s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886531439 (586h49m5.757614474s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886740027 (586h49m5.757823062s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886748043 (586h49m5.757831078s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886767589 (586h49m5.757850624s) [seg_pos={0 0}, upload=514424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:18:05.886902897 (586h49m5.757985952s) [upload=514424]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:18:05.886912465 (586h49m5.7579955s) [upload=514424]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:18:06.167498600 (586h49m6.038581655s) [upload=514424]: Done uploading segments... Dec 30 02:18:06.167524630 (586h49m6.038607665s) [upload=514424]: Upload finished. err: <nil> Duration: 11.951469165s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X........................X......] duration: 3.51412215s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:19:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514153961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX.] duration: 5.847553419s trace: https://jaeger.dev.storj.io/trace/37e31797b0061bf3 FAIL (2025-12-30 02:18:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ) context deadline exceeded ======LOGS====== Dec 30 02:18:03.288068866 (586h49m3.159151911s) [upload=514424]: Starting upload Dec 30 02:18:03.288169066 (586h49m3.159252112s) [upload=514424]: Done starting upload Dec 30 02:18:03.288322318 (586h49m3.159405373s) [upload=514424]: Uploading segments... Dec 30 02:18:03.288487964 (586h49m3.159571009s) [upload=514424]: Got next segment. Inline: false Dec 30 02:18:03.288571452 (586h49m3.159654507s) [upload=514424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:18:03.288580379 (586h49m3.159663424s) [upload=514424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:18:03.531253932 (586h49m3.402336977s) [seg_pos={0 0}, upload=514424]: Begin upload segment... Dec 30 02:18:03.531326290 (586h49m3.402409335s) [seg_pos={0 0}, upload=514424]: Done begin upload segment. Dec 30 02:18:03.531364372 (586h49m3.402447417s) [noise=true , address=145.79.227.51:4410 , piece_id=ACIVJ6L4XUPJHV6IZAVKRRSHOTPMNJKWK2JSSCPUW7KB2DKC7AOA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.531440958 (586h49m3.402523993s) [noise=true , address=145.79.227.19:4078 , piece_id=SYHXAA63GI7SIIB77JLONRWUPBGZ2ABJLRZRCLW3Z4JHTAPDD7MQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532201977 (586h49m3.403285012s) [noise=true , address=45.91.116.5:4299 , piece_id=6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532427166 (586h49m3.403510211s) [noise=true , address=145.79.227.16:4911 , piece_id=NRIA24QRHAMXGQXFV5FD6WT5KSZSWBJCW2MM3LCKOY42PXKL2UMA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532874448 (586h49m3.403957483s) [noise=true , address=45.91.116.12:4225 , piece_id=EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533208073 (586h49m3.404291118s) [noise=true , address=45.91.116.14:4191 , piece_id=BCJ5ROPBZW7PFQYOIUQJNRDDLZ4UHWE7LT63F7IVJU6KKREVQ35A, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533446597 (586h49m3.404529642s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533533303 (586h49m3.404616338s) [noise=true , address=77.90.31.18:4805 , piece_id=55OYHJT4XH5Y6HZRI7AWOVI4C5E6CXPZNL6K4GHAHBV2XT2IBBYA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533922614 (586h49m3.405005659s) [noise=true , address=145.79.227.36:4519 , piece_id=63CAL2PC7NLUHQXTRSN4UNUX477DYI2TGETWHFZGBXU67W3TFWYQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534182819 (586h49m3.405265864s) [noise=true , address=145.79.227.17:4971 , piece_id=AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534306886 (586h49m3.405389931s) [noise=true , address=145.79.227.3:4151 , piece_id=HHAWFW7WFN4KRLULZV2PI3JARDBXBYLIGLC2SQ26M2JQ7563NGAQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534645760 (586h49m3.405728795s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534811366 (586h49m3.405894411s) [noise=true , address=145.79.227.32:4834 , piece_id=7VME5HIQC7YW32BEPS6TIHMQS4KKEKH3DSOZW5LILVEMHFKSA4VA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.535659571 (586h49m3.406742607s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.536519608 (586h49m3.407602654s) [noise=true , address=145.79.227.26:4654 , piece_id=FLGXCZIPI6USZVJ6IWOXCXFS5X3FQK7MWAYVYGMAE6ESU55ASG4Q, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.537229591 (586h49m3.408312636s) [noise=true , address=145.79.227.38:4979 , piece_id=N4PGG5GSKZPRTMECGFLRO2V52TK5ULWHXS6BKSFGLSQUSWSI4JFA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:05.184357438 (586h49m5.055440483s) [upload=514424]: Next returned nil segment Dec 30 02:18:05.184373058 (586h49m5.055456093s) [upload=514424]: Waiting for error group managing segments... Dec 30 02:18:05.526931920 (586h49m5.398014965s) [noise=true , address=145.79.227.51:4410 , piece_id=ACIVJ6L4XUPJHV6IZAVKRRSHOTPMNJKWK2JSSCPUW7KB2DKC7AOA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.529517132 (586h49m5.400600167s) [noise=true , address=145.79.227.16:4911 , piece_id=NRIA24QRHAMXGQXFV5FD6WT5KSZSWBJCW2MM3LCKOY42PXKL2UMA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.544046158 (586h49m5.415129193s) [noise=true , address=145.79.227.26:4654 , piece_id=FLGXCZIPI6USZVJ6IWOXCXFS5X3FQK7MWAYVYGMAE6ESU55ASG4Q, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.545510566 (586h49m5.416593601s) [noise=true , address=145.79.227.3:4151 , piece_id=HHAWFW7WFN4KRLULZV2PI3JARDBXBYLIGLC2SQ26M2JQ7563NGAQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.545568056 (586h49m5.416651101s) [noise=true , address=145.79.227.19:4078 , piece_id=SYHXAA63GI7SIIB77JLONRWUPBGZ2ABJLRZRCLW3Z4JHTAPDD7MQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.547341713 (586h49m5.418424759s) [noise=true , address=145.79.227.17:4971 , piece_id=AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.551635028 (586h49m5.422718073s) [noise=true , address=145.79.227.38:4979 , piece_id=N4PGG5GSKZPRTMECGFLRO2V52TK5ULWHXS6BKSFGLSQUSWSI4JFA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.555804167 (586h49m5.426887222s) [noise=true , address=145.79.227.32:4834 , piece_id=7VME5HIQC7YW32BEPS6TIHMQS4KKEKH3DSOZW5LILVEMHFKSA4VA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.562253107 (586h49m5.433336152s) [noise=true , address=145.79.227.36:4519 , piece_id=63CAL2PC7NLUHQXTRSN4UNUX477DYI2TGETWHFZGBXU67W3TFWYQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.762458634 (586h49m5.633541679s) [noise=true , address=45.91.116.5:4299 , piece_id=6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.773374660 (586h49m5.644457695s) [noise=true , address=45.91.116.12:4225 , piece_id=EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.786080053 (586h49m5.657163088s) [noise=true , address=77.90.31.18:4805 , piece_id=55OYHJT4XH5Y6HZRI7AWOVI4C5E6CXPZNL6K4GHAHBV2XT2IBBYA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.886238001 (586h49m5.757321046s) [noise=true , address=45.91.116.14:4191 , piece_id=BCJ5ROPBZW7PFQYOIUQJNRDDLZ4UHWE7LT63F7IVJU6KKREVQ35A, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.886259142 (586h49m5.757342187s) [seg_pos={0 0}, upload=514424]: Segment reached optimal threshold of 13 pieces. Dec 30 02:18:05.886436720 (586h49m5.757519765s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886442631 (586h49m5.757525666s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886511101 (586h49m5.757594146s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886531439 (586h49m5.757614474s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886740027 (586h49m5.757823062s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886748043 (586h49m5.757831078s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886767589 (586h49m5.757850624s) [seg_pos={0 0}, upload=514424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:18:05.886902897 (586h49m5.757985952s) [upload=514424]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:18:05.886912465 (586h49m5.7579955s) [upload=514424]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:18:06.167498600 (586h49m6.038581655s) [upload=514424]: Done uploading segments... Dec 30 02:18:06.167524630 (586h49m6.038607665s) [upload=514424]: Upload finished. err: <nil> Duration: 5.847582343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXXX] duration: 30.00337769s trace: https://jaeger.dev.storj.io/trace/37e31797b0061bf3 FAIL (2025-12-30 02:18:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ) context deadline exceeded ======LOGS====== Dec 30 02:18:03.288068866 (586h49m3.159151911s) [upload=514424]: Starting upload Dec 30 02:18:03.288169066 (586h49m3.159252112s) [upload=514424]: Done starting upload Dec 30 02:18:03.288322318 (586h49m3.159405373s) [upload=514424]: Uploading segments... Dec 30 02:18:03.288487964 (586h49m3.159571009s) [upload=514424]: Got next segment. Inline: false Dec 30 02:18:03.288571452 (586h49m3.159654507s) [upload=514424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:18:03.288580379 (586h49m3.159663424s) [upload=514424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:18:03.531253932 (586h49m3.402336977s) [seg_pos={0 0}, upload=514424]: Begin upload segment... Dec 30 02:18:03.531326290 (586h49m3.402409335s) [seg_pos={0 0}, upload=514424]: Done begin upload segment. Dec 30 02:18:03.531364372 (586h49m3.402447417s) [noise=true , address=145.79.227.51:4410 , piece_id=ACIVJ6L4XUPJHV6IZAVKRRSHOTPMNJKWK2JSSCPUW7KB2DKC7AOA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.531440958 (586h49m3.402523993s) [noise=true , address=145.79.227.19:4078 , piece_id=SYHXAA63GI7SIIB77JLONRWUPBGZ2ABJLRZRCLW3Z4JHTAPDD7MQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532201977 (586h49m3.403285012s) [noise=true , address=45.91.116.5:4299 , piece_id=6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532427166 (586h49m3.403510211s) [noise=true , address=145.79.227.16:4911 , piece_id=NRIA24QRHAMXGQXFV5FD6WT5KSZSWBJCW2MM3LCKOY42PXKL2UMA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.532874448 (586h49m3.403957483s) [noise=true , address=45.91.116.12:4225 , piece_id=EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533208073 (586h49m3.404291118s) [noise=true , address=45.91.116.14:4191 , piece_id=BCJ5ROPBZW7PFQYOIUQJNRDDLZ4UHWE7LT63F7IVJU6KKREVQ35A, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533446597 (586h49m3.404529642s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533533303 (586h49m3.404616338s) [noise=true , address=77.90.31.18:4805 , piece_id=55OYHJT4XH5Y6HZRI7AWOVI4C5E6CXPZNL6K4GHAHBV2XT2IBBYA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.533922614 (586h49m3.405005659s) [noise=true , address=145.79.227.36:4519 , piece_id=63CAL2PC7NLUHQXTRSN4UNUX477DYI2TGETWHFZGBXU67W3TFWYQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534182819 (586h49m3.405265864s) [noise=true , address=145.79.227.17:4971 , piece_id=AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534306886 (586h49m3.405389931s) [noise=true , address=145.79.227.3:4151 , piece_id=HHAWFW7WFN4KRLULZV2PI3JARDBXBYLIGLC2SQ26M2JQ7563NGAQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534645760 (586h49m3.405728795s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.534811366 (586h49m3.405894411s) [noise=true , address=145.79.227.32:4834 , piece_id=7VME5HIQC7YW32BEPS6TIHMQS4KKEKH3DSOZW5LILVEMHFKSA4VA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.535659571 (586h49m3.406742607s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.536519608 (586h49m3.407602654s) [noise=true , address=145.79.227.26:4654 , piece_id=FLGXCZIPI6USZVJ6IWOXCXFS5X3FQK7MWAYVYGMAE6ESU55ASG4Q, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:03.537229591 (586h49m3.408312636s) [noise=true , address=145.79.227.38:4979 , piece_id=N4PGG5GSKZPRTMECGFLRO2V52TK5ULWHXS6BKSFGLSQUSWSI4JFA, seg_pos={0 0}, upload=514424]: Uploading piece... Dec 30 02:18:05.184357438 (586h49m5.055440483s) [upload=514424]: Next returned nil segment Dec 30 02:18:05.184373058 (586h49m5.055456093s) [upload=514424]: Waiting for error group managing segments... Dec 30 02:18:05.526931920 (586h49m5.398014965s) [noise=true , address=145.79.227.51:4410 , piece_id=ACIVJ6L4XUPJHV6IZAVKRRSHOTPMNJKWK2JSSCPUW7KB2DKC7AOA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.529517132 (586h49m5.400600167s) [noise=true , address=145.79.227.16:4911 , piece_id=NRIA24QRHAMXGQXFV5FD6WT5KSZSWBJCW2MM3LCKOY42PXKL2UMA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.544046158 (586h49m5.415129193s) [noise=true , address=145.79.227.26:4654 , piece_id=FLGXCZIPI6USZVJ6IWOXCXFS5X3FQK7MWAYVYGMAE6ESU55ASG4Q, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.545510566 (586h49m5.416593601s) [noise=true , address=145.79.227.3:4151 , piece_id=HHAWFW7WFN4KRLULZV2PI3JARDBXBYLIGLC2SQ26M2JQ7563NGAQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.545568056 (586h49m5.416651101s) [noise=true , address=145.79.227.19:4078 , piece_id=SYHXAA63GI7SIIB77JLONRWUPBGZ2ABJLRZRCLW3Z4JHTAPDD7MQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.547341713 (586h49m5.418424759s) [noise=true , address=145.79.227.17:4971 , piece_id=AIOVZGB6V2EWLNNJBISHLQXIZ36DVHOD7PV5M6TXGR36NZZI7WLA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.551635028 (586h49m5.422718073s) [noise=true , address=145.79.227.38:4979 , piece_id=N4PGG5GSKZPRTMECGFLRO2V52TK5ULWHXS6BKSFGLSQUSWSI4JFA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.555804167 (586h49m5.426887222s) [noise=true , address=145.79.227.32:4834 , piece_id=7VME5HIQC7YW32BEPS6TIHMQS4KKEKH3DSOZW5LILVEMHFKSA4VA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.562253107 (586h49m5.433336152s) [noise=true , address=145.79.227.36:4519 , piece_id=63CAL2PC7NLUHQXTRSN4UNUX477DYI2TGETWHFZGBXU67W3TFWYQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.762458634 (586h49m5.633541679s) [noise=true , address=45.91.116.5:4299 , piece_id=6LVFPIX5K4CTQW5MPHTTWXTNZ7ORV4UCVY4RL4CRBC45GPKLEOCQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.773374660 (586h49m5.644457695s) [noise=true , address=45.91.116.12:4225 , piece_id=EAQ4XBX6TGEJO2JELJYC2RWLXJMNI4OD44SY6SLSRIHJ5YRWPODQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.786080053 (586h49m5.657163088s) [noise=true , address=77.90.31.18:4805 , piece_id=55OYHJT4XH5Y6HZRI7AWOVI4C5E6CXPZNL6K4GHAHBV2XT2IBBYA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.886238001 (586h49m5.757321046s) [noise=true , address=45.91.116.14:4191 , piece_id=BCJ5ROPBZW7PFQYOIUQJNRDDLZ4UHWE7LT63F7IVJU6KKREVQ35A, seg_pos={0 0}, upload=514424]: Done uploading piece. err: <nil> Dec 30 02:18:05.886259142 (586h49m5.757342187s) [seg_pos={0 0}, upload=514424]: Segment reached optimal threshold of 13 pieces. Dec 30 02:18:05.886436720 (586h49m5.757519765s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886442631 (586h49m5.757525666s) [noise=true , address=77.90.31.13:4629 , piece_id=AQUD6OHVI7VZPPMGSKTMFW4VT2E6YY25L4IAXKLP256PRLNYSWYQ, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886511101 (586h49m5.757594146s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886531439 (586h49m5.757614474s) [noise=true , address=145.79.227.49:4339 , piece_id=OK5TE62DTUNDLPAB6UPQBP5KOROZZLCCYXKE32CO2VC5VN4REPEA, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886740027 (586h49m5.757823062s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 02:18:05.886748043 (586h49m5.757831078s) [noise=true , address=145.79.227.44:4195 , piece_id=7DELKIRO7R53EHQRJF6E3WBNGAVEILTVG5TI2LKPNFD5NET6ETMA, seg_pos={0 0}, upload=514424]: Long tail cancellation due to optimal threshold Dec 30 02:18:05.886767589 (586h49m5.757850624s) [seg_pos={0 0}, upload=514424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:18:05.886902897 (586h49m5.757985952s) [upload=514424]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:18:05.886912465 (586h49m5.7579955s) [upload=514424]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:18:06.167498600 (586h49m6.038581655s) [upload=514424]: Done uploading segments... Dec 30 02:18:06.167524630 (586h49m6.038607665s) [upload=514424]: Upload finished. err: <nil> Duration: 30.003407316s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X........................X.....] duration: 3.416245045s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:18:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416261516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XXX] duration: 30.002712704s trace: https://jaeger.dev.storj.io/trace/52715b25115cf94c FAIL (2025-12-30 02:17:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:17:04.582973152 (586h48m4.454056198s) [upload=514410]: Starting upload Dec 30 02:17:04.583065899 (586h48m4.454148945s) [upload=514410]: Done starting upload Dec 30 02:17:04.583163054 (586h48m4.454246099s) [upload=514410]: Uploading segments... Dec 30 02:17:04.583369006 (586h48m4.454452061s) [upload=514410]: Got next segment. Inline: false Dec 30 02:17:04.583412760 (586h48m4.454495805s) [upload=514410]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:17:04.583421276 (586h48m4.454504321s) [upload=514410]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:17:04.828578207 (586h48m4.699661263s) [seg_pos={0 0}, upload=514410]: Begin upload segment... Dec 30 02:17:04.828636077 (586h48m4.699719122s) [seg_pos={0 0}, upload=514410]: Done begin upload segment. Dec 30 02:17:04.828672076 (586h48m4.699755121s) [noise=true , address=45.91.116.4:4937 , piece_id=ABIU67UQPEFJL555BTR6DWZRLAY5C6PUVH5YWODVO4SMNNH6KDTQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.828853021 (586h48m4.699936066s) [noise=true , address=77.90.31.15:4700 , piece_id=TFB5J6GU3SWP6IOXV6CI7O7BVSE5WDWRXN2SEE2DFDEIEM3A4OLA, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.829488451 (586h48m4.700571496s) [noise=true , address=145.79.227.19:4079 , piece_id=FPTYGWXXDJPPDFH74UC2BO7NICTUXAMCYIDJOP4S4DXR5PNBITMQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.830137577 (586h48m4.701220622s) [noise=true , address=45.91.116.11:4119 , piece_id=WI5WNDBBOIRX4CK2AX2UI3CFHKBHQH3BBPG4KXW36BBRRFLTMG6A, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.830331556 (586h48m4.701414601s) [noise=true , address=145.79.227.5:4222 , piece_id=4Z5WBZ25RCWT5U57LJDGSGDR7YORKUEBKWX4A7QCYH6M2WF2DHNA, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.830629754 (586h48m4.701712809s) [noise=true , address=45.91.116.5:4298 , piece_id=SNHNMAIODSGO3ITFL4P4DJMMDFUCUM7G3YEGPPXRB4TIXZDGTSEQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.831011922 (586h48m4.702094967s) [noise=true , address=145.79.227.16:4912 , piece_id=2ICNSWHPLRZK36L74VCXLWO23QXCZFXTR5XCWUT4LQ2VS7U2Z56Q, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.831179691 (586h48m4.702262736s) [noise=true , address=145.79.227.22:4510 , piece_id=JYTFWSO57TL2KBC2V4ZBP6UHPMXUYTJAA2RCDG3JLMHBZVBPCSUA, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.831433214 (586h48m4.702516249s) [noise=true , address=145.79.227.49:4338 , piece_id=XUY3RWKGYFNOQWK7IVGFUIVFMQWQSDV6FR6SWILZ2UQ6WHMY576Q, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.831623215 (586h48m4.70270626s) [noise=true , address=145.79.227.3:4151 , piece_id=HW3NHFFIX2QHCZEOXOFDHVQJOAHGQK3LM6DI2ZJACJOK3JODYDAA, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.831892498 (586h48m4.702975543s) [noise=true , address=145.79.227.42:4122 , piece_id=5HZAORLB7SLOALLV2SHVFW3XUKMCL334IJB2LOC55DRRDY5D6XBQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.832136002 (586h48m4.703219047s) [noise=true , address=145.79.227.36:4518 , piece_id=YFBSWKQFUOLY74PC4ADMRRN6FHM366J3MY4NELOVV5QK55JOYNXA, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.832267843 (586h48m4.703350888s) [noise=true , address=77.90.31.21:4880 , piece_id=WDQKVHPLKLE3WRHY4R4V7U3SKHVKCLIJBTPZHVL4G763MF45QZAQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.832623410 (586h48m4.703706445s) [noise=true , address=45.91.116.8:4046 , piece_id=WOM7RTKLUNJLVLTRXVKCPZ2EU5WJ7YUW55RFB5L2TE4XJIPFZ6BA, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.833086682 (586h48m4.704169727s) [noise=true , address=145.79.227.11:4731 , piece_id=HGFU2D6EQTSATXFP5DEXJ3W5FNPJLELOHGEV24AL2NVQXPIY3UJQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:04.833553281 (586h48m4.704636326s) [noise=true , address=145.79.227.46:4602 , piece_id=3GKJFBFNSQGSTQ2NCP24OICTWQLCJ7THWGUC3ZKPBFHKWHHMANKQ, seg_pos={0 0}, upload=514410]: Uploading piece... Dec 30 02:17:06.435819273 (586h48m6.306902308s) [upload=514410]: Next returned nil segment Dec 30 02:17:06.435832679 (586h48m6.306915714s) [upload=514410]: Waiting for error group managing segments... Dec 30 02:17:06.817389590 (586h48m6.688472625s) [noise=true , address=145.79.227.11:4731 , piece_id=HGFU2D6EQTSATXFP5DEXJ3W5FNPJLELOHGEV24AL2NVQXPIY3UJQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.824639145 (586h48m6.69572219s) [noise=true , address=145.79.227.5:4222 , piece_id=4Z5WBZ25RCWT5U57LJDGSGDR7YORKUEBKWX4A7QCYH6M2WF2DHNA, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.825002887 (586h48m6.696085922s) [noise=true , address=145.79.227.3:4151 , piece_id=HW3NHFFIX2QHCZEOXOFDHVQJOAHGQK3LM6DI2ZJACJOK3JODYDAA, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.826543310 (586h48m6.697626345s) [noise=true , address=145.79.227.22:4510 , piece_id=JYTFWSO57TL2KBC2V4ZBP6UHPMXUYTJAA2RCDG3JLMHBZVBPCSUA, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.834544646 (586h48m6.705627681s) [noise=true , address=145.79.227.46:4602 , piece_id=3GKJFBFNSQGSTQ2NCP24OICTWQLCJ7THWGUC3ZKPBFHKWHHMANKQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.839418436 (586h48m6.710501482s) [noise=true , address=145.79.227.16:4912 , piece_id=2ICNSWHPLRZK36L74VCXLWO23QXCZFXTR5XCWUT4LQ2VS7U2Z56Q, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.847940464 (586h48m6.719023499s) [noise=true , address=145.79.227.42:4122 , piece_id=5HZAORLB7SLOALLV2SHVFW3XUKMCL334IJB2LOC55DRRDY5D6XBQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:06.853661758 (586h48m6.724744803s) [noise=true , address=145.79.227.36:4518 , piece_id=YFBSWKQFUOLY74PC4ADMRRN6FHM366J3MY4NELOVV5QK55JOYNXA, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:07.047187014 (586h48m6.918270059s) [noise=true , address=45.91.116.8:4046 , piece_id=WOM7RTKLUNJLVLTRXVKCPZ2EU5WJ7YUW55RFB5L2TE4XJIPFZ6BA, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:07.079291146 (586h48m6.950374191s) [noise=true , address=77.90.31.21:4880 , piece_id=WDQKVHPLKLE3WRHY4R4V7U3SKHVKCLIJBTPZHVL4G763MF45QZAQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:07.086986689 (586h48m6.958069734s) [noise=true , address=77.90.31.15:4700 , piece_id=TFB5J6GU3SWP6IOXV6CI7O7BVSE5WDWRXN2SEE2DFDEIEM3A4OLA, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:07.398548778 (586h48m7.269631813s) [noise=true , address=45.91.116.4:4937 , piece_id=ABIU67UQPEFJL555BTR6DWZRLAY5C6PUVH5YWODVO4SMNNH6KDTQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:07.583914366 (586h48m7.454997411s) [noise=true , address=45.91.116.11:4119 , piece_id=WI5WNDBBOIRX4CK2AX2UI3CFHKBHQH3BBPG4KXW36BBRRFLTMG6A, seg_pos={0 0}, upload=514410]: Done uploading piece. err: <nil> Dec 30 02:17:07.583937901 (586h48m7.455020947s) [seg_pos={0 0}, upload=514410]: Segment reached optimal threshold of 13 pieces. Dec 30 02:17:07.584186775 (586h48m7.45526982s) [noise=true , address=145.79.227.49:4338 , piece_id=XUY3RWKGYFNOQWK7IVGFUIVFMQWQSDV6FR6SWILZ2UQ6WHMY576Q, seg_pos={0 0}, upload=514410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 02:17:07.584197105 (586h48m7.45528015s) [noise=true , address=145.79.227.49:4338 , piece_id=XUY3RWKGYFNOQWK7IVGFUIVFMQWQSDV6FR6SWILZ2UQ6WHMY576Q, seg_pos={0 0}, upload=514410]: Long tail cancellation due to optimal threshold Dec 30 02:17:07.584236479 (586h48m7.455319524s) [noise=true , address=45.91.116.5:4298 , piece_id=SNHNMAIODSGO3ITFL4P4DJMMDFUCUM7G3YEGPPXRB4TIXZDGTSEQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 02:17:07.584245878 (586h48m7.455328923s) [noise=true , address=45.91.116.5:4298 , piece_id=SNHNMAIODSGO3ITFL4P4DJMMDFUCUM7G3YEGPPXRB4TIXZDGTSEQ, seg_pos={0 0}, upload=514410]: Long tail cancellation due to optimal threshold Dec 30 02:17:07.584369272 (586h48m7.455452318s) [noise=true , address=145.79.227.19:4079 , piece_id=FPTYGWXXDJPPDFH74UC2BO7NICTUXAMCYIDJOP4S4DXR5PNBITMQ, seg_pos={0 0}, upload=514410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:60852->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:60852->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60852->145.79.227.19:4079: use of closed network connection Dec 30 02:17:07.584377748 (586h48m7.455460793s) [noise=true , address=145.79.227.19:4079 , piece_id=FPTYGWXXDJPPDFH74UC2BO7NICTUXAMCYIDJOP4S4DXR5PNBITMQ, seg_pos={0 0}, upload=514410]: Long tail cancellation due to optimal threshold Dec 30 02:17:07.584400883 (586h48m7.455483918s) [seg_pos={0 0}, upload=514410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:17:07.584482047 (586h48m7.455565102s) [upload=514410]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:17:07.584488941 (586h48m7.455571976s) [upload=514410]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:17:07.881261306 (586h48m7.752344351s) [upload=514410]: Done uploading segments... Dec 30 02:17:07.881285813 (586h48m7.752368848s) [upload=514410]: Upload finished. err: <nil> Duration: 30.002739463s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X........................X....] duration: 3.565136008s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:16:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565176926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X........................X...] duration: 3.63792438s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:16:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637952574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.XX] duration: 30.003265285s trace: https://jaeger.dev.storj.io/trace/5ce79efc8171e87a FAIL (2025-12-30 02:16:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: FVFXYFG7PHXZ343HVVWQGJPIC6GKNKFJHCLXNCHSBZTKZZI6JKIQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: UNRPPHPACKUXBZV6S7GEWUICY65IPCQYE4KF5SXIYYSAQ72S64JQ) context deadline exceeded; piecestore: (Node ID: 1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, Piece ID: 335XPONZXLQL4IFLTBKRPCPVDANHYTIFZPM5RJWOVR6QOHAQRFGQ) context deadline exceeded ======LOGS====== ec 30 02:15:53.477390327 (586h46m53.348473372s) [upload=514394]: Starting upload Dec 30 02:15:53.477503082 (586h46m53.348586127s) [upload=514394]: Done starting upload Dec 30 02:15:53.477582863 (586h46m53.348665908s) [upload=514394]: Uploading segments... Dec 30 02:15:53.477750152 (586h46m53.348833197s) [upload=514394]: Got next segment. Inline: false Dec 30 02:15:53.477870341 (586h46m53.348953397s) [upload=514394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:15:53.477877214 (586h46m53.348960249s) [upload=514394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:15:53.721666622 (586h46m53.592749667s) [seg_pos={0 0}, upload=514394]: Begin upload segment... Dec 30 02:15:53.721723149 (586h46m53.592806184s) [seg_pos={0 0}, upload=514394]: Done begin upload segment. Dec 30 02:15:53.721833449 (586h46m53.592916484s) [noise=true , address=45.91.116.13:4155 , piece_id=FVFXYFG7PHXZ343HVVWQGJPIC6GKNKFJHCLXNCHSBZTKZZI6JKIQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.721840233 (586h46m53.592923268s) [noise=true , address=145.79.227.17:4971 , piece_id=GRWZDDTKCC3OAHCQWZHPPJMMU2QSOT2FMNJLW2K747AO4YEVIKNQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.722640757 (586h46m53.593723792s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXBYSVBAOTBCPFXKPZVRKN2IE3IO3F5P4E76BBJ74WQIAV5FRUQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.723078049 (586h46m53.594161094s) [noise=true , address=145.79.227.31:4798 , piece_id=HQM7HZ3FP73MPIDY5EHJT5SAAQPEOUB7SD6DUJM6YWPRSBDXIRCQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.723512888 (586h46m53.594595953s) [noise=true , address=77.90.31.14:4662 , piece_id=PL7ZXA5WXMW2R3XXLOV3P5TZ4RBQBVVLFCENRLW7EDDJUFYP5GYQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.723535340 (586h46m53.594618385s) [noise=true , address=145.79.227.49:4337 , piece_id=JOOVM77RSDU4ZIRVAXOZWCHQG7BPUIKTSC5MQVWU72X6GH4UE6SA, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.723891989 (586h46m53.594975024s) [noise=true , address=45.91.116.14:4190 , piece_id=PPX2LMFZDVJYNUZKGNGCRBHWF72YS2XUDJGPB4LLFLIMYYYTSSOQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.724015584 (586h46m53.59509863s) [noise=true , address=145.79.227.46:4603 , piece_id=Y34HMX6UYM2ZLD4YEAXXRRQV7O5AQSDMIWLK5D25JULAAGML7CCA, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.724191139 (586h46m53.595274184s) [noise=true , address=45.91.116.10:4084 , piece_id=UNRPPHPACKUXBZV6S7GEWUICY65IPCQYE4KF5SXIYYSAQ72S64JQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.724408973 (586h46m53.595492008s) [noise=true , address=145.79.227.7:4293 , piece_id=AVU3HLGF2HMAVKSBZMPC57ANEZWYJ6ZWZKMNPUC5CJL74QSVODDQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.724618923 (586h46m53.595701968s) [noise=true , address=145.79.227.6:4259 , piece_id=335XPONZXLQL4IFLTBKRPCPVDANHYTIFZPM5RJWOVR6QOHAQRFGQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.724781002 (586h46m53.595864037s) [noise=true , address=145.79.227.15:4876 , piece_id=IJEKARMP3FLKF2D4SKPN5CHKQ7QFUJRZBNTT62GMAKRKWH2PBTVQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.725028593 (586h46m53.596111638s) [noise=true , address=145.79.227.36:4519 , piece_id=SWTJBN26NOFU6XMBDKROB7XVFEVZ35ER6V5LC222W47RDIVC4ETQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.725296193 (586h46m53.596379238s) [noise=true , address=145.79.227.19:4078 , piece_id=HNKYFV2FZ7BDXNFRU7LJ25JCVRIJFJRQIINSRM54I2NQ2XZJFJ4A, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.725415270 (586h46m53.596498315s) [noise=true , address=145.79.227.45:4229 , piece_id=VWZ6KKPVITYR222KAIYTPNKCOPM3FL7DNZAOORCUVMA2ODIKJLMQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:53.725742744 (586h46m53.596825779s) [noise=true , address=145.79.227.2:4114 , piece_id=NIGWU6RBEUYALCWZOUBOJ26ITWDMX5EFIW34G2TDRD3AOA75ZILQ, seg_pos={0 0}, upload=514394]: Uploading piece... Dec 30 02:15:55.413248187 (586h46m55.284331222s) [upload=514394]: Next returned nil segment Dec 30 02:15:55.413275920 (586h46m55.284358955s) [upload=514394]: Waiting for error group managing segments... Dec 30 02:15:55.699530789 (586h46m55.570613834s) [noise=true , address=145.79.227.31:4798 , piece_id=HQM7HZ3FP73MPIDY5EHJT5SAAQPEOUB7SD6DUJM6YWPRSBDXIRCQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.704933256 (586h46m55.576016291s) [noise=true , address=145.79.227.15:4876 , piece_id=IJEKARMP3FLKF2D4SKPN5CHKQ7QFUJRZBNTT62GMAKRKWH2PBTVQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.708477816 (586h46m55.579560861s) [noise=true , address=145.79.227.19:4078 , piece_id=HNKYFV2FZ7BDXNFRU7LJ25JCVRIJFJRQIINSRM54I2NQ2XZJFJ4A, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.712170317 (586h46m55.583253352s) [noise=true , address=145.79.227.6:4259 , piece_id=335XPONZXLQL4IFLTBKRPCPVDANHYTIFZPM5RJWOVR6QOHAQRFGQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.720388065 (586h46m55.59147111s) [noise=true , address=145.79.227.46:4603 , piece_id=Y34HMX6UYM2ZLD4YEAXXRRQV7O5AQSDMIWLK5D25JULAAGML7CCA, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.724768436 (586h46m55.595851481s) [noise=true , address=145.79.227.17:4971 , piece_id=GRWZDDTKCC3OAHCQWZHPPJMMU2QSOT2FMNJLW2K747AO4YEVIKNQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.729002409 (586h46m55.600085454s) [noise=true , address=145.79.227.36:4519 , piece_id=SWTJBN26NOFU6XMBDKROB7XVFEVZ35ER6V5LC222W47RDIVC4ETQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.729217127 (586h46m55.600300173s) [noise=true , address=145.79.227.45:4229 , piece_id=VWZ6KKPVITYR222KAIYTPNKCOPM3FL7DNZAOORCUVMA2ODIKJLMQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.729455982 (586h46m55.600539027s) [noise=true , address=145.79.227.7:4293 , piece_id=AVU3HLGF2HMAVKSBZMPC57ANEZWYJ6ZWZKMNPUC5CJL74QSVODDQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.782331276 (586h46m55.653414311s) [noise=true , address=145.79.227.2:4114 , piece_id=NIGWU6RBEUYALCWZOUBOJ26ITWDMX5EFIW34G2TDRD3AOA75ZILQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.903216131 (586h46m55.774299166s) [noise=true , address=77.90.31.14:4662 , piece_id=PL7ZXA5WXMW2R3XXLOV3P5TZ4RBQBVVLFCENRLW7EDDJUFYP5GYQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.995183153 (586h46m55.866266198s) [noise=true , address=45.91.116.13:4155 , piece_id=FVFXYFG7PHXZ343HVVWQGJPIC6GKNKFJHCLXNCHSBZTKZZI6JKIQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.995649731 (586h46m55.866732776s) [noise=true , address=45.91.116.10:4084 , piece_id=UNRPPHPACKUXBZV6S7GEWUICY65IPCQYE4KF5SXIYYSAQ72S64JQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: <nil> Dec 30 02:15:55.995662325 (586h46m55.86674537s) [seg_pos={0 0}, upload=514394]: Segment reached optimal threshold of 13 pieces. Dec 30 02:15:55.996019035 (586h46m55.86710208s) [noise=true , address=45.91.116.14:4190 , piece_id=PPX2LMFZDVJYNUZKGNGCRBHWF72YS2XUDJGPB4LLFLIMYYYTSSOQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 02:15:55.996030036 (586h46m55.867113071s) [noise=true , address=45.91.116.14:4190 , piece_id=PPX2LMFZDVJYNUZKGNGCRBHWF72YS2XUDJGPB4LLFLIMYYYTSSOQ, seg_pos={0 0}, upload=514394]: Long tail cancellation due to optimal threshold Dec 30 02:15:55.996199639 (586h46m55.867282674s) [noise=true , address=145.79.227.49:4337 , piece_id=JOOVM77RSDU4ZIRVAXOZWCHQG7BPUIKTSC5MQVWU72X6GH4UE6SA, seg_pos={0 0}, upload=514394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 02:15:55.996208565 (586h46m55.86729161s) [noise=true , address=145.79.227.49:4337 , piece_id=JOOVM77RSDU4ZIRVAXOZWCHQG7BPUIKTSC5MQVWU72X6GH4UE6SA, seg_pos={0 0}, upload=514394]: Long tail cancellation due to optimal threshold Dec 30 02:15:55.996236078 (586h46m55.867319123s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXBYSVBAOTBCPFXKPZVRKN2IE3IO3F5P4E76BBJ74WQIAV5FRUQ, seg_pos={0 0}, upload=514394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:52964->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:52964->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->77.90.31.11:4553: use of closed network connection Dec 30 02:15:55.996253350 (586h46m55.867336395s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXBYSVBAOTBCPFXKPZVRKN2IE3IO3F5P4E76BBJ74WQIAV5FRUQ, seg_pos={0 0}, upload=514394]: Long tail cancellation due to optimal threshold Dec 30 02:15:55.996288517 (586h46m55.867371562s) [seg_pos={0 0}, upload=514394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:15:55.996389299 (586h46m55.867472334s) [upload=514394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:15:55.996394639 (586h46m55.867477674s) [upload=514394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:15:56.283546246 (586h46m56.154629281s) [upload=514394]: Done uploading segments... Dec 30 02:15:56.283570793 (586h46m56.154653838s) [upload=514394]: Upload finished. err: <nil> Duration: 30.003282758s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X........................X..] duration: 5.001224993s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:16:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00131807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.X] duration: 30.003375124s trace: https://jaeger.dev.storj.io/trace/502e30b6d7829ae3 FAIL (2025-12-30 02:15:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 02:15:16.786687929 (586h46m16.657770974s) [upload=514387]: Starting upload Dec 30 02:15:16.786789172 (586h46m16.657872217s) [upload=514387]: Done starting upload Dec 30 02:15:16.786877109 (586h46m16.657960164s) [upload=514387]: Uploading segments... Dec 30 02:15:16.787148927 (586h46m16.658231982s) [upload=514387]: Got next segment. Inline: false Dec 30 02:15:16.787193452 (586h46m16.658276487s) [upload=514387]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:15:16.787199453 (586h46m16.658282498s) [upload=514387]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:15:17.032007811 (586h46m16.903090856s) [seg_pos={0 0}, upload=514387]: Begin upload segment... Dec 30 02:15:17.032057145 (586h46m16.90314019s) [seg_pos={0 0}, upload=514387]: Done begin upload segment. Dec 30 02:15:17.032086651 (586h46m16.903169686s) [noise=true , address=77.90.31.19:4844 , piece_id=GEZXHVW77Y5XO5QKYWGPBUWEFJN3XD4AOLKCKVHF4J5TPKKNSWEA, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.032333842 (586h46m16.903416877s) [noise=true , address=145.79.227.42:4123 , piece_id=7B2NN2XA7HCO2F6R4ZDL3OACOHZNVVWRQGLXJZVQR55BVOT5ROZA, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.032795080 (586h46m16.903878125s) [noise=true , address=145.79.227.47:4266 , piece_id=MJMJHQJITHJ3TSJISFMHNPZEPBCE7ZIKDE7AHM67FVXEK3AM2VCQ, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.033170164 (586h46m16.904253209s) [noise=true , address=145.79.227.5:4223 , piece_id=Y66LBYTZM5UCFK7F4O6CRQMCY4V4BDO4FXY3U27WI2QS5EN6FDBQ, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.033210651 (586h46m16.904293696s) [noise=true , address=77.90.31.12:4589 , piece_id=KBEMA7OXSGSGC67XXNK7UNCLBF7BCHPRZEG4MYKZY7VICFGH5UCA, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.033554686 (586h46m16.904637731s) [noise=true , address=77.90.31.13:4627 , piece_id=BJMNEMNVHQVIHA6H3TYMOTJ3OFUPSMWBSJUPI5BBOCH7QBUTHJ2Q, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.033764566 (586h46m16.904847611s) [noise=true , address=145.79.227.43:4157 , piece_id=MCXBGSUKDXNUUI4ZJZXVJK6GQDGTYYEYJM2TWUIAWUCJCBBQ3TYQ, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.034151953 (586h46m16.905234988s) [noise=true , address=145.79.227.13:4803 , piece_id=H7GQ6OWG4SZIUEYAMQ5PD5CXCQBZBENBV3H4LW6TWXSSA3HVRQRA, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.034466332 (586h46m16.905549367s) [noise=true , address=145.79.227.21:4475 , piece_id=FNMIYZ6QOQF7WFZZYUW2YJC7WRXDL7FSUO62TF7AYQ2EHTYXSW5A, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.034482674 (586h46m16.905565709s) [noise=true , address=145.79.227.11:4730 , piece_id=MTGPSX4A4JKTAAHZHVG3IUJWQ5VI7WL7MFMZE4LGZ3K3YMFYVVKQ, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.034833802 (586h46m16.905916847s) [noise=true , address=77.90.31.21:4880 , piece_id=HF2EAWXZGBIVL55UCYV4TCPRMEWITF7QMQEZR4H2SNPG5FYXF7OA, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.035023683 (586h46m16.906106718s) [noise=true , address=45.91.116.12:4226 , piece_id=EU2FUXTZEM3JH7DI3KCQPBAQVFGUIAXYI6W5UYQYX2JGJTCV6VZQ, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.035226410 (586h46m16.906309445s) [noise=true , address=145.79.227.14:4838 , piece_id=LIF26V7AHFLBLFNTKSFXIEWNRK7PBTD7FE2RB25IZRLFR3KYDXOA, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.035562861 (586h46m16.906645906s) [noise=true , address=45.91.116.5:4298 , piece_id=EKMWRLM6OQ5NZCUQ56ASCHCD2LEZUZB5UL7PQ52LVYFKNKFTDR2A, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.036178032 (586h46m16.907261067s) [noise=true , address=145.79.227.52:4447 , piece_id=RADOGCGLUPD4ABFPJ56U5QTU5IQ36L4IWTIBP73RRCW6FYYIBK3A, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:17.036692221 (586h46m16.907775256s) [noise=true , address=45.91.116.6:4976 , piece_id=N4EN7ULMZFJ6OBJ5SAO5YDFQMLY7H6TVSJH2UBSQGWQWRJMQLYYQ, seg_pos={0 0}, upload=514387]: Uploading piece... Dec 30 02:15:18.632148897 (586h46m18.503231932s) [upload=514387]: Next returned nil segment Dec 30 02:15:18.632166742 (586h46m18.503249777s) [upload=514387]: Waiting for error group managing segments... Dec 30 02:15:19.012343134 (586h46m18.883426179s) [noise=true , address=145.79.227.11:4730 , piece_id=MTGPSX4A4JKTAAHZHVG3IUJWQ5VI7WL7MFMZE4LGZ3K3YMFYVVKQ, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.018124663 (586h46m18.889207698s) [noise=true , address=145.79.227.14:4838 , piece_id=LIF26V7AHFLBLFNTKSFXIEWNRK7PBTD7FE2RB25IZRLFR3KYDXOA, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.018922051 (586h46m18.890005096s) [noise=true , address=145.79.227.21:4475 , piece_id=FNMIYZ6QOQF7WFZZYUW2YJC7WRXDL7FSUO62TF7AYQ2EHTYXSW5A, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.020211827 (586h46m18.891294872s) [noise=true , address=145.79.227.42:4123 , piece_id=7B2NN2XA7HCO2F6R4ZDL3OACOHZNVVWRQGLXJZVQR55BVOT5ROZA, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.020900789 (586h46m18.891983834s) [noise=true , address=145.79.227.13:4803 , piece_id=H7GQ6OWG4SZIUEYAMQ5PD5CXCQBZBENBV3H4LW6TWXSSA3HVRQRA, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.041027103 (586h46m18.912110148s) [noise=true , address=145.79.227.52:4447 , piece_id=RADOGCGLUPD4ABFPJ56U5QTU5IQ36L4IWTIBP73RRCW6FYYIBK3A, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.043946512 (586h46m18.915029557s) [noise=true , address=145.79.227.47:4266 , piece_id=MJMJHQJITHJ3TSJISFMHNPZEPBCE7ZIKDE7AHM67FVXEK3AM2VCQ, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.048420481 (586h46m18.919503526s) [noise=true , address=145.79.227.5:4223 , piece_id=Y66LBYTZM5UCFK7F4O6CRQMCY4V4BDO4FXY3U27WI2QS5EN6FDBQ, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.222658700 (586h46m19.093741735s) [noise=true , address=77.90.31.21:4880 , piece_id=HF2EAWXZGBIVL55UCYV4TCPRMEWITF7QMQEZR4H2SNPG5FYXF7OA, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.320499376 (586h46m19.191582421s) [noise=true , address=77.90.31.13:4627 , piece_id=BJMNEMNVHQVIHA6H3TYMOTJ3OFUPSMWBSJUPI5BBOCH7QBUTHJ2Q, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.340556810 (586h46m19.211639855s) [noise=true , address=77.90.31.19:4844 , piece_id=GEZXHVW77Y5XO5QKYWGPBUWEFJN3XD4AOLKCKVHF4J5TPKKNSWEA, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:19.993716329 (586h46m19.864799364s) [noise=true , address=45.91.116.5:4298 , piece_id=EKMWRLM6OQ5NZCUQ56ASCHCD2LEZUZB5UL7PQ52LVYFKNKFTDR2A, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:20.050832768 (586h46m19.921915813s) [noise=true , address=45.91.116.6:4976 , piece_id=N4EN7ULMZFJ6OBJ5SAO5YDFQMLY7H6TVSJH2UBSQGWQWRJMQLYYQ, seg_pos={0 0}, upload=514387]: Done uploading piece. err: <nil> Dec 30 02:15:20.050858908 (586h46m19.921941943s) [seg_pos={0 0}, upload=514387]: Segment reached optimal threshold of 13 pieces. Dec 30 02:15:20.051059590 (586h46m19.922142625s) [noise=true , address=77.90.31.12:4589 , piece_id=KBEMA7OXSGSGC67XXNK7UNCLBF7BCHPRZEG4MYKZY7VICFGH5UCA, seg_pos={0 0}, upload=514387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 02:15:20.051069549 (586h46m19.922152594s) [noise=true , address=77.90.31.12:4589 , piece_id=KBEMA7OXSGSGC67XXNK7UNCLBF7BCHPRZEG4MYKZY7VICFGH5UCA, seg_pos={0 0}, upload=514387]: Long tail cancellation due to optimal threshold Dec 30 02:15:20.051141195 (586h46m19.922224241s) [noise=true , address=45.91.116.12:4226 , piece_id=EU2FUXTZEM3JH7DI3KCQPBAQVFGUIAXYI6W5UYQYX2JGJTCV6VZQ, seg_pos={0 0}, upload=514387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 02:15:20.051162907 (586h46m19.922245942s) [noise=true , address=45.91.116.12:4226 , piece_id=EU2FUXTZEM3JH7DI3KCQPBAQVFGUIAXYI6W5UYQYX2JGJTCV6VZQ, seg_pos={0 0}, upload=514387]: Long tail cancellation due to optimal threshold Dec 30 02:15:20.051337720 (586h46m19.922420755s) [noise=true , address=145.79.227.43:4157 , piece_id=MCXBGSUKDXNUUI4ZJZXVJK6GQDGTYYEYJM2TWUIAWUCJCBBQ3TYQ, seg_pos={0 0}, upload=514387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 02:15:20.051344653 (586h46m19.922427698s) [noise=true , address=145.79.227.43:4157 , piece_id=MCXBGSUKDXNUUI4ZJZXVJK6GQDGTYYEYJM2TWUIAWUCJCBBQ3TYQ, seg_pos={0 0}, upload=514387]: Long tail cancellation due to optimal threshold Dec 30 02:15:20.051361114 (586h46m19.922444149s) [seg_pos={0 0}, upload=514387]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:15:20.051428142 (586h46m19.922511187s) [upload=514387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:15:20.051433252 (586h46m19.922516297s) [upload=514387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:15:20.345780631 (586h46m20.216863666s) [upload=514387]: Done uploading segments... Dec 30 02:15:20.345843521 (586h46m20.216926566s) [upload=514387]: Upload finished. err: <nil> Duration: 30.003402136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X......X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X.] duration: 6.664627931s trace: https://jaeger.dev.storj.io/trace/79f3a5d976a8c9fa FAIL (2025-12-30 02:15:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:14:22.512787774 (586h45m22.383870809s) [upload=514372]: Starting upload Dec 30 02:14:22.512846587 (586h45m22.383929632s) [upload=514372]: Done starting upload Dec 30 02:14:22.512924615 (586h45m22.38400766s) [upload=514372]: Uploading segments... Dec 30 02:14:22.513022612 (586h45m22.384105647s) [upload=514372]: Got next segment. Inline: false Dec 30 02:14:22.513064672 (586h45m22.384147707s) [upload=514372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:14:22.513069521 (586h45m22.384152566s) [upload=514372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:14:22.761600617 (586h45m22.632683652s) [seg_pos={0 0}, upload=514372]: Begin upload segment... Dec 30 02:14:22.761650542 (586h45m22.632733587s) [seg_pos={0 0}, upload=514372]: Done begin upload segment. Dec 30 02:14:22.761678365 (586h45m22.6327614s) [noise=true , address=145.79.227.33:4871 , piece_id=YIYSEXUTY3MNHGJ4ZLUEZTIIEKQZBU32ZU3BVP462Q7J4QAINIYA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.761699456 (586h45m22.632782491s) [noise=true , address=145.79.227.10:4401 , piece_id=DHXV56MQMDNO4A2LOHW4YUIDCWTIMDKYXMSSI3OKQXM7AOBCSZZQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.762170071 (586h45m22.633253106s) [noise=true , address=145.79.227.27:4689 , piece_id=SF3PUQZCEXNODSQE2XNL47KA66W2JFDYLA6ZZHPBST7GFZVT7G5A, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.762470945 (586h45m22.63355398s) [noise=true , address=77.90.31.21:4880 , piece_id=P4GUBGYPDAASFPQQKBYRAO2C4ZV6YV36FGZUSFLNLUEWQKUZEEKA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.762742362 (586h45m22.633825407s) [noise=true , address=145.79.227.26:4653 , piece_id=UPNS7RQXE2BGR4WPXVS6WI4C2KXSK64LZPICCLO4AITZRGTTS2WQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763005212 (586h45m22.634088247s) [noise=true , address=145.79.227.17:4972 , piece_id=34B6HM23MNPYOG74YSRFUKRTT3DKY6WHTXLLON2JP3F4NSLWMMOQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763297568 (586h45m22.634380603s) [noise=true , address=145.79.227.16:4912 , piece_id=C7LMDZEEVSCJXSYLCA5LRFRKNXRMMCHQUWQP6653GBFGEYLCZGRA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763303200 (586h45m22.634386245s) [noise=true , address=45.91.116.13:4153 , piece_id=XDRGNX3HR3LJSI3IPKUPUHCNDGOOIS5J7SLPZZPZPVO2FFSL5FNQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763622718 (586h45m22.634705753s) [noise=true , address=145.79.227.48:4303 , piece_id=4WGDFFZUH2GU26B7DIFU2MK2FUDEYVPIH4YLOAUEEI7PSMWH4DKQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763754900 (586h45m22.634837935s) [noise=true , address=45.91.116.9:4263 , piece_id=LAGUWA64QYDBF62EBW3HCQ3OCPL3C2SH34TMPE7TZE25J2MKEO2A, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763967865 (586h45m22.63505091s) [noise=true , address=145.79.227.7:4293 , piece_id=UW4U7L55OKKQ355O77QZ2CNJD6UA26KTLBZ3QLD3R4VB7MQ7HX5A, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764166693 (586h45m22.635249728s) [noise=true , address=45.91.116.14:4192 , piece_id=QE33FY4MVXTOO342ACMMWBC5NUEU5IXNODFJ6BAUUBVERW5H6CTA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764294507 (586h45m22.635377542s) [noise=true , address=145.79.227.50:4375 , piece_id=AIUNXBFFZZYTHFKQHAFFWJW36EH5N2N6RX3WGY4N4P3ER2EXYBDA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764616239 (586h45m22.635699274s) [noise=true , address=145.79.227.41:4085 , piece_id=XU7OVUBDAR2WYXNW5622YVCSURGJ42QVDH5GEZQ76C2NM3I5M2VA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764953962 (586h45m22.636036998s) [noise=true , address=145.79.227.46:4601 , piece_id=PXJ7XG2CEQUCQTHVNJCAWMXJTNIB66AJIPFGWWEVRZGLIOEU5MSA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.765281726 (586h45m22.636364761s) [noise=true , address=145.79.227.18:4007 , piece_id=SDEPLXT5KOGF7MJ6V3LKFIAGXPH57MCWXPHEZRJUOT22QWXQETOQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:24.412375049 (586h45m24.283458084s) [upload=514372]: Next returned nil segment Dec 30 02:14:24.412395979 (586h45m24.283479014s) [upload=514372]: Waiting for error group managing segments... Dec 30 02:14:24.742545842 (586h45m24.613628887s) [noise=true , address=145.79.227.16:4912 , piece_id=C7LMDZEEVSCJXSYLCA5LRFRKNXRMMCHQUWQP6653GBFGEYLCZGRA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.744511925 (586h45m24.61559497s) [noise=true , address=145.79.227.17:4972 , piece_id=34B6HM23MNPYOG74YSRFUKRTT3DKY6WHTXLLON2JP3F4NSLWMMOQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.747222417 (586h45m24.618305452s) [noise=true , address=145.79.227.50:4375 , piece_id=AIUNXBFFZZYTHFKQHAFFWJW36EH5N2N6RX3WGY4N4P3ER2EXYBDA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.748888489 (586h45m24.619971534s) [noise=true , address=145.79.227.48:4303 , piece_id=4WGDFFZUH2GU26B7DIFU2MK2FUDEYVPIH4YLOAUEEI7PSMWH4DKQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.752754932 (586h45m24.623837967s) [noise=true , address=145.79.227.46:4601 , piece_id=PXJ7XG2CEQUCQTHVNJCAWMXJTNIB66AJIPFGWWEVRZGLIOEU5MSA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.757219082 (586h45m24.628302127s) [noise=true , address=145.79.227.41:4085 , piece_id=XU7OVUBDAR2WYXNW5622YVCSURGJ42QVDH5GEZQ76C2NM3I5M2VA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.767586894 (586h45m24.638669929s) [noise=true , address=145.79.227.27:4689 , piece_id=SF3PUQZCEXNODSQE2XNL47KA66W2JFDYLA6ZZHPBST7GFZVT7G5A, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.769660142 (586h45m24.640743187s) [noise=true , address=145.79.227.10:4401 , piece_id=DHXV56MQMDNO4A2LOHW4YUIDCWTIMDKYXMSSI3OKQXM7AOBCSZZQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.770534167 (586h45m24.641617202s) [noise=true , address=145.79.227.26:4653 , piece_id=UPNS7RQXE2BGR4WPXVS6WI4C2KXSK64LZPICCLO4AITZRGTTS2WQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.782425409 (586h45m24.653508454s) [noise=true , address=145.79.227.7:4293 , piece_id=UW4U7L55OKKQ355O77QZ2CNJD6UA26KTLBZ3QLD3R4VB7MQ7HX5A, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.934214242 (586h45m24.805297277s) [noise=true , address=77.90.31.21:4880 , piece_id=P4GUBGYPDAASFPQQKBYRAO2C4ZV6YV36FGZUSFLNLUEWQKUZEEKA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:25.177783419 (586h45m25.048866455s) [noise=true , address=45.91.116.13:4153 , piece_id=XDRGNX3HR3LJSI3IPKUPUHCNDGOOIS5J7SLPZZPZPVO2FFSL5FNQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:26.386609845 (586h45m26.25769288s) [noise=true , address=145.79.227.33:4871 , piece_id=YIYSEXUTY3MNHGJ4ZLUEZTIIEKQZBU32ZU3BVP462Q7J4QAINIYA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:26.386630124 (586h45m26.257713169s) [seg_pos={0 0}, upload=514372]: Segment reached optimal threshold of 13 pieces. Dec 30 02:14:26.386861966 (586h45m26.257945011s) [noise=true , address=45.91.116.14:4192 , piece_id=QE33FY4MVXTOO342ACMMWBC5NUEU5IXNODFJ6BAUUBVERW5H6CTA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 02:14:26.386872205 (586h45m26.25795526s) [noise=true , address=45.91.116.14:4192 , piece_id=QE33FY4MVXTOO342ACMMWBC5NUEU5IXNODFJ6BAUUBVERW5H6CTA, seg_pos={0 0}, upload=514372]: Long tail cancellation due to optimal threshold Dec 30 02:14:26.386896021 (586h45m26.257979056s) [noise=true , address=45.91.116.9:4263 , piece_id=LAGUWA64QYDBF62EBW3HCQ3OCPL3C2SH34TMPE7TZE25J2MKEO2A, seg_pos={0 0}, upload=514372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 02:14:26.386924394 (586h45m26.258007459s) [noise=true , address=45.91.116.9:4263 , piece_id=LAGUWA64QYDBF62EBW3HCQ3OCPL3C2SH34TMPE7TZE25J2MKEO2A, seg_pos={0 0}, upload=514372]: Long tail cancellation due to optimal threshold Dec 30 02:14:26.387021399 (586h45m26.258104444s) [noise=true , address=145.79.227.18:4007 , piece_id=SDEPLXT5KOGF7MJ6V3LKFIAGXPH57MCWXPHEZRJUOT22QWXQETOQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:35262->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35262->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35262->145.79.227.18:4007: use of closed network connection Dec 30 02:14:26.387031197 (586h45m26.258114232s) [noise=true , address=145.79.227.18:4007 , piece_id=SDEPLXT5KOGF7MJ6V3LKFIAGXPH57MCWXPHEZRJUOT22QWXQETOQ, seg_pos={0 0}, upload=514372]: Long tail cancellation due to optimal threshold Dec 30 02:14:26.387050805 (586h45m26.25813386s) [seg_pos={0 0}, upload=514372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:14:26.387127341 (586h45m26.258210376s) [upload=514372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:14:26.387134554 (586h45m26.258217609s) [upload=514372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:14:26.690151492 (586h45m26.561234527s) [upload=514372]: Done uploading segments... Dec 30 02:14:26.690170337 (586h45m26.561253382s) [upload=514372]: Upload finished. err: <nil> Duration: 6.664664279s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X........................X.] duration: 3.47417835s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:15:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474204851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..X] duration: 30.003175193s trace: https://jaeger.dev.storj.io/trace/79f3a5d976a8c9fa FAIL (2025-12-30 02:14:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:14:22.512787774 (586h45m22.383870809s) [upload=514372]: Starting upload Dec 30 02:14:22.512846587 (586h45m22.383929632s) [upload=514372]: Done starting upload Dec 30 02:14:22.512924615 (586h45m22.38400766s) [upload=514372]: Uploading segments... Dec 30 02:14:22.513022612 (586h45m22.384105647s) [upload=514372]: Got next segment. Inline: false Dec 30 02:14:22.513064672 (586h45m22.384147707s) [upload=514372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:14:22.513069521 (586h45m22.384152566s) [upload=514372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:14:22.761600617 (586h45m22.632683652s) [seg_pos={0 0}, upload=514372]: Begin upload segment... Dec 30 02:14:22.761650542 (586h45m22.632733587s) [seg_pos={0 0}, upload=514372]: Done begin upload segment. Dec 30 02:14:22.761678365 (586h45m22.6327614s) [noise=true , address=145.79.227.33:4871 , piece_id=YIYSEXUTY3MNHGJ4ZLUEZTIIEKQZBU32ZU3BVP462Q7J4QAINIYA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.761699456 (586h45m22.632782491s) [noise=true , address=145.79.227.10:4401 , piece_id=DHXV56MQMDNO4A2LOHW4YUIDCWTIMDKYXMSSI3OKQXM7AOBCSZZQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.762170071 (586h45m22.633253106s) [noise=true , address=145.79.227.27:4689 , piece_id=SF3PUQZCEXNODSQE2XNL47KA66W2JFDYLA6ZZHPBST7GFZVT7G5A, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.762470945 (586h45m22.63355398s) [noise=true , address=77.90.31.21:4880 , piece_id=P4GUBGYPDAASFPQQKBYRAO2C4ZV6YV36FGZUSFLNLUEWQKUZEEKA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.762742362 (586h45m22.633825407s) [noise=true , address=145.79.227.26:4653 , piece_id=UPNS7RQXE2BGR4WPXVS6WI4C2KXSK64LZPICCLO4AITZRGTTS2WQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763005212 (586h45m22.634088247s) [noise=true , address=145.79.227.17:4972 , piece_id=34B6HM23MNPYOG74YSRFUKRTT3DKY6WHTXLLON2JP3F4NSLWMMOQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763297568 (586h45m22.634380603s) [noise=true , address=145.79.227.16:4912 , piece_id=C7LMDZEEVSCJXSYLCA5LRFRKNXRMMCHQUWQP6653GBFGEYLCZGRA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763303200 (586h45m22.634386245s) [noise=true , address=45.91.116.13:4153 , piece_id=XDRGNX3HR3LJSI3IPKUPUHCNDGOOIS5J7SLPZZPZPVO2FFSL5FNQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763622718 (586h45m22.634705753s) [noise=true , address=145.79.227.48:4303 , piece_id=4WGDFFZUH2GU26B7DIFU2MK2FUDEYVPIH4YLOAUEEI7PSMWH4DKQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763754900 (586h45m22.634837935s) [noise=true , address=45.91.116.9:4263 , piece_id=LAGUWA64QYDBF62EBW3HCQ3OCPL3C2SH34TMPE7TZE25J2MKEO2A, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.763967865 (586h45m22.63505091s) [noise=true , address=145.79.227.7:4293 , piece_id=UW4U7L55OKKQ355O77QZ2CNJD6UA26KTLBZ3QLD3R4VB7MQ7HX5A, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764166693 (586h45m22.635249728s) [noise=true , address=45.91.116.14:4192 , piece_id=QE33FY4MVXTOO342ACMMWBC5NUEU5IXNODFJ6BAUUBVERW5H6CTA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764294507 (586h45m22.635377542s) [noise=true , address=145.79.227.50:4375 , piece_id=AIUNXBFFZZYTHFKQHAFFWJW36EH5N2N6RX3WGY4N4P3ER2EXYBDA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764616239 (586h45m22.635699274s) [noise=true , address=145.79.227.41:4085 , piece_id=XU7OVUBDAR2WYXNW5622YVCSURGJ42QVDH5GEZQ76C2NM3I5M2VA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.764953962 (586h45m22.636036998s) [noise=true , address=145.79.227.46:4601 , piece_id=PXJ7XG2CEQUCQTHVNJCAWMXJTNIB66AJIPFGWWEVRZGLIOEU5MSA, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:22.765281726 (586h45m22.636364761s) [noise=true , address=145.79.227.18:4007 , piece_id=SDEPLXT5KOGF7MJ6V3LKFIAGXPH57MCWXPHEZRJUOT22QWXQETOQ, seg_pos={0 0}, upload=514372]: Uploading piece... Dec 30 02:14:24.412375049 (586h45m24.283458084s) [upload=514372]: Next returned nil segment Dec 30 02:14:24.412395979 (586h45m24.283479014s) [upload=514372]: Waiting for error group managing segments... Dec 30 02:14:24.742545842 (586h45m24.613628887s) [noise=true , address=145.79.227.16:4912 , piece_id=C7LMDZEEVSCJXSYLCA5LRFRKNXRMMCHQUWQP6653GBFGEYLCZGRA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.744511925 (586h45m24.61559497s) [noise=true , address=145.79.227.17:4972 , piece_id=34B6HM23MNPYOG74YSRFUKRTT3DKY6WHTXLLON2JP3F4NSLWMMOQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.747222417 (586h45m24.618305452s) [noise=true , address=145.79.227.50:4375 , piece_id=AIUNXBFFZZYTHFKQHAFFWJW36EH5N2N6RX3WGY4N4P3ER2EXYBDA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.748888489 (586h45m24.619971534s) [noise=true , address=145.79.227.48:4303 , piece_id=4WGDFFZUH2GU26B7DIFU2MK2FUDEYVPIH4YLOAUEEI7PSMWH4DKQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.752754932 (586h45m24.623837967s) [noise=true , address=145.79.227.46:4601 , piece_id=PXJ7XG2CEQUCQTHVNJCAWMXJTNIB66AJIPFGWWEVRZGLIOEU5MSA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.757219082 (586h45m24.628302127s) [noise=true , address=145.79.227.41:4085 , piece_id=XU7OVUBDAR2WYXNW5622YVCSURGJ42QVDH5GEZQ76C2NM3I5M2VA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.767586894 (586h45m24.638669929s) [noise=true , address=145.79.227.27:4689 , piece_id=SF3PUQZCEXNODSQE2XNL47KA66W2JFDYLA6ZZHPBST7GFZVT7G5A, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.769660142 (586h45m24.640743187s) [noise=true , address=145.79.227.10:4401 , piece_id=DHXV56MQMDNO4A2LOHW4YUIDCWTIMDKYXMSSI3OKQXM7AOBCSZZQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.770534167 (586h45m24.641617202s) [noise=true , address=145.79.227.26:4653 , piece_id=UPNS7RQXE2BGR4WPXVS6WI4C2KXSK64LZPICCLO4AITZRGTTS2WQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.782425409 (586h45m24.653508454s) [noise=true , address=145.79.227.7:4293 , piece_id=UW4U7L55OKKQ355O77QZ2CNJD6UA26KTLBZ3QLD3R4VB7MQ7HX5A, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:24.934214242 (586h45m24.805297277s) [noise=true , address=77.90.31.21:4880 , piece_id=P4GUBGYPDAASFPQQKBYRAO2C4ZV6YV36FGZUSFLNLUEWQKUZEEKA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:25.177783419 (586h45m25.048866455s) [noise=true , address=45.91.116.13:4153 , piece_id=XDRGNX3HR3LJSI3IPKUPUHCNDGOOIS5J7SLPZZPZPVO2FFSL5FNQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:26.386609845 (586h45m26.25769288s) [noise=true , address=145.79.227.33:4871 , piece_id=YIYSEXUTY3MNHGJ4ZLUEZTIIEKQZBU32ZU3BVP462Q7J4QAINIYA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: <nil> Dec 30 02:14:26.386630124 (586h45m26.257713169s) [seg_pos={0 0}, upload=514372]: Segment reached optimal threshold of 13 pieces. Dec 30 02:14:26.386861966 (586h45m26.257945011s) [noise=true , address=45.91.116.14:4192 , piece_id=QE33FY4MVXTOO342ACMMWBC5NUEU5IXNODFJ6BAUUBVERW5H6CTA, seg_pos={0 0}, upload=514372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 02:14:26.386872205 (586h45m26.25795526s) [noise=true , address=45.91.116.14:4192 , piece_id=QE33FY4MVXTOO342ACMMWBC5NUEU5IXNODFJ6BAUUBVERW5H6CTA, seg_pos={0 0}, upload=514372]: Long tail cancellation due to optimal threshold Dec 30 02:14:26.386896021 (586h45m26.257979056s) [noise=true , address=45.91.116.9:4263 , piece_id=LAGUWA64QYDBF62EBW3HCQ3OCPL3C2SH34TMPE7TZE25J2MKEO2A, seg_pos={0 0}, upload=514372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 02:14:26.386924394 (586h45m26.258007459s) [noise=true , address=45.91.116.9:4263 , piece_id=LAGUWA64QYDBF62EBW3HCQ3OCPL3C2SH34TMPE7TZE25J2MKEO2A, seg_pos={0 0}, upload=514372]: Long tail cancellation due to optimal threshold Dec 30 02:14:26.387021399 (586h45m26.258104444s) [noise=true , address=145.79.227.18:4007 , piece_id=SDEPLXT5KOGF7MJ6V3LKFIAGXPH57MCWXPHEZRJUOT22QWXQETOQ, seg_pos={0 0}, upload=514372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:35262->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:35262->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35262->145.79.227.18:4007: use of closed network connection Dec 30 02:14:26.387031197 (586h45m26.258114232s) [noise=true , address=145.79.227.18:4007 , piece_id=SDEPLXT5KOGF7MJ6V3LKFIAGXPH57MCWXPHEZRJUOT22QWXQETOQ, seg_pos={0 0}, upload=514372]: Long tail cancellation due to optimal threshold Dec 30 02:14:26.387050805 (586h45m26.25813386s) [seg_pos={0 0}, upload=514372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:14:26.387127341 (586h45m26.258210376s) [upload=514372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:14:26.387134554 (586h45m26.258217609s) [upload=514372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:14:26.690151492 (586h45m26.561234527s) [upload=514372]: Done uploading segments... Dec 30 02:14:26.690170337 (586h45m26.561253382s) [upload=514372]: Upload finished. err: <nil> Duration: 30.003203878s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X........................X] duration: 5.00141279s trace: https://jaeger.dev.storj.io/trace/56bb2b2c8bb1cfa5 FAIL (2025-12-30 02:14:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00142895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX..] duration: 5.98575621s trace: https://jaeger.dev.storj.io/trace/3b45eb49677368d6 FAIL (2025-12-30 02:14:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: UCZYLSFYTXOPKHC7HGUXMPPLDO7CQ7VSSIT75HDS3SZQPTBOPETQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3FPRTWUV2UKDDBUCGXYYWHLT74WM7ICCLNAMTWSGLT5XR7LOEZNA) context deadline exceeded ======LOGS====== Dec 30 02:12:05.527613018 (586h43m5.398696053s) [upload=514338]: Starting upload Dec 30 02:12:05.527701816 (586h43m5.398784851s) [upload=514338]: Done starting upload Dec 30 02:12:05.527894123 (586h43m5.398977168s) [upload=514338]: Uploading segments... Dec 30 02:12:05.528002559 (586h43m5.399085594s) [upload=514338]: Got next segment. Inline: false Dec 30 02:12:05.528085948 (586h43m5.399168993s) [upload=514338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:12:05.528094094 (586h43m5.399177129s) [upload=514338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:12:05.770785715 (586h43m5.64186876s) [seg_pos={0 0}, upload=514338]: Begin upload segment... Dec 30 02:12:05.770838034 (586h43m5.641921079s) [seg_pos={0 0}, upload=514338]: Done begin upload segment. Dec 30 02:12:05.770873341 (586h43m5.641956386s) [noise=true , address=77.90.31.18:4805 , piece_id=NR62LQE4VE3G7DOSW6ONBTNNJXDTPP4UKOMFHOHE7B3FR3LJPQQQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.771003660 (586h43m5.642086695s) [noise=true , address=77.90.31.17:4773 , piece_id=JDTIDAQ37JIFLWBTY7XGLHH2KGK7AYIV52T6YDBURGCIGY43LQSA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.771767534 (586h43m5.642850579s) [noise=true , address=145.79.227.7:4295 , piece_id=NVZDQ4GXUWIZFCUY5CAPMOAWBTRJQOPKFESMLMP3DCHJWEFU5CNQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.772286783 (586h43m5.643369818s) [noise=true , address=145.79.227.46:4603 , piece_id=USEZASR4L5QG66X6C5PPIBWYXOC74PBWESQEMBLL6HV3ADEG57FA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.772807084 (586h43m5.643890129s) [noise=true , address=145.79.227.11:4729 , piece_id=FUN57QVY5SLCNDNXWMTYBSP7QC5RUI4EJRHGN767MD7TDHR6QX7Q, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.773428046 (586h43m5.644511091s) [noise=true , address=45.91.116.5:4299 , piece_id=UCZYLSFYTXOPKHC7HGUXMPPLDO7CQ7VSSIT75HDS3SZQPTBOPETQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.774181460 (586h43m5.645264495s) [noise=true , address=145.79.227.38:4978 , piece_id=7NA3WNRT3GOVQSHMTOYOFPFG7KRF4G2QMCOBBIG7TARDGOXPAMUA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.775675425 (586h43m5.64675854s) [noise=true , address=145.79.227.45:4230 , piece_id=CG7723GTCD3MNWKCPVUHZUERV7RXZJUVJTV66IDYSXWWLKXH7XHQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.776628961 (586h43m5.647712006s) [noise=true , address=77.90.31.10:4517 , piece_id=IQ3J2YDS6B464VXSWQ4HYOKMZBXWLCWRIZPUSRPVBQ6BSQGH5X3Q, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.777418314 (586h43m5.648501359s) [noise=true , address=77.90.31.14:4662 , piece_id=3FPRTWUV2UKDDBUCGXYYWHLT74WM7ICCLNAMTWSGLT5XR7LOEZNA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.778814082 (586h43m5.649897117s) [noise=true , address=145.79.227.15:4874 , piece_id=UGPMISWEANLBBHV6JRH4HGLQSNFZYEEDJKBM5EY4PDCYCLZR7SMA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.778919592 (586h43m5.650002628s) [noise=true , address=145.79.227.17:4972 , piece_id=KSPOILQFHB6WRDJNGF2XYHGXWQBM5VVZDL2H4AE3FVIDIAP5RPJQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.780393388 (586h43m5.651476434s) [noise=true , address=145.79.227.16:4909 , piece_id=H5KU7IDVOACVSV6ZZGKA24B7K4IG2TD6M2DPWMB6CFTGGXEK5MPQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.781891431 (586h43m5.652974506s) [noise=true , address=145.79.227.2:4114 , piece_id=B3YU5BSNG63O5JZVQMJWWOAQ7JWZIR6KDNC55U23HATCNGTH5VGA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.782938004 (586h43m5.654021059s) [noise=true , address=45.91.116.7:4009 , piece_id=32GVSDO2O7JKWXYZYY5FND27C7JBGUQYABNS4L6R3POPXEXUG5YQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.784998668 (586h43m5.656081714s) [noise=true , address=145.79.227.47:4266 , piece_id=OOFBNKGXF6W33UKDDOELOJSPADK5OYRIRFN357IPDIJ5SWIYQ4RA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:07.470975521 (586h43m7.342058566s) [upload=514338]: Next returned nil segment Dec 30 02:12:07.471003454 (586h43m7.342086489s) [upload=514338]: Waiting for error group managing segments... Dec 30 02:12:07.754390787 (586h43m7.625473842s) [noise=true , address=145.79.227.45:4230 , piece_id=CG7723GTCD3MNWKCPVUHZUERV7RXZJUVJTV66IDYSXWWLKXH7XHQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.761704634 (586h43m7.632787679s) [noise=true , address=145.79.227.15:4874 , piece_id=UGPMISWEANLBBHV6JRH4HGLQSNFZYEEDJKBM5EY4PDCYCLZR7SMA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.761787441 (586h43m7.632870486s) [noise=true , address=145.79.227.46:4603 , piece_id=USEZASR4L5QG66X6C5PPIBWYXOC74PBWESQEMBLL6HV3ADEG57FA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.774395659 (586h43m7.645478694s) [noise=true , address=145.79.227.17:4972 , piece_id=KSPOILQFHB6WRDJNGF2XYHGXWQBM5VVZDL2H4AE3FVIDIAP5RPJQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.776349629 (586h43m7.647432675s) [noise=true , address=145.79.227.16:4909 , piece_id=H5KU7IDVOACVSV6ZZGKA24B7K4IG2TD6M2DPWMB6CFTGGXEK5MPQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.780842905 (586h43m7.65192594s) [noise=true , address=145.79.227.11:4729 , piece_id=FUN57QVY5SLCNDNXWMTYBSP7QC5RUI4EJRHGN767MD7TDHR6QX7Q, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.794451939 (586h43m7.665534974s) [noise=true , address=145.79.227.2:4114 , piece_id=B3YU5BSNG63O5JZVQMJWWOAQ7JWZIR6KDNC55U23HATCNGTH5VGA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.796793147 (586h43m7.667876182s) [noise=true , address=145.79.227.7:4295 , piece_id=NVZDQ4GXUWIZFCUY5CAPMOAWBTRJQOPKFESMLMP3DCHJWEFU5CNQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.811451157 (586h43m7.682534193s) [noise=true , address=145.79.227.47:4266 , piece_id=OOFBNKGXF6W33UKDDOELOJSPADK5OYRIRFN357IPDIJ5SWIYQ4RA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.858668237 (586h43m7.729751282s) [noise=true , address=145.79.227.38:4978 , piece_id=7NA3WNRT3GOVQSHMTOYOFPFG7KRF4G2QMCOBBIG7TARDGOXPAMUA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.011929238 (586h43m7.883012273s) [noise=true , address=45.91.116.5:4299 , piece_id=UCZYLSFYTXOPKHC7HGUXMPPLDO7CQ7VSSIT75HDS3SZQPTBOPETQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.031031072 (586h43m7.902114117s) [noise=true , address=77.90.31.18:4805 , piece_id=NR62LQE4VE3G7DOSW6ONBTNNJXDTPP4UKOMFHOHE7B3FR3LJPQQQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.056128861 (586h43m7.927211896s) [noise=true , address=77.90.31.14:4662 , piece_id=3FPRTWUV2UKDDBUCGXYYWHLT74WM7ICCLNAMTWSGLT5XR7LOEZNA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.056148869 (586h43m7.927231904s) [seg_pos={0 0}, upload=514338]: Segment reached optimal threshold of 13 pieces. Dec 30 02:12:08.056506020 (586h43m7.927589055s) [noise=true , address=77.90.31.17:4773 , piece_id=JDTIDAQ37JIFLWBTY7XGLHH2KGK7AYIV52T6YDBURGCIGY43LQSA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 30 02:12:08.056515788 (586h43m7.927598833s) [noise=true , address=77.90.31.17:4773 , piece_id=JDTIDAQ37JIFLWBTY7XGLHH2KGK7AYIV52T6YDBURGCIGY43LQSA, seg_pos={0 0}, upload=514338]: Long tail cancellation due to optimal threshold Dec 30 02:12:08.056661515 (586h43m7.92774456s) [noise=true , address=45.91.116.7:4009 , piece_id=32GVSDO2O7JKWXYZYY5FND27C7JBGUQYABNS4L6R3POPXEXUG5YQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:57620->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:57620->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57620->45.91.116.7:4009: use of closed network connection Dec 30 02:12:08.056678328 (586h43m7.927761373s) [noise=true , address=45.91.116.7:4009 , piece_id=32GVSDO2O7JKWXYZYY5FND27C7JBGUQYABNS4L6R3POPXEXUG5YQ, seg_pos={0 0}, upload=514338]: Long tail cancellation due to optimal threshold Dec 30 02:12:08.056759843 (586h43m7.927842878s) [noise=true , address=77.90.31.10:4517 , piece_id=IQ3J2YDS6B464VXSWQ4HYOKMZBXWLCWRIZPUSRPVBQ6BSQGH5X3Q, seg_pos={0 0}, upload=514338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 30 02:12:08.056768619 (586h43m7.927851654s) [noise=true , address=77.90.31.10:4517 , piece_id=IQ3J2YDS6B464VXSWQ4HYOKMZBXWLCWRIZPUSRPVBQ6BSQGH5X3Q, seg_pos={0 0}, upload=514338]: Long tail cancellation due to optimal threshold Dec 30 02:12:08.056787886 (586h43m7.927870931s) [seg_pos={0 0}, upload=514338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:12:08.056861456 (586h43m7.927944491s) [upload=514338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:12:08.056866646 (586h43m7.927949681s) [upload=514338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:12:08.343376083 (586h43m8.214459118s) [upload=514338]: Done uploading segments... Dec 30 02:12:08.343395971 (586h43m8.214479016s) [upload=514338]: Upload finished. err: <nil> Duration: 5.985786609s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.494854651s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:13:54 UTC) orbiter warning: context deadline exceeded Duration: 3.494880629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.487703936s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:13:28 UTC) orbiter warning: context deadline exceeded Duration: 3.48772738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X..X.XXX.] duration: 13.830904436s trace: https://jaeger.dev.storj.io/trace/3b45eb49677368d6 FAIL (2025-12-30 02:12:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: UCZYLSFYTXOPKHC7HGUXMPPLDO7CQ7VSSIT75HDS3SZQPTBOPETQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3FPRTWUV2UKDDBUCGXYYWHLT74WM7ICCLNAMTWSGLT5XR7LOEZNA) context deadline exceeded ======LOGS====== Dec 30 02:12:05.527613018 (586h43m5.398696053s) [upload=514338]: Starting upload Dec 30 02:12:05.527701816 (586h43m5.398784851s) [upload=514338]: Done starting upload Dec 30 02:12:05.527894123 (586h43m5.398977168s) [upload=514338]: Uploading segments... Dec 30 02:12:05.528002559 (586h43m5.399085594s) [upload=514338]: Got next segment. Inline: false Dec 30 02:12:05.528085948 (586h43m5.399168993s) [upload=514338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:12:05.528094094 (586h43m5.399177129s) [upload=514338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:12:05.770785715 (586h43m5.64186876s) [seg_pos={0 0}, upload=514338]: Begin upload segment... Dec 30 02:12:05.770838034 (586h43m5.641921079s) [seg_pos={0 0}, upload=514338]: Done begin upload segment. Dec 30 02:12:05.770873341 (586h43m5.641956386s) [noise=true , address=77.90.31.18:4805 , piece_id=NR62LQE4VE3G7DOSW6ONBTNNJXDTPP4UKOMFHOHE7B3FR3LJPQQQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.771003660 (586h43m5.642086695s) [noise=true , address=77.90.31.17:4773 , piece_id=JDTIDAQ37JIFLWBTY7XGLHH2KGK7AYIV52T6YDBURGCIGY43LQSA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.771767534 (586h43m5.642850579s) [noise=true , address=145.79.227.7:4295 , piece_id=NVZDQ4GXUWIZFCUY5CAPMOAWBTRJQOPKFESMLMP3DCHJWEFU5CNQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.772286783 (586h43m5.643369818s) [noise=true , address=145.79.227.46:4603 , piece_id=USEZASR4L5QG66X6C5PPIBWYXOC74PBWESQEMBLL6HV3ADEG57FA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.772807084 (586h43m5.643890129s) [noise=true , address=145.79.227.11:4729 , piece_id=FUN57QVY5SLCNDNXWMTYBSP7QC5RUI4EJRHGN767MD7TDHR6QX7Q, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.773428046 (586h43m5.644511091s) [noise=true , address=45.91.116.5:4299 , piece_id=UCZYLSFYTXOPKHC7HGUXMPPLDO7CQ7VSSIT75HDS3SZQPTBOPETQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.774181460 (586h43m5.645264495s) [noise=true , address=145.79.227.38:4978 , piece_id=7NA3WNRT3GOVQSHMTOYOFPFG7KRF4G2QMCOBBIG7TARDGOXPAMUA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.775675425 (586h43m5.64675854s) [noise=true , address=145.79.227.45:4230 , piece_id=CG7723GTCD3MNWKCPVUHZUERV7RXZJUVJTV66IDYSXWWLKXH7XHQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.776628961 (586h43m5.647712006s) [noise=true , address=77.90.31.10:4517 , piece_id=IQ3J2YDS6B464VXSWQ4HYOKMZBXWLCWRIZPUSRPVBQ6BSQGH5X3Q, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.777418314 (586h43m5.648501359s) [noise=true , address=77.90.31.14:4662 , piece_id=3FPRTWUV2UKDDBUCGXYYWHLT74WM7ICCLNAMTWSGLT5XR7LOEZNA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.778814082 (586h43m5.649897117s) [noise=true , address=145.79.227.15:4874 , piece_id=UGPMISWEANLBBHV6JRH4HGLQSNFZYEEDJKBM5EY4PDCYCLZR7SMA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.778919592 (586h43m5.650002628s) [noise=true , address=145.79.227.17:4972 , piece_id=KSPOILQFHB6WRDJNGF2XYHGXWQBM5VVZDL2H4AE3FVIDIAP5RPJQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.780393388 (586h43m5.651476434s) [noise=true , address=145.79.227.16:4909 , piece_id=H5KU7IDVOACVSV6ZZGKA24B7K4IG2TD6M2DPWMB6CFTGGXEK5MPQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.781891431 (586h43m5.652974506s) [noise=true , address=145.79.227.2:4114 , piece_id=B3YU5BSNG63O5JZVQMJWWOAQ7JWZIR6KDNC55U23HATCNGTH5VGA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.782938004 (586h43m5.654021059s) [noise=true , address=45.91.116.7:4009 , piece_id=32GVSDO2O7JKWXYZYY5FND27C7JBGUQYABNS4L6R3POPXEXUG5YQ, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:05.784998668 (586h43m5.656081714s) [noise=true , address=145.79.227.47:4266 , piece_id=OOFBNKGXF6W33UKDDOELOJSPADK5OYRIRFN357IPDIJ5SWIYQ4RA, seg_pos={0 0}, upload=514338]: Uploading piece... Dec 30 02:12:07.470975521 (586h43m7.342058566s) [upload=514338]: Next returned nil segment Dec 30 02:12:07.471003454 (586h43m7.342086489s) [upload=514338]: Waiting for error group managing segments... Dec 30 02:12:07.754390787 (586h43m7.625473842s) [noise=true , address=145.79.227.45:4230 , piece_id=CG7723GTCD3MNWKCPVUHZUERV7RXZJUVJTV66IDYSXWWLKXH7XHQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.761704634 (586h43m7.632787679s) [noise=true , address=145.79.227.15:4874 , piece_id=UGPMISWEANLBBHV6JRH4HGLQSNFZYEEDJKBM5EY4PDCYCLZR7SMA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.761787441 (586h43m7.632870486s) [noise=true , address=145.79.227.46:4603 , piece_id=USEZASR4L5QG66X6C5PPIBWYXOC74PBWESQEMBLL6HV3ADEG57FA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.774395659 (586h43m7.645478694s) [noise=true , address=145.79.227.17:4972 , piece_id=KSPOILQFHB6WRDJNGF2XYHGXWQBM5VVZDL2H4AE3FVIDIAP5RPJQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.776349629 (586h43m7.647432675s) [noise=true , address=145.79.227.16:4909 , piece_id=H5KU7IDVOACVSV6ZZGKA24B7K4IG2TD6M2DPWMB6CFTGGXEK5MPQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.780842905 (586h43m7.65192594s) [noise=true , address=145.79.227.11:4729 , piece_id=FUN57QVY5SLCNDNXWMTYBSP7QC5RUI4EJRHGN767MD7TDHR6QX7Q, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.794451939 (586h43m7.665534974s) [noise=true , address=145.79.227.2:4114 , piece_id=B3YU5BSNG63O5JZVQMJWWOAQ7JWZIR6KDNC55U23HATCNGTH5VGA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.796793147 (586h43m7.667876182s) [noise=true , address=145.79.227.7:4295 , piece_id=NVZDQ4GXUWIZFCUY5CAPMOAWBTRJQOPKFESMLMP3DCHJWEFU5CNQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.811451157 (586h43m7.682534193s) [noise=true , address=145.79.227.47:4266 , piece_id=OOFBNKGXF6W33UKDDOELOJSPADK5OYRIRFN357IPDIJ5SWIYQ4RA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:07.858668237 (586h43m7.729751282s) [noise=true , address=145.79.227.38:4978 , piece_id=7NA3WNRT3GOVQSHMTOYOFPFG7KRF4G2QMCOBBIG7TARDGOXPAMUA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.011929238 (586h43m7.883012273s) [noise=true , address=45.91.116.5:4299 , piece_id=UCZYLSFYTXOPKHC7HGUXMPPLDO7CQ7VSSIT75HDS3SZQPTBOPETQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.031031072 (586h43m7.902114117s) [noise=true , address=77.90.31.18:4805 , piece_id=NR62LQE4VE3G7DOSW6ONBTNNJXDTPP4UKOMFHOHE7B3FR3LJPQQQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.056128861 (586h43m7.927211896s) [noise=true , address=77.90.31.14:4662 , piece_id=3FPRTWUV2UKDDBUCGXYYWHLT74WM7ICCLNAMTWSGLT5XR7LOEZNA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: <nil> Dec 30 02:12:08.056148869 (586h43m7.927231904s) [seg_pos={0 0}, upload=514338]: Segment reached optimal threshold of 13 pieces. Dec 30 02:12:08.056506020 (586h43m7.927589055s) [noise=true , address=77.90.31.17:4773 , piece_id=JDTIDAQ37JIFLWBTY7XGLHH2KGK7AYIV52T6YDBURGCIGY43LQSA, seg_pos={0 0}, upload=514338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 30 02:12:08.056515788 (586h43m7.927598833s) [noise=true , address=77.90.31.17:4773 , piece_id=JDTIDAQ37JIFLWBTY7XGLHH2KGK7AYIV52T6YDBURGCIGY43LQSA, seg_pos={0 0}, upload=514338]: Long tail cancellation due to optimal threshold Dec 30 02:12:08.056661515 (586h43m7.92774456s) [noise=true , address=45.91.116.7:4009 , piece_id=32GVSDO2O7JKWXYZYY5FND27C7JBGUQYABNS4L6R3POPXEXUG5YQ, seg_pos={0 0}, upload=514338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:57620->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:57620->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57620->45.91.116.7:4009: use of closed network connection Dec 30 02:12:08.056678328 (586h43m7.927761373s) [noise=true , address=45.91.116.7:4009 , piece_id=32GVSDO2O7JKWXYZYY5FND27C7JBGUQYABNS4L6R3POPXEXUG5YQ, seg_pos={0 0}, upload=514338]: Long tail cancellation due to optimal threshold Dec 30 02:12:08.056759843 (586h43m7.927842878s) [noise=true , address=77.90.31.10:4517 , piece_id=IQ3J2YDS6B464VXSWQ4HYOKMZBXWLCWRIZPUSRPVBQ6BSQGH5X3Q, seg_pos={0 0}, upload=514338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 30 02:12:08.056768619 (586h43m7.927851654s) [noise=true , address=77.90.31.10:4517 , piece_id=IQ3J2YDS6B464VXSWQ4HYOKMZBXWLCWRIZPUSRPVBQ6BSQGH5X3Q, seg_pos={0 0}, upload=514338]: Long tail cancellation due to optimal threshold Dec 30 02:12:08.056787886 (586h43m7.927870931s) [seg_pos={0 0}, upload=514338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:12:08.056861456 (586h43m7.927944491s) [upload=514338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:12:08.056866646 (586h43m7.927949681s) [upload=514338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:12:08.343376083 (586h43m8.214459118s) [upload=514338]: Done uploading segments... Dec 30 02:12:08.343395971 (586h43m8.214479016s) [upload=514338]: Upload finished. err: <nil> Duration: 13.830938641s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)