Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......XXX......XX......X..X.....X..X.X...XX......X......XXX] duration: 30.005402239s trace: https://jaeger.dev.storj.io/trace/4fb9fbef16b57194 FAIL (2025-12-28 13:54:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:54:07.892295606 (550h25m7.763378651s) [upload=482591]: Starting upload Dec 28 13:54:07.892638519 (550h25m7.763721564s) [upload=482591]: Done starting upload Dec 28 13:54:07.892725125 (550h25m7.76380816s) [upload=482591]: Uploading segments... Dec 28 13:54:07.892984839 (550h25m7.764067874s) [upload=482591]: Got next segment. Inline: false Dec 28 13:54:07.893069029 (550h25m7.764152064s) [upload=482591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:54:07.893076804 (550h25m7.764159859s) [upload=482591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:54:08.138637550 (550h25m8.009720595s) [seg_pos={0 0}, upload=482591]: Begin upload segment... Dec 28 13:54:08.138687265 (550h25m8.0097703s) [seg_pos={0 0}, upload=482591]: Done begin upload segment. Dec 28 13:54:08.138716119 (550h25m8.009799165s) [noise=true , address=145.79.227.11:4730 , piece_id=AKJABTZOTSS27L7QDOPPQSOCOLSLXQ6CDPST7GGTTZMSYAP3TX7Q, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.138851578 (550h25m8.009934613s) [noise=true , address=145.79.227.23:4546 , piece_id=IWEMT3UNIKGSCRPGWHUMYP5SZTVUG6E3JQIESFRJAVLIFKIKJIYA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.139339016 (550h25m8.010422051s) [noise=true , address=77.90.31.13:4627 , piece_id=I7GS6E3BYNHM5W2BAXG2FDK54CCYUQPM76LZ73YTHJVYROPUH7DQ, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.139627536 (550h25m8.010710571s) [noise=true , address=145.79.227.38:4979 , piece_id=BLM7TYHVH6C7OQF7P5V4HNWXAEPAUBH3B4RDUZ6UI6CMRGTFELKA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.139928699 (550h25m8.011011744s) [noise=true , address=145.79.227.6:4259 , piece_id=2XMG6CM3O5D2E4ZXTPSMVZVZ52EEU6LEKA3MRHFD3HLF56EV55UA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.140225825 (550h25m8.01130887s) [noise=true , address=145.79.227.14:4840 , piece_id=NIYBPLN223WWYZRUQRRVKCNV5EBRKNDDRMDMCCWYQDZC2ZWV2TBA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.140521227 (550h25m8.011604262s) [noise=true , address=77.90.31.12:4589 , piece_id=H2LESPSUSYMAYTH72PCISTEVXZI2WERV7PBVBADDUKPV7KXRR7QQ, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.140820557 (550h25m8.011903602s) [noise=true , address=45.91.116.8:4048 , piece_id=AIEOFBFML3UN6I6TOVOLZZIJMQTZQ4EJB66RD7KYOOVK4OVCMGSQ, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.141246358 (550h25m8.012329403s) [noise=true , address=145.79.227.50:4374 , piece_id=W35KEUZPN4L6Y7SWTKKYJX7R77Z24YYKV52GB3CTYQOJDREYKW7Q, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.142794867 (550h25m8.013877902s) [noise=true , address=145.79.227.17:4971 , piece_id=M32YCCFIQCGJCFM754UORL5SGXTFCD25OA2DPQW5IU2PN2MFXAVA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.143146877 (550h25m8.014229912s) [noise=true , address=145.79.227.36:4519 , piece_id=S44J2YNK3WEUQPWS6S7WZQCMKUGPXZE26DOGTSYPAQ5VHZA64NCA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.144413108 (550h25m8.015496143s) [noise=true , address=45.91.116.11:4118 , piece_id=EUFBXYKVNTSSIHSNN2YPGBFRY2GFN2U5UZPIK7TAHYQYXLLXNA2A, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.145605598 (550h25m8.016688643s) [noise=true , address=77.90.31.14:4663 , piece_id=DK4LUEKQCDMKHHGSJCIBWURFYVKQ2TOODEK5YXOWRDF75TPSVSMA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.147040171 (550h25m8.018123216s) [noise=true , address=145.79.227.15:4874 , piece_id=I2H5LFQFEDT4BXB2ONOFD6FLRXNAHHCBPSWW2DW2WOWS43ZWGL6A, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.148331640 (550h25m8.019414685s) [noise=true , address=145.79.227.52:4445 , piece_id=IOKTVGZFTWJS5PAIPUUZ5SANHR6INYOT7OT5ZFICW7JZWGNQB6CA, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:08.149517398 (550h25m8.020600443s) [noise=true , address=45.91.116.9:4264 , piece_id=C4OJ6BIRRS5RANENT23CFT26JEWOJCCOXPAX32G36IGY2W4UEF7A, seg_pos={0 0}, upload=482591]: Uploading piece... Dec 28 13:54:09.873788995 (550h25m9.74487204s) [upload=482591]: Next returned nil segment Dec 28 13:54:09.873828500 (550h25m9.744911545s) [upload=482591]: Waiting for error group managing segments... Dec 28 13:54:10.151954466 (550h25m10.023037501s) [noise=true , address=145.79.227.38:4979 , piece_id=BLM7TYHVH6C7OQF7P5V4HNWXAEPAUBH3B4RDUZ6UI6CMRGTFELKA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.178287208 (550h25m10.049370243s) [noise=true , address=145.79.227.23:4546 , piece_id=IWEMT3UNIKGSCRPGWHUMYP5SZTVUG6E3JQIESFRJAVLIFKIKJIYA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.179386951 (550h25m10.050469986s) [noise=true , address=145.79.227.6:4259 , piece_id=2XMG6CM3O5D2E4ZXTPSMVZVZ52EEU6LEKA3MRHFD3HLF56EV55UA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.180013755 (550h25m10.0510968s) [noise=true , address=145.79.227.17:4971 , piece_id=M32YCCFIQCGJCFM754UORL5SGXTFCD25OA2DPQW5IU2PN2MFXAVA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.181047694 (550h25m10.052130729s) [noise=true , address=145.79.227.50:4374 , piece_id=W35KEUZPN4L6Y7SWTKKYJX7R77Z24YYKV52GB3CTYQOJDREYKW7Q, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.181293772 (550h25m10.052376807s) [noise=true , address=145.79.227.14:4840 , piece_id=NIYBPLN223WWYZRUQRRVKCNV5EBRKNDDRMDMCCWYQDZC2ZWV2TBA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.185528246 (550h25m10.056611281s) [noise=true , address=145.79.227.36:4519 , piece_id=S44J2YNK3WEUQPWS6S7WZQCMKUGPXZE26DOGTSYPAQ5VHZA64NCA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.189889630 (550h25m10.060972665s) [noise=true , address=145.79.227.11:4730 , piece_id=AKJABTZOTSS27L7QDOPPQSOCOLSLXQ6CDPST7GGTTZMSYAP3TX7Q, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.193016113 (550h25m10.064099178s) [noise=true , address=145.79.227.52:4445 , piece_id=IOKTVGZFTWJS5PAIPUUZ5SANHR6INYOT7OT5ZFICW7JZWGNQB6CA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.216808879 (550h25m10.087891914s) [noise=true , address=145.79.227.15:4874 , piece_id=I2H5LFQFEDT4BXB2ONOFD6FLRXNAHHCBPSWW2DW2WOWS43ZWGL6A, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.431385576 (550h25m10.302468621s) [noise=true , address=77.90.31.13:4627 , piece_id=I7GS6E3BYNHM5W2BAXG2FDK54CCYUQPM76LZ73YTHJVYROPUH7DQ, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.457634068 (550h25m10.328717103s) [noise=true , address=77.90.31.14:4663 , piece_id=DK4LUEKQCDMKHHGSJCIBWURFYVKQ2TOODEK5YXOWRDF75TPSVSMA, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.511532239 (550h25m10.382615274s) [noise=true , address=77.90.31.12:4589 , piece_id=H2LESPSUSYMAYTH72PCISTEVXZI2WERV7PBVBADDUKPV7KXRR7QQ, seg_pos={0 0}, upload=482591]: Done uploading piece. err: <nil> Dec 28 13:54:10.511552477 (550h25m10.382635512s) [seg_pos={0 0}, upload=482591]: Segment reached optimal threshold of 13 pieces. Dec 28 13:54:10.511734674 (550h25m10.382817709s) [noise=true , address=45.91.116.9:4264 , piece_id=C4OJ6BIRRS5RANENT23CFT26JEWOJCCOXPAX32G36IGY2W4UEF7A, seg_pos={0 0}, upload=482591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 13:54:10.511740195 (550h25m10.38282325s) [noise=true , address=45.91.116.9:4264 , piece_id=C4OJ6BIRRS5RANENT23CFT26JEWOJCCOXPAX32G36IGY2W4UEF7A, seg_pos={0 0}, upload=482591]: Long tail cancellation due to optimal threshold Dec 28 13:54:10.511906191 (550h25m10.382989226s) [noise=true , address=45.91.116.8:4048 , piece_id=AIEOFBFML3UN6I6TOVOLZZIJMQTZQ4EJB66RD7KYOOVK4OVCMGSQ, seg_pos={0 0}, upload=482591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 13:54:10.511911601 (550h25m10.382994636s) [noise=true , address=45.91.116.8:4048 , piece_id=AIEOFBFML3UN6I6TOVOLZZIJMQTZQ4EJB66RD7KYOOVK4OVCMGSQ, seg_pos={0 0}, upload=482591]: Long tail cancellation due to optimal threshold Dec 28 13:54:10.512126380 (550h25m10.383209425s) [noise=true , address=45.91.116.11:4118 , piece_id=EUFBXYKVNTSSIHSNN2YPGBFRY2GFN2U5UZPIK7TAHYQYXLLXNA2A, seg_pos={0 0}, upload=482591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:46260->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:54:10.512170834 (550h25m10.383253869s) [noise=true , address=45.91.116.11:4118 , piece_id=EUFBXYKVNTSSIHSNN2YPGBFRY2GFN2U5UZPIK7TAHYQYXLLXNA2A, seg_pos={0 0}, upload=482591]: Long tail cancellation due to optimal threshold Dec 28 13:54:10.512199619 (550h25m10.383282654s) [seg_pos={0 0}, upload=482591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:54:10.512291374 (550h25m10.38337442s) [upload=482591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:54:10.512296755 (550h25m10.3833798s) [upload=482591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:54:10.798283096 (550h25m10.669366132s) [upload=482591]: Done uploading segments... Dec 28 13:54:10.798306982 (550h25m10.669390017s) [upload=482591]: Upload finished. err: <nil> Duration: 30.005414984s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X....................X............X....................X] duration: 30.001187905s trace: https://jaeger.dev.storj.io/trace/1573b0780f8c077d FAIL (2025-12-28 13:54:34 UTC) orbiter warning: context deadline exceeded Duration: 30.001211519s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........X.................................] duration: 53.734364317s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:54:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 53.734393492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 17.532068873s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:54:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 17.53209879s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..] duration: 3.456709593s trace: https://jaeger.dev.storj.io/trace/cfb8c711e1c5c36 FAIL (2025-12-28 13:54:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456730533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.......XXX......XX......X..X.....X..X.X...XX......X......XX] duration: 30.003469194s trace: https://jaeger.dev.storj.io/trace/48471605740d554a FAIL (2025-12-28 13:54:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 13:53:37.647863050 (550h24m37.518946095s) [upload=482582]: Starting upload Dec 28 13:53:37.647984251 (550h24m37.519067286s) [upload=482582]: Done starting upload Dec 28 13:53:37.648037392 (550h24m37.519120437s) [upload=482582]: Uploading segments... Dec 28 13:53:37.648194210 (550h24m37.519277245s) [upload=482582]: Got next segment. Inline: false Dec 28 13:53:37.648220070 (550h24m37.519303125s) [upload=482582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:53:37.648226782 (550h24m37.519309838s) [upload=482582]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:53:37.888385612 (550h24m37.759468647s) [seg_pos={0 0}, upload=482582]: Begin upload segment... Dec 28 13:53:37.888441649 (550h24m37.759524695s) [seg_pos={0 0}, upload=482582]: Done begin upload segment. Dec 28 13:53:37.888469092 (550h24m37.759552137s) [noise=true , address=145.79.227.6:4259 , piece_id=MN35PA4HOMYA64IYR2BTT65SXMHNS2XJLUE5RJLEDKZUA5VFXGDA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.888496134 (550h24m37.759579179s) [noise=true , address=77.90.31.21:4877 , piece_id=O2DVEV6VPFFJU3K3VZWZXBOZMEOFYDLHD2TD76P4VWZFED7JRHMA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.888681366 (550h24m37.759764411s) [noise=true , address=145.79.227.13:4804 , piece_id=3RUWM6D33SZXL4T7CBPZQF2ZTA6EOCVA65KPA2ERELJ5AOEJBFZA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.889110994 (550h24m37.760194029s) [noise=true , address=77.90.31.19:4844 , piece_id=AZ6W3GRUET3LT7ZDZERBIWXDSCZWUVAPTS6TLHOSYT3HGGJ6XCAQ, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.889434230 (550h24m37.760517265s) [noise=true , address=77.90.31.14:4662 , piece_id=YWQESVEPZKPNOTXOOTOMEHE466WLGCCQP2GHZGNIKDYIWBBXAFEQ, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.889742486 (550h24m37.760825531s) [noise=true , address=145.79.227.20:4439 , piece_id=6L43XONYSBSN3RLMF7BHZMYRAXCT5BDF46R4N6WIOD4WXEZZAX5A, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.890276614 (550h24m37.761359659s) [noise=true , address=145.79.227.29:4942 , piece_id=NMGKZRV7HE27BAH74NVRFQGO3XI4QLU2ARDF4MB5MB2B6M6I6H5A, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.890635487 (550h24m37.761718532s) [noise=true , address=145.79.227.47:4266 , piece_id=42BHO2CSP26GPSLJNUU5SL72Z2QBNJUZKJSCFZP4J62ERVVNKAEQ, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.891093860 (550h24m37.762176895s) [noise=true , address=145.79.227.44:4194 , piece_id=ZCGQ46FRWWG7ZAWDGL6ZN4SH4QPV7KUX3YS5NYWW54SMRBS7KFLQ, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.891445739 (550h24m37.762528785s) [noise=true , address=145.79.227.46:4602 , piece_id=PF7VI6VUBQ6TGMTZY7C7ESHBILXAV5CPIPY4CO75ETYOL73MEM6Q, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.891780658 (550h24m37.762863703s) [noise=true , address=145.79.227.4:4187 , piece_id=22M6P5ZE2IGLAFDMEACJGOJK7JPN46GZF2HFLGBZD5JRKEXDHNGQ, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.895834366 (550h24m37.766917411s) [noise=true , address=145.79.227.51:4410 , piece_id=H3A5CRBVP5HN3RFI7ESTQEZKJNJJUGRUUO6GZISFJQSN7QHKZISA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.897542669 (550h24m37.768625714s) [noise=true , address=145.79.227.37:4566 , piece_id=E4GFWTP7XKO2YCSPBNKGM57K6UBX3LRNAVC47XB6F56ZNWHTUPYA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.898959076 (550h24m37.770042121s) [noise=true , address=145.79.227.39:4015 , piece_id=247CL6Y6AIBUWJPQESKAAXZFZI4X4X5EG2ST35LUXO2GUBQTYPAQ, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.902035795 (550h24m37.77311901s) [noise=true , address=145.79.227.3:4150 , piece_id=BW6WEZBBYFJDEXL5Z6QO5CWNXLR24J7MEUX4VMOTNNZZQAYPFERA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:37.903502918 (550h24m37.774585963s) [noise=true , address=77.90.31.13:4629 , piece_id=SWPVT7YIQJUFJTSF25H4YZNO56ZJIOWMMH43B7RNS2ONENWNU7RA, seg_pos={0 0}, upload=482582]: Uploading piece... Dec 28 13:53:39.591034818 (550h24m39.462117853s) [upload=482582]: Next returned nil segment Dec 28 13:53:39.591071968 (550h24m39.462155014s) [upload=482582]: Waiting for error group managing segments... Dec 28 13:53:39.890954001 (550h24m39.762037036s) [noise=true , address=145.79.227.20:4439 , piece_id=6L43XONYSBSN3RLMF7BHZMYRAXCT5BDF46R4N6WIOD4WXEZZAX5A, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.899356611 (550h24m39.770439646s) [noise=true , address=145.79.227.37:4566 , piece_id=E4GFWTP7XKO2YCSPBNKGM57K6UBX3LRNAVC47XB6F56ZNWHTUPYA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.906122234 (550h24m39.777205269s) [noise=true , address=145.79.227.6:4259 , piece_id=MN35PA4HOMYA64IYR2BTT65SXMHNS2XJLUE5RJLEDKZUA5VFXGDA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.914119392 (550h24m39.785202438s) [noise=true , address=145.79.227.39:4015 , piece_id=247CL6Y6AIBUWJPQESKAAXZFZI4X4X5EG2ST35LUXO2GUBQTYPAQ, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.919808405 (550h24m39.79089145s) [noise=true , address=145.79.227.29:4942 , piece_id=NMGKZRV7HE27BAH74NVRFQGO3XI4QLU2ARDF4MB5MB2B6M6I6H5A, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.921258215 (550h24m39.79234125s) [noise=true , address=145.79.227.51:4410 , piece_id=H3A5CRBVP5HN3RFI7ESTQEZKJNJJUGRUUO6GZISFJQSN7QHKZISA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.921560110 (550h24m39.792643155s) [noise=true , address=145.79.227.46:4602 , piece_id=PF7VI6VUBQ6TGMTZY7C7ESHBILXAV5CPIPY4CO75ETYOL73MEM6Q, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.923708321 (550h24m39.794791356s) [noise=true , address=145.79.227.13:4804 , piece_id=3RUWM6D33SZXL4T7CBPZQF2ZTA6EOCVA65KPA2ERELJ5AOEJBFZA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.926826839 (550h24m39.797909874s) [noise=true , address=145.79.227.4:4187 , piece_id=22M6P5ZE2IGLAFDMEACJGOJK7JPN46GZF2HFLGBZD5JRKEXDHNGQ, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.931127669 (550h24m39.802210704s) [noise=true , address=145.79.227.3:4150 , piece_id=BW6WEZBBYFJDEXL5Z6QO5CWNXLR24J7MEUX4VMOTNNZZQAYPFERA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:39.933044599 (550h24m39.804127634s) [noise=true , address=145.79.227.47:4266 , piece_id=42BHO2CSP26GPSLJNUU5SL72Z2QBNJUZKJSCFZP4J62ERVVNKAEQ, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:40.088852712 (550h24m39.959935757s) [noise=true , address=77.90.31.14:4662 , piece_id=YWQESVEPZKPNOTXOOTOMEHE466WLGCCQP2GHZGNIKDYIWBBXAFEQ, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:40.100675214 (550h24m39.971758249s) [noise=true , address=77.90.31.19:4844 , piece_id=AZ6W3GRUET3LT7ZDZERBIWXDSCZWUVAPTS6TLHOSYT3HGGJ6XCAQ, seg_pos={0 0}, upload=482582]: Done uploading piece. err: <nil> Dec 28 13:53:40.100692567 (550h24m39.971775612s) [seg_pos={0 0}, upload=482582]: Segment reached optimal threshold of 13 pieces. Dec 28 13:53:40.100844276 (550h24m39.971927311s) [noise=true , address=145.79.227.44:4194 , piece_id=ZCGQ46FRWWG7ZAWDGL6ZN4SH4QPV7KUX3YS5NYWW54SMRBS7KFLQ, seg_pos={0 0}, upload=482582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 13:53:40.100850247 (550h24m39.971933282s) [noise=true , address=145.79.227.44:4194 , piece_id=ZCGQ46FRWWG7ZAWDGL6ZN4SH4QPV7KUX3YS5NYWW54SMRBS7KFLQ, seg_pos={0 0}, upload=482582]: Long tail cancellation due to optimal threshold Dec 28 13:53:40.100909921 (550h24m39.971992956s) [noise=true , address=77.90.31.13:4629 , piece_id=SWPVT7YIQJUFJTSF25H4YZNO56ZJIOWMMH43B7RNS2ONENWNU7RA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:39328->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:39328->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39328->77.90.31.13:4629: use of closed network connection Dec 28 13:53:40.100919629 (550h24m39.972002674s) [noise=true , address=77.90.31.13:4629 , piece_id=SWPVT7YIQJUFJTSF25H4YZNO56ZJIOWMMH43B7RNS2ONENWNU7RA, seg_pos={0 0}, upload=482582]: Long tail cancellation due to optimal threshold Dec 28 13:53:40.101096416 (550h24m39.972179451s) [noise=true , address=77.90.31.21:4877 , piece_id=O2DVEV6VPFFJU3K3VZWZXBOZMEOFYDLHD2TD76P4VWZFED7JRHMA, seg_pos={0 0}, upload=482582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 13:53:40.101104572 (550h24m39.972187607s) [noise=true , address=77.90.31.21:4877 , piece_id=O2DVEV6VPFFJU3K3VZWZXBOZMEOFYDLHD2TD76P4VWZFED7JRHMA, seg_pos={0 0}, upload=482582]: Long tail cancellation due to optimal threshold Dec 28 13:53:40.101124309 (550h24m39.972207344s) [seg_pos={0 0}, upload=482582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:53:40.101210864 (550h24m39.972293899s) [upload=482582]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:53:40.101216275 (550h24m39.97229931s) [upload=482582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:53:40.384808388 (550h24m40.255891433s) [upload=482582]: Done uploading segments... Dec 28 13:53:40.384835410 (550h24m40.255918445s) [upload=482582]: Upload finished. err: <nil> Duration: 30.003495075s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....................X............X....................] duration: 20.63440987s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:53:50 UTC) orbiter warning: context deadline exceeded Duration: 20.634436431s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX.] duration: 3.595862202s trace: https://jaeger.dev.storj.io/trace/cfb8c711e1c5c36 FAIL (2025-12-28 13:53:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.595880055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.......XXX......XX......X..X.....X..X.X...XX......X......X] duration: 30.002689012s trace: https://jaeger.dev.storj.io/trace/33d5ee4e5763af74 FAIL (2025-12-28 13:53:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2GSG3RT3BZYSVYKAD67BKIOFMZQTGO4DHSGC2TAFX5GC7WXNZN3Q) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TRR7BAYKCIFSUWTADUBGCNKLPAKOHGDF5RZ53F6VUGW5EHKF47TQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: GPBHO76GEODHOYFYLMC5TF67CAGKTVW5IAAEWHG3SBPKJA4O6YAA) context deadline exceeded ======LOGS====== Dec 28 13:53:06.003292264 (550h24m5.874375329s) [upload=482575]: Starting upload Dec 28 13:53:06.003388548 (550h24m5.874471613s) [upload=482575]: Done starting upload Dec 28 13:53:06.003524096 (550h24m5.874607171s) [upload=482575]: Uploading segments... Dec 28 13:53:06.003660976 (550h24m5.874744032s) [upload=482575]: Got next segment. Inline: false Dec 28 13:53:06.003714549 (550h24m5.874797614s) [upload=482575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:53:06.003721722 (550h24m5.874804767s) [upload=482575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:53:06.262297451 (550h24m6.133380496s) [seg_pos={0 0}, upload=482575]: Begin upload segment... Dec 28 13:53:06.262393875 (550h24m6.13347693s) [seg_pos={0 0}, upload=482575]: Done begin upload segment. Dec 28 13:53:06.262484197 (550h24m6.133567242s) [noise=true , address=145.79.227.19:4077 , piece_id=S34KJX7XLKASIXAX3BJ7SVJZJVQ63SPTNQQZUJKLG3M7FUVAWXHQ, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.262495929 (550h24m6.133578974s) [noise=true , address=145.79.227.17:4971 , piece_id=IP7GVOKXOYP5HTVCHNYC5DPMK7R2ZS5WVN5X3TFVDBY6BYYDTAXA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.262631137 (550h24m6.133714182s) [noise=true , address=45.91.116.12:4225 , piece_id=GPBHO76GEODHOYFYLMC5TF67CAGKTVW5IAAEWHG3SBPKJA4O6YAA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.263175193 (550h24m6.134258248s) [noise=true , address=145.79.227.14:4837 , piece_id=HLK5JPBBL3JHWHXFI2LTVNPESYZJUWLE2XXNX3A6K3FGF23V5AOQ, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.263190793 (550h24m6.134273828s) [noise=true , address=145.79.227.45:4230 , piece_id=GOA4GNYVM4TUP7Q5UYR63GOZCJG36BVXOOQY23KVIV4DHD6N7HMQ, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.263626422 (550h24m6.134709467s) [noise=true , address=145.79.227.29:4943 , piece_id=UIPKPVKC32TY2P6LGUWZHYWF66K4K4DKHX2HTUG3TLC5WA3GM6XA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.263644286 (550h24m6.134727331s) [noise=true , address=45.91.116.6:4974 , piece_id=6SKUPECAVQVL4E3IZT3SWM5O4UX75MBS3SPPQDVUFB2FFQXHPUJQ, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.264036172 (550h24m6.135119217s) [noise=true , address=77.90.31.19:4844 , piece_id=2GSG3RT3BZYSVYKAD67BKIOFMZQTGO4DHSGC2TAFX5GC7WXNZN3Q, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.264087741 (550h24m6.135170786s) [noise=true , address=77.90.31.21:4878 , piece_id=4YWHX4ASHFHZ5CTLMV3ACINHC2E7BUUA5J44R7SBALRKV66KAK5A, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.265457108 (550h24m6.136540153s) [noise=true , address=45.91.116.10:4081 , piece_id=TRR7BAYKCIFSUWTADUBGCNKLPAKOHGDF5RZ53F6VUGW5EHKF47TQ, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.266938689 (550h24m6.138021734s) [noise=true , address=145.79.227.51:4410 , piece_id=RAKUHL7OAIYKCUCUEXML7KYJKCFV4RL27Z4AVM3BVAHMGGTBXU7A, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.268468492 (550h24m6.139551537s) [noise=true , address=145.79.227.50:4373 , piece_id=IEJKD34MWSYWVDS33XPWG64ZEEHNJUNDWL5QZ5PIUOEPGJZWQTQA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.269978978 (550h24m6.141062024s) [noise=true , address=145.79.227.16:4909 , piece_id=WREJQMDIW3P3K4DG7JTEBIANRK2JVJ62NPTWEOZJ2EYQFJBUXXBA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.271607459 (550h24m6.142690514s) [noise=true , address=45.91.116.14:4190 , piece_id=555JJDQA5ROF36QPMX5GKTG6VOUPFWWA5KPWGD4EHMRA3ZNVBFGA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.272571184 (550h24m6.143654229s) [noise=true , address=145.79.227.5:4223 , piece_id=SWO32427EO5JTARXY7PUT73PGYMRG4P2ILN2M7GEQTI4X7KQYKZA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:06.274876904 (550h24m6.145959949s) [noise=true , address=45.91.116.4:4939 , piece_id=DXOCYWUJ3AIHUPQ6DP7EZQ3LQIUSCPIC6PIOFEGY6LO47H7SKPOA, seg_pos={0 0}, upload=482575]: Uploading piece... Dec 28 13:53:07.982376084 (550h24m7.853459119s) [upload=482575]: Next returned nil segment Dec 28 13:53:07.982388938 (550h24m7.853471973s) [upload=482575]: Waiting for error group managing segments... Dec 28 13:53:08.266556447 (550h24m8.137639493s) [noise=true , address=145.79.227.17:4971 , piece_id=IP7GVOKXOYP5HTVCHNYC5DPMK7R2ZS5WVN5X3TFVDBY6BYYDTAXA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.276967552 (550h24m8.148050597s) [noise=true , address=145.79.227.50:4373 , piece_id=IEJKD34MWSYWVDS33XPWG64ZEEHNJUNDWL5QZ5PIUOEPGJZWQTQA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.278289840 (550h24m8.149372885s) [noise=true , address=145.79.227.5:4223 , piece_id=SWO32427EO5JTARXY7PUT73PGYMRG4P2ILN2M7GEQTI4X7KQYKZA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.290063248 (550h24m8.161146293s) [noise=true , address=145.79.227.29:4943 , piece_id=UIPKPVKC32TY2P6LGUWZHYWF66K4K4DKHX2HTUG3TLC5WA3GM6XA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.302610770 (550h24m8.173693815s) [noise=true , address=145.79.227.45:4230 , piece_id=GOA4GNYVM4TUP7Q5UYR63GOZCJG36BVXOOQY23KVIV4DHD6N7HMQ, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.304391962 (550h24m8.175475007s) [noise=true , address=145.79.227.16:4909 , piece_id=WREJQMDIW3P3K4DG7JTEBIANRK2JVJ62NPTWEOZJ2EYQFJBUXXBA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.315964979 (550h24m8.187048015s) [noise=true , address=145.79.227.14:4837 , piece_id=HLK5JPBBL3JHWHXFI2LTVNPESYZJUWLE2XXNX3A6K3FGF23V5AOQ, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.326117401 (550h24m8.197200436s) [noise=true , address=145.79.227.19:4077 , piece_id=S34KJX7XLKASIXAX3BJ7SVJZJVQ63SPTNQQZUJKLG3M7FUVAWXHQ, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.350312472 (550h24m8.221395507s) [noise=true , address=145.79.227.51:4410 , piece_id=RAKUHL7OAIYKCUCUEXML7KYJKCFV4RL27Z4AVM3BVAHMGGTBXU7A, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.518537549 (550h24m8.389620594s) [noise=true , address=45.91.116.12:4225 , piece_id=GPBHO76GEODHOYFYLMC5TF67CAGKTVW5IAAEWHG3SBPKJA4O6YAA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.542082523 (550h24m8.413165558s) [noise=true , address=77.90.31.21:4878 , piece_id=4YWHX4ASHFHZ5CTLMV3ACINHC2E7BUUA5J44R7SBALRKV66KAK5A, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.577223166 (550h24m8.448306201s) [noise=true , address=77.90.31.19:4844 , piece_id=2GSG3RT3BZYSVYKAD67BKIOFMZQTGO4DHSGC2TAFX5GC7WXNZN3Q, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.660488577 (550h24m8.531571622s) [noise=true , address=45.91.116.10:4081 , piece_id=TRR7BAYKCIFSUWTADUBGCNKLPAKOHGDF5RZ53F6VUGW5EHKF47TQ, seg_pos={0 0}, upload=482575]: Done uploading piece. err: <nil> Dec 28 13:53:08.660513654 (550h24m8.531596689s) [seg_pos={0 0}, upload=482575]: Segment reached optimal threshold of 13 pieces. Dec 28 13:53:08.660754583 (550h24m8.531837658s) [noise=true , address=45.91.116.6:4974 , piece_id=6SKUPECAVQVL4E3IZT3SWM5O4UX75MBS3SPPQDVUFB2FFQXHPUJQ, seg_pos={0 0}, upload=482575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 13:53:08.660764822 (550h24m8.531847867s) [noise=true , address=45.91.116.6:4974 , piece_id=6SKUPECAVQVL4E3IZT3SWM5O4UX75MBS3SPPQDVUFB2FFQXHPUJQ, seg_pos={0 0}, upload=482575]: Long tail cancellation due to optimal threshold Dec 28 13:53:08.660808636 (550h24m8.531891671s) [noise=true , address=45.91.116.4:4939 , piece_id=DXOCYWUJ3AIHUPQ6DP7EZQ3LQIUSCPIC6PIOFEGY6LO47H7SKPOA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:54148->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:54148->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->45.91.116.4:4939: use of closed network connection Dec 28 13:53:08.660817713 (550h24m8.531900748s) [noise=true , address=45.91.116.4:4939 , piece_id=DXOCYWUJ3AIHUPQ6DP7EZQ3LQIUSCPIC6PIOFEGY6LO47H7SKPOA, seg_pos={0 0}, upload=482575]: Long tail cancellation due to optimal threshold Dec 28 13:53:08.661014358 (550h24m8.532097403s) [noise=true , address=45.91.116.14:4190 , piece_id=555JJDQA5ROF36QPMX5GKTG6VOUPFWWA5KPWGD4EHMRA3ZNVBFGA, seg_pos={0 0}, upload=482575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 13:53:08.661021291 (550h24m8.532104326s) [noise=true , address=45.91.116.14:4190 , piece_id=555JJDQA5ROF36QPMX5GKTG6VOUPFWWA5KPWGD4EHMRA3ZNVBFGA, seg_pos={0 0}, upload=482575]: Long tail cancellation due to optimal threshold Dec 28 13:53:08.661039465 (550h24m8.5321225s) [seg_pos={0 0}, upload=482575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:53:08.661107935 (550h24m8.53219097s) [upload=482575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:53:08.661113857 (550h24m8.532196892s) [upload=482575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:53:08.940564757 (550h24m8.811647802s) [upload=482575]: Done uploading segments... Dec 28 13:53:08.940590897 (550h24m8.811673942s) [upload=482575]: Upload finished. err: <nil> Duration: 30.002718358s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 11.106460775s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:53:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 11.106490692s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........X................................] duration: 45.511856847s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 45.511877726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX] duration: 5.001009993s trace: https://jaeger.dev.storj.io/trace/cfb8c711e1c5c36 FAIL (2025-12-28 13:52:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001033408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 13.288042629s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:52:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 13.288130706s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X....................X............X...................] duration: 21.074531902s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:52:39 UTC) orbiter warning: context deadline exceeded Duration: 21.074623216s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X........X...............................] duration: 41.632697932s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:52:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 41.632724975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.......XXX......XX......X..X.....X..X.X...XX......X......] duration: 6.081865877s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 6.081973802s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.605396652s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:52:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 7.60542623s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X....................X............X..................] duration: 13.771115348s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:52:09 UTC) orbiter warning: context deadline exceeded Duration: 13.77115282s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X....................X............X.................] duration: 18.425112245s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:51:49 UTC) orbiter warning: context deadline exceeded Duration: 18.425137293s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.X] duration: 5.000969257s trace: https://jaeger.dev.storj.io/trace/50355be6da35aa1a FAIL (2025-12-28 13:51:43 UTC) orbiter warning: context deadline exceeded Duration: 5.000984747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X........X..............................] duration: 56.604945651s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:51:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 56.604965689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X....................X............X................] duration: 14.571362616s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:51:27 UTC) orbiter warning: context deadline exceeded Duration: 14.571392373s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X.......XXX......XX......X..X.....X..X.X...XX......X.....] duration: 6.155524735s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 6.15555369s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 8.463535023s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:51:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 8.46356517s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.] duration: 3.527695863s trace: https://jaeger.dev.storj.io/trace/79fedd7adff3825e FAIL (2025-12-28 13:51:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527727303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX..X.......XXX......XX......X..X.....X..X.X...XX......X....] duration: 5.921275156s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:50:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 5.921311345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X........X.............................] duration: 29.700172671s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:50:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 29.700202369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX] duration: 5.000897904s trace: https://jaeger.dev.storj.io/trace/79fedd7adff3825e FAIL (2025-12-28 13:50:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000920557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X....................X............X...............] duration: 21.03045485s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:50:25 UTC) orbiter warning: context deadline exceeded Duration: 21.030486201s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X.......XXX......XX......X..X.....X..X.X...XX......X...] duration: 5.739957551s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:50:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 5.739991134s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXX] duration: 5.001183538s trace: https://jaeger.dev.storj.io/trace/6cfe0d03c86ed345 FAIL (2025-12-28 13:50:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001208406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 10.505456878s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:50:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 10.505485273s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X........X............................] duration: 54.063941219s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:49:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 54.06395752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX..X.......XXX......XX......X..X.....X..X.X...XX......X..] duration: 5.862472297s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 5.862518145s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 10.591300386s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:49:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 10.591334912s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....................X............X..............] duration: 16.513321812s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:49:39 UTC) orbiter warning: context deadline exceeded Duration: 16.513343995s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXX] duration: 5.001531812s trace: https://jaeger.dev.storj.io/trace/60da6b21f26af91d FAIL (2025-12-28 13:49:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001560898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..XX..X.......XXX......XX......X..X.....X..X.X...XX......X.] duration: 5.710623533s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:49:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 5.710653059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X........X...........................] duration: 49.729651582s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 49.729684654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...........X....................X............X.............] duration: 18.695856199s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:48:42 UTC) orbiter warning: context deadline exceeded Duration: 18.695880657s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.640137053s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:48:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 6.640170897s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX..XX..X.......XXX......XX......X..X.....X..X.X...XX......X] duration: 30.0025801s trace: https://jaeger.dev.storj.io/trace/3653ba8968e6aa9 FAIL (2025-12-28 13:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:48:09.882973958 (550h19m9.754057003s) [upload=482503]: Starting upload Dec 28 13:48:09.883064009 (550h19m9.754147054s) [upload=482503]: Done starting upload Dec 28 13:48:09.883275030 (550h19m9.754358065s) [upload=482503]: Uploading segments... Dec 28 13:48:09.883371785 (550h19m9.75445483s) [upload=482503]: Got next segment. Inline: false Dec 28 13:48:09.883402544 (550h19m9.754485579s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:48:09.883406862 (550h19m9.754489897s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:48:10.130290127 (550h19m10.001373182s) [seg_pos={0 0}, upload=482503]: Begin upload segment... Dec 28 13:48:10.130412820 (550h19m10.001495865s) [seg_pos={0 0}, upload=482503]: Done begin upload segment. Dec 28 13:48:10.130452576 (550h19m10.001535622s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130521788 (550h19m10.001604833s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.130764650 (550h19m10.001847695s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.131662520 (550h19m10.002745565s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132169025 (550h19m10.00325207s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.132667534 (550h19m10.003750569s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133147277 (550h19m10.004230322s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.133584569 (550h19m10.004667614s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134263272 (550h19m10.005346317s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.134877050 (550h19m10.005960095s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.135348629 (550h19m10.006431664s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.138933946 (550h19m10.010016991s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.141582549 (550h19m10.012665594s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.144042664 (550h19m10.015125719s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.146330289 (550h19m10.017413324s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:10.148503458 (550h19m10.019586503s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Uploading piece... Dec 28 13:48:11.844149097 (550h19m11.715232142s) [upload=482503]: Next returned nil segment Dec 28 13:48:11.844173483 (550h19m11.715256599s) [upload=482503]: Waiting for error group managing segments... Dec 28 13:48:12.139626573 (550h19m12.010709608s) [noise=true , address=145.79.227.19:4077 , piece_id=EIUFP5AWE6CYH3E4AAKQZEHVMAJVUIF5ZOAXVDEIHNSQTLREZLTQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.146302846 (550h19m12.017385891s) [noise=true , address=145.79.227.39:4015 , piece_id=XZZEQKFLLDEQ5HDHEM2TGUFQ4SN37JNVJZOG6ZYKUA6OSWNQ65MQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.157724544 (550h19m12.028807579s) [noise=true , address=145.79.227.16:4909 , piece_id=UO3LUWASHF455KAE3XMFD67GJICALWQBXHBLXPPGRSXLB3HSBYIA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.160647650 (550h19m12.031730686s) [noise=true , address=145.79.227.25:4619 , piece_id=K3I23CQLDCT6CNTZVJLUBPKEEKOUBANKPMLQMJMWHKTBXIIR2ZBA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.162708144 (550h19m12.033791189s) [noise=true , address=145.79.227.32:4836 , piece_id=25LGK5RH4FWSTWWSD2OP3KSWMCLY7GNSEO2EY52GMUTSOMUWVX6Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.168387859 (550h19m12.039470904s) [noise=true , address=145.79.227.10:4401 , piece_id=ZG4YMWP2GBRH5PWSK7MF2OLDGOJXAMCHDHOHN5YFLFFAVHFCQCEQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.170764624 (550h19m12.041847659s) [noise=true , address=145.79.227.3:4149 , piece_id=TQFWT34DJWVCEVKHJEUBIX3EFETSA522MHGFW2PVHBBEH6YNCBVA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.172738243 (550h19m12.043821278s) [noise=true , address=145.79.227.52:4446 , piece_id=WYNFWL6ZFKH77U4U3XRQ5KB2B7M25A3FWZPO7ZHZ6KJOJMBZ7QAA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.174723743 (550h19m12.045806778s) [noise=true , address=145.79.227.36:4518 , piece_id=KKGAE5N2LSKDVBVUMAUQO2CXOZIJ3Z47OYEOSIWW272UVXRCKSOA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.190856312 (550h19m12.061939347s) [noise=true , address=145.79.227.20:4439 , piece_id=BR63NRL4C6GSMNGBHKNTOS4F6P3TFCAGBGXHUZDX7HNK5QHHDMYA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.191038288 (550h19m12.062121324s) [noise=true , address=145.79.227.41:4085 , piece_id=5YVBIY76ARL7PHAGLOLEIID6G4IEYNPMVOS7TMKIEOYG562O6AWA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.405894990 (550h19m12.276978025s) [noise=true , address=45.91.116.14:4191 , piece_id=5HJQMM27ICI4L3SLZUWDZZ76MKA7C5LQGV5GIE46IASBWB2MX7ZQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412053376 (550h19m12.283136411s) [noise=true , address=77.90.31.21:4878 , piece_id=MY2MDJ7JLHKD7JI7VILLJFIHTCIXUAT456KWXTRPP5HZLLTYYGDA, seg_pos={0 0}, upload=482503]: Done uploading piece. err: <nil> Dec 28 13:48:12.412083784 (550h19m12.283166819s) [seg_pos={0 0}, upload=482503]: Segment reached optimal threshold of 13 pieces. Dec 28 13:48:12.412256172 (550h19m12.283339207s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412262775 (550h19m12.28334582s) [noise=true , address=77.90.31.14:4661 , piece_id=L72PV6GVCDIVD5D5SY7INUFFTVKZHPIIXSOGIMSUYVHFMBVYH67Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412638810 (550h19m12.283721845s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412648660 (550h19m12.283731705s) [noise=true , address=45.91.116.5:4297 , piece_id=565PUJYRNUB4DTKRGCCYSVHQEQ4MICIHAYRGEV5NBQJVFLLP6K5Q, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412827660 (550h19m12.283910706s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 13:48:12.412835706 (550h19m12.283918741s) [noise=true , address=77.90.31.10:4519 , piece_id=CIYOMDE3MVAICJZZCSP337FSSV2JLSIU23DZDBHYRVFFDJDEEAFQ, seg_pos={0 0}, upload=482503]: Long tail cancellation due to optimal threshold Dec 28 13:48:12.412856506 (550h19m12.283939541s) [seg_pos={0 0}, upload=482503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:48:12.412928422 (550h19m12.284011457s) [upload=482503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:48:12.412934043 (550h19m12.284017078s) [upload=482503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:48:12.699891445 (550h19m12.57097449s) [upload=482503]: Done uploading segments... Dec 28 13:48:12.699914239 (550h19m12.570997284s) [upload=482503]: Upload finished. err: <nil> Duration: 30.002604657s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...........X....................X............X............] duration: 19.596265066s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:48:14 UTC) orbiter warning: context deadline exceeded Duration: 19.596293571s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.436818761s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:48:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 7.436851544s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X........X..........................] duration: 37.281005254s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:48:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 37.281020563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXXX] duration: 5.001296674s trace: https://jaeger.dev.storj.io/trace/296d8362edad428e FAIL (2025-12-28 13:48:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001322783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..XX..X.......XXX......XX......X..X.....X..X.X...XX......] duration: 5.930291378s trace: https://jaeger.dev.storj.io/trace/94c6dc4e5f68a6e FAIL (2025-12-28 13:47:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA) context deadline exceeded ======LOGS====== 550h13m27.258022633s) [upload=482419]: Done starting upload Dec 28 13:42:27.387099462 (550h13m27.258182497s) [upload=482419]: Uploading segments... Dec 28 13:42:27.387357324 (550h13m27.258440459s) [upload=482419]: Got next segment. Inline: false Dec 28 13:42:27.387408080 (550h13m27.258491115s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:42:27.387413841 (550h13m27.258496886s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:42:27.638563682 (550h13m27.509646727s) [seg_pos={0 0}, upload=482419]: Begin upload segment... Dec 28 13:42:27.638646689 (550h13m27.509729724s) [seg_pos={0 0}, upload=482419]: Done begin upload segment. Dec 28 13:42:27.638711563 (550h13m27.509794608s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.638725659 (550h13m27.509808705s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.639592059 (550h13m27.510675094s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640287844 (550h13m27.511370879s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640759092 (550h13m27.511842137s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.641490505 (550h13m27.51257354s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.642874049 (550h13m27.513957084s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.643700002 (550h13m27.514783047s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.644468685 (550h13m27.51555173s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.645884662 (550h13m27.516967697s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.647456905 (550h13m27.518539941s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648041389 (550h13m27.519124424s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648974235 (550h13m27.52005727s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.650328172 (550h13m27.521411217s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.651182510 (550h13m27.522265545s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.653389852 (550h13m27.524472887s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:29.259891664 (550h13m29.130974699s) [upload=482419]: Next returned nil segment Dec 28 13:42:29.259904918 (550h13m29.130987954s) [upload=482419]: Waiting for error group managing segments... Dec 28 13:42:29.646472706 (550h13m29.517555751s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.649315019 (550h13m29.520398064s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655433360 (550h13m29.526516395s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655570040 (550h13m29.526653075s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.656816143 (550h13m29.527899178s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.662269416 (550h13m29.533352461s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.664572742 (550h13m29.535655787s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.678652472 (550h13m29.549735517s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.683770548 (550h13m29.554853584s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684176270 (550h13m29.555259315s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684578737 (550h13m29.555661772s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.901472728 (550h13m29.772555773s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905337859 (550h13m29.776420894s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905361523 (550h13m29.776444568s) [seg_pos={0 0}, upload=482419]: Segment reached optimal threshold of 13 pieces. Dec 28 13:42:29.905586202 (550h13m29.776669237s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection Dec 28 13:42:29.905593856 (550h13m29.776676891s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.905731959 (550h13m29.776814994s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 13:42:29.905738521 (550h13m29.776821556s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906284732 (550h13m29.777367777s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:58736->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:42:29.906304299 (550h13m29.777387344s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906332723 (550h13m29.777415768s) [seg_pos={0 0}, upload=482419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:42:29.906459084 (550h13m29.777542129s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:42:29.906466838 (550h13m29.777549883s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:42:30.209359044 (550h13m30.080442089s) [upload=482419]: Done uploading segments... Dec 28 13:42:30.209387108 (550h13m30.080470143s) [upload=482419]: Upload finished. err: <nil> Duration: 5.930325462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X........X.........................] duration: 57.996571017s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:47:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 57.996634259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...........X....................X............X...........] duration: 29.25548964s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:47:25 UTC) orbiter warning: context deadline exceeded Duration: 29.25550984s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX.XXX] duration: 5.001060182s trace: https://jaeger.dev.storj.io/trace/600b4c8ea26b8501 FAIL (2025-12-28 13:47:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001079479s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)