Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.] duration: 6.425738488s trace: https://jaeger.dev.storj.io/trace/4cc195fca15a7c47 FAIL (2025-12-26 17:19:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:18:37.777870471 (505h49m37.648953506s) [upload=443856]: Starting upload Dec 26 17:18:37.777923402 (505h49m37.649006437s) [upload=443856]: Done starting upload Dec 26 17:18:37.778097073 (505h49m37.649180108s) [upload=443856]: Uploading segments... Dec 26 17:18:37.778129054 (505h49m37.649212089s) [upload=443856]: Got next segment. Inline: false Dec 26 17:18:37.778265934 (505h49m37.649348969s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:18:37.778273369 (505h49m37.649356444s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:18:38.025749897 (505h49m37.896832932s) [seg_pos={0 0}, upload=443856]: Begin upload segment... Dec 26 17:18:38.025816554 (505h49m37.896899599s) [seg_pos={0 0}, upload=443856]: Done begin upload segment. Dec 26 17:18:38.025841882 (505h49m37.896924917s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025873222 (505h49m37.896956257s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025928557 (505h49m37.897011592s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026419021 (505h49m37.897502097s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026903673 (505h49m37.897986708s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027235215 (505h49m37.89831825s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027531730 (505h49m37.898614765s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027824537 (505h49m37.898907572s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028138094 (505h49m37.89922114s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028493782 (505h49m37.899576897s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029905250 (505h49m37.900988285s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029982947 (505h49m37.901065992s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.031149188 (505h49m37.902232233s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.032260233 (505h49m37.903343268s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.033446021 (505h49m37.904529066s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.034788017 (505h49m37.905871062s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:39.644387957 (505h49m39.515471002s) [upload=443856]: Next returned nil segment Dec 26 17:18:39.644405520 (505h49m39.515488555s) [upload=443856]: Waiting for error group managing segments... Dec 26 17:18:40.030562751 (505h49m39.901645786s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.031064616 (505h49m39.902147651s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.033601967 (505h49m39.904685002s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.034869291 (505h49m39.905952336s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.047423045 (505h49m39.91850609s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.053560312 (505h49m39.924643347s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.059360767 (505h49m39.930443812s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.066785935 (505h49m39.93786897s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.070927642 (505h49m39.942010717s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.270553644 (505h49m40.14163668s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.401567638 (505h49m40.272650673s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.403630647 (505h49m40.274713682s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623347386 (505h49m40.494430421s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623372052 (505h49m40.494455097s) [seg_pos={0 0}, upload=443856]: Segment reached optimal threshold of 13 pieces. Dec 26 17:18:40.623576141 (505h49m40.494659186s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623585970 (505h49m40.494669016s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623803705 (505h49m40.49488674s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623810177 (505h49m40.494893212s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623827299 (505h49m40.494910334s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623846997 (505h49m40.494930042s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623871514 (505h49m40.494954559s) [seg_pos={0 0}, upload=443856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:18:40.623992634 (505h49m40.495075669s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:18:40.623999096 (505h49m40.495082141s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:18:40.914858874 (505h49m40.785941909s) [upload=443856]: Done uploading segments... Dec 26 17:18:40.914878332 (505h49m40.785961367s) [upload=443856]: Upload finished. err: <nil> Duration: 6.425763315s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..XXX...X........X..........................................] duration: 3.589037824s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:19:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5890623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......X....X......................................] duration: 26.602469261s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:19:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 26.602490131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X] duration: 30.003418542s trace: https://jaeger.dev.storj.io/trace/4cc195fca15a7c47 FAIL (2025-12-26 17:19:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:18:37.777870471 (505h49m37.648953506s) [upload=443856]: Starting upload Dec 26 17:18:37.777923402 (505h49m37.649006437s) [upload=443856]: Done starting upload Dec 26 17:18:37.778097073 (505h49m37.649180108s) [upload=443856]: Uploading segments... Dec 26 17:18:37.778129054 (505h49m37.649212089s) [upload=443856]: Got next segment. Inline: false Dec 26 17:18:37.778265934 (505h49m37.649348969s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:18:37.778273369 (505h49m37.649356444s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:18:38.025749897 (505h49m37.896832932s) [seg_pos={0 0}, upload=443856]: Begin upload segment... Dec 26 17:18:38.025816554 (505h49m37.896899599s) [seg_pos={0 0}, upload=443856]: Done begin upload segment. Dec 26 17:18:38.025841882 (505h49m37.896924917s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025873222 (505h49m37.896956257s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025928557 (505h49m37.897011592s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026419021 (505h49m37.897502097s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026903673 (505h49m37.897986708s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027235215 (505h49m37.89831825s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027531730 (505h49m37.898614765s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027824537 (505h49m37.898907572s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028138094 (505h49m37.89922114s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028493782 (505h49m37.899576897s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029905250 (505h49m37.900988285s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029982947 (505h49m37.901065992s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.031149188 (505h49m37.902232233s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.032260233 (505h49m37.903343268s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.033446021 (505h49m37.904529066s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.034788017 (505h49m37.905871062s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:39.644387957 (505h49m39.515471002s) [upload=443856]: Next returned nil segment Dec 26 17:18:39.644405520 (505h49m39.515488555s) [upload=443856]: Waiting for error group managing segments... Dec 26 17:18:40.030562751 (505h49m39.901645786s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.031064616 (505h49m39.902147651s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.033601967 (505h49m39.904685002s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.034869291 (505h49m39.905952336s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.047423045 (505h49m39.91850609s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.053560312 (505h49m39.924643347s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.059360767 (505h49m39.930443812s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.066785935 (505h49m39.93786897s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.070927642 (505h49m39.942010717s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.270553644 (505h49m40.14163668s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.401567638 (505h49m40.272650673s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.403630647 (505h49m40.274713682s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623347386 (505h49m40.494430421s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623372052 (505h49m40.494455097s) [seg_pos={0 0}, upload=443856]: Segment reached optimal threshold of 13 pieces. Dec 26 17:18:40.623576141 (505h49m40.494659186s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623585970 (505h49m40.494669016s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623803705 (505h49m40.49488674s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623810177 (505h49m40.494893212s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623827299 (505h49m40.494910334s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623846997 (505h49m40.494930042s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623871514 (505h49m40.494954559s) [seg_pos={0 0}, upload=443856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:18:40.623992634 (505h49m40.495075669s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:18:40.623999096 (505h49m40.495082141s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:18:40.914858874 (505h49m40.785941909s) [upload=443856]: Done uploading segments... Dec 26 17:18:40.914878332 (505h49m40.785961367s) [upload=443856]: Upload finished. err: <nil> Duration: 30.003442388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......X....X.....................................] duration: 54.719769566s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 54.719799121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...XXX...X........X.........................................] duration: 3.507813589s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:18:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507837985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....XXX...X........X........................................] duration: 3.604349189s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:17:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604375369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.] duration: 6.187458491s trace: https://jaeger.dev.storj.io/trace/747586e3aa30dcfa FAIL (2025-12-26 17:17:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA) context deadline exceeded ======LOGS====== Dec 26 17:16:43.166520031 (505h47m43.037603066s) [upload=443828]: Starting upload Dec 26 17:16:43.166580586 (505h47m43.037663622s) [upload=443828]: Done starting upload Dec 26 17:16:43.166652694 (505h47m43.037735739s) [upload=443828]: Uploading segments... Dec 26 17:16:43.166810954 (505h47m43.03789399s) [upload=443828]: Got next segment. Inline: false Dec 26 17:16:43.166849488 (505h47m43.037932533s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:16:43.166854188 (505h47m43.037937233s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:16:43.426770333 (505h47m43.297853398s) [seg_pos={0 0}, upload=443828]: Begin upload segment... Dec 26 17:16:43.426845216 (505h47m43.297928261s) [seg_pos={0 0}, upload=443828]: Done begin upload segment. Dec 26 17:16:43.426880583 (505h47m43.297963628s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433145322 (505h47m43.304228357s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433884860 (505h47m43.304967895s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.435975611 (505h47m43.307058646s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.438307441 (505h47m43.309390487s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.440534272 (505h47m43.311617317s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.442626315 (505h47m43.31370937s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.445255162 (505h47m43.316338197s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.446248623 (505h47m43.317331678s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.447418741 (505h47m43.318501786s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448881366 (505h47m43.319964401s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448983050 (505h47m43.320066125s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.450465342 (505h47m43.321548387s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.451848827 (505h47m43.322931872s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.452402511 (505h47m43.323485576s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.454342485 (505h47m43.32542552s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:45.157230350 (505h47m45.028313385s) [upload=443828]: Next returned nil segment Dec 26 17:16:45.157243164 (505h47m45.028326199s) [upload=443828]: Waiting for error group managing segments... Dec 26 17:16:45.430886079 (505h47m45.301969124s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.445045310 (505h47m45.316128345s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.451806074 (505h47m45.322889119s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.452867274 (505h47m45.323950319s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.456499641 (505h47m45.327582686s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.462561865 (505h47m45.3336449s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.470207082 (505h47m45.341290137s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.473631944 (505h47m45.344714989s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.474290298 (505h47m45.345373333s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.496098745 (505h47m45.36718178s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.758545770 (505h47m45.629628815s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.799247743 (505h47m45.670330778s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.979342215 (505h47m45.85042526s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 17:16:46.329608904 (505h47m46.200691949s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:46.329631226 (505h47m46.200714271s) [seg_pos={0 0}, upload=443828]: Segment reached optimal threshold of 13 pieces. Dec 26 17:16:46.329863578 (505h47m46.200946614s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 17:16:46.329873096 (505h47m46.200956141s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329908023 (505h47m46.200991058s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection Dec 26 17:16:46.329916971 (505h47m46.201000006s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329946526 (505h47m46.201029571s) [seg_pos={0 0}, upload=443828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:16:46.330107573 (505h47m46.201190608s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:16:46.330116700 (505h47m46.201199735s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:16:46.631634422 (505h47m46.502717467s) [upload=443828]: Done uploading segments... Dec 26 17:16:46.631658008 (505h47m46.502741043s) [upload=443828]: Upload finished. err: <nil> Duration: 6.187481283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......X....X....................................] duration: 49.487538093s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 49.487555887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....XXX...X........X.......................................] duration: 3.497761838s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:17:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497802135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X] duration: 30.002123195s trace: https://jaeger.dev.storj.io/trace/747586e3aa30dcfa FAIL (2025-12-26 17:17:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA) context deadline exceeded ======LOGS====== Dec 26 17:16:43.166520031 (505h47m43.037603066s) [upload=443828]: Starting upload Dec 26 17:16:43.166580586 (505h47m43.037663622s) [upload=443828]: Done starting upload Dec 26 17:16:43.166652694 (505h47m43.037735739s) [upload=443828]: Uploading segments... Dec 26 17:16:43.166810954 (505h47m43.03789399s) [upload=443828]: Got next segment. Inline: false Dec 26 17:16:43.166849488 (505h47m43.037932533s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:16:43.166854188 (505h47m43.037937233s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:16:43.426770333 (505h47m43.297853398s) [seg_pos={0 0}, upload=443828]: Begin upload segment... Dec 26 17:16:43.426845216 (505h47m43.297928261s) [seg_pos={0 0}, upload=443828]: Done begin upload segment. Dec 26 17:16:43.426880583 (505h47m43.297963628s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433145322 (505h47m43.304228357s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433884860 (505h47m43.304967895s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.435975611 (505h47m43.307058646s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.438307441 (505h47m43.309390487s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.440534272 (505h47m43.311617317s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.442626315 (505h47m43.31370937s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.445255162 (505h47m43.316338197s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.446248623 (505h47m43.317331678s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.447418741 (505h47m43.318501786s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448881366 (505h47m43.319964401s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448983050 (505h47m43.320066125s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.450465342 (505h47m43.321548387s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.451848827 (505h47m43.322931872s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.452402511 (505h47m43.323485576s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.454342485 (505h47m43.32542552s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:45.157230350 (505h47m45.028313385s) [upload=443828]: Next returned nil segment Dec 26 17:16:45.157243164 (505h47m45.028326199s) [upload=443828]: Waiting for error group managing segments... Dec 26 17:16:45.430886079 (505h47m45.301969124s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.445045310 (505h47m45.316128345s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.451806074 (505h47m45.322889119s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.452867274 (505h47m45.323950319s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.456499641 (505h47m45.327582686s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.462561865 (505h47m45.3336449s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.470207082 (505h47m45.341290137s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.473631944 (505h47m45.344714989s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.474290298 (505h47m45.345373333s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.496098745 (505h47m45.36718178s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.758545770 (505h47m45.629628815s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.799247743 (505h47m45.670330778s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.979342215 (505h47m45.85042526s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 17:16:46.329608904 (505h47m46.200691949s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:46.329631226 (505h47m46.200714271s) [seg_pos={0 0}, upload=443828]: Segment reached optimal threshold of 13 pieces. Dec 26 17:16:46.329863578 (505h47m46.200946614s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 17:16:46.329873096 (505h47m46.200956141s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329908023 (505h47m46.200991058s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection Dec 26 17:16:46.329916971 (505h47m46.201000006s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329946526 (505h47m46.201029571s) [seg_pos={0 0}, upload=443828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:16:46.330107573 (505h47m46.201190608s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:16:46.330116700 (505h47m46.201199735s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:16:46.631634422 (505h47m46.502717467s) [upload=443828]: Done uploading segments... Dec 26 17:16:46.631658008 (505h47m46.502741043s) [upload=443828]: Upload finished. err: <nil> Duration: 30.002305051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X....X...................................] duration: 27.789723662s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:16:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 27.789743168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......XXX...X........X......................................] duration: 4.392848019s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:16:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.392870173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....] duration: 16.035107872s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 16.035135233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X....X..................................] duration: 39.408699659s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 39.408729024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X......XXX...X........X.....................................] duration: 3.445370075s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:15:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445399351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X...] duration: 6.230150855s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:15:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 6.230183606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......X....X.................................] duration: 45.043686444s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 45.043717793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X..] duration: 28.746248739s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:15:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 28.746279277s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X......XXX...X........X....................................] duration: 3.509858643s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:14:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509908949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X.] duration: 7.283177672s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 7.283205083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......X....X................................] duration: 37.656463194s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 37.656481339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X......XXX...X........X...................................] duration: 3.515350922s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:13:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515427417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X] duration: 30.001930678s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:13:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 30.001954693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X....X...............................] duration: 50.283932746s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 50.283949257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X......XXX...X........X..................................] duration: 3.369785414s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:13:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.369812375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...............X.......X....X..............................] duration: 54.486857909s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:12:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 54.486886302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X......XXX...X........X.................................] duration: 3.539637442s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:12:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539656789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..] duration: 6.44131592s trace: https://jaeger.dev.storj.io/trace/2a33c8a0896e2812 FAIL (2025-12-26 17:12:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:09:59.412405166 (505h40m59.283488221s) [upload=443734]: Starting upload Dec 26 17:09:59.412488264 (505h40m59.283571299s) [upload=443734]: Done starting upload Dec 26 17:09:59.412584588 (505h40m59.283667633s) [upload=443734]: Uploading segments... Dec 26 17:09:59.412702623 (505h40m59.283785668s) [upload=443734]: Got next segment. Inline: false Dec 26 17:09:59.412739273 (505h40m59.283822318s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:59.412746376 (505h40m59.283829431s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:59.656808823 (505h40m59.527891868s) [seg_pos={0 0}, upload=443734]: Begin upload segment... Dec 26 17:09:59.656851325 (505h40m59.52793436s) [seg_pos={0 0}, upload=443734]: Done begin upload segment. Dec 26 17:09:59.656875851 (505h40m59.527958896s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.656927179 (505h40m59.528010224s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657442641 (505h40m59.528525676s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657819287 (505h40m59.528902322s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657989451 (505h40m59.529072486s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658360388 (505h40m59.529443433s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658632737 (505h40m59.529715772s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658977594 (505h40m59.530060639s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.661716348 (505h40m59.532799393s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.662944547 (505h40m59.534027592s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.663187159 (505h40m59.534270194s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664207722 (505h40m59.535290767s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664395620 (505h40m59.535478665s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.665648896 (505h40m59.536731931s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.667001091 (505h40m59.538084126s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.668250921 (505h40m59.539333956s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:10:01.276948018 (505h41m1.148031064s) [upload=443734]: Next returned nil segment Dec 26 17:10:01.276989468 (505h41m1.148072513s) [upload=443734]: Waiting for error group managing segments... Dec 26 17:10:01.660212240 (505h41m1.531295285s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.667666294 (505h41m1.538749329s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.670284259 (505h41m1.541367304s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.680257970 (505h41m1.551341016s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.683131332 (505h41m1.554214377s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.687154904 (505h41m1.558237969s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.696419345 (505h41m1.56750239s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.724453487 (505h41m1.595536532s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.916593334 (505h41m1.787676399s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.917178638 (505h41m1.788261683s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.918198120 (505h41m1.789281165s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.050222307 (505h41m1.921305362s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055406538 (505h41m1.926489583s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055439792 (505h41m1.926522837s) [seg_pos={0 0}, upload=443734]: Segment reached optimal threshold of 13 pieces. Dec 26 17:10:02.055713814 (505h41m1.926796869s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.055726138 (505h41m1.926809213s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056135126 (505h41m1.927218181s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056147029 (505h41m1.927230074s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056404779 (505h41m1.927487835s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056416652 (505h41m1.927499727s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056520950 (505h41m1.927604035s) [seg_pos={0 0}, upload=443734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:10:02.056636571 (505h41m1.927719616s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:10:02.056644276 (505h41m1.927727321s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:10:02.409013006 (505h41m2.280096051s) [upload=443734]: Done uploading segments... Dec 26 17:10:02.409034688 (505h41m2.280117733s) [upload=443734]: Upload finished. err: <nil> Duration: 6.441345376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...............X.......X....X.............................] duration: 42.531558931s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:11:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.531572937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X......XXX...X........X................................] duration: 3.526666924s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:11:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526694897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX.] duration: 9.919684705s trace: https://jaeger.dev.storj.io/trace/2a33c8a0896e2812 FAIL (2025-12-26 17:11:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:09:59.412405166 (505h40m59.283488221s) [upload=443734]: Starting upload Dec 26 17:09:59.412488264 (505h40m59.283571299s) [upload=443734]: Done starting upload Dec 26 17:09:59.412584588 (505h40m59.283667633s) [upload=443734]: Uploading segments... Dec 26 17:09:59.412702623 (505h40m59.283785668s) [upload=443734]: Got next segment. Inline: false Dec 26 17:09:59.412739273 (505h40m59.283822318s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:59.412746376 (505h40m59.283829431s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:59.656808823 (505h40m59.527891868s) [seg_pos={0 0}, upload=443734]: Begin upload segment... Dec 26 17:09:59.656851325 (505h40m59.52793436s) [seg_pos={0 0}, upload=443734]: Done begin upload segment. Dec 26 17:09:59.656875851 (505h40m59.527958896s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.656927179 (505h40m59.528010224s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657442641 (505h40m59.528525676s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657819287 (505h40m59.528902322s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657989451 (505h40m59.529072486s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658360388 (505h40m59.529443433s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658632737 (505h40m59.529715772s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658977594 (505h40m59.530060639s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.661716348 (505h40m59.532799393s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.662944547 (505h40m59.534027592s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.663187159 (505h40m59.534270194s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664207722 (505h40m59.535290767s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664395620 (505h40m59.535478665s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.665648896 (505h40m59.536731931s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.667001091 (505h40m59.538084126s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.668250921 (505h40m59.539333956s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:10:01.276948018 (505h41m1.148031064s) [upload=443734]: Next returned nil segment Dec 26 17:10:01.276989468 (505h41m1.148072513s) [upload=443734]: Waiting for error group managing segments... Dec 26 17:10:01.660212240 (505h41m1.531295285s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.667666294 (505h41m1.538749329s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.670284259 (505h41m1.541367304s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.680257970 (505h41m1.551341016s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.683131332 (505h41m1.554214377s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.687154904 (505h41m1.558237969s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.696419345 (505h41m1.56750239s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.724453487 (505h41m1.595536532s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.916593334 (505h41m1.787676399s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.917178638 (505h41m1.788261683s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.918198120 (505h41m1.789281165s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.050222307 (505h41m1.921305362s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055406538 (505h41m1.926489583s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055439792 (505h41m1.926522837s) [seg_pos={0 0}, upload=443734]: Segment reached optimal threshold of 13 pieces. Dec 26 17:10:02.055713814 (505h41m1.926796869s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.055726138 (505h41m1.926809213s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056135126 (505h41m1.927218181s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056147029 (505h41m1.927230074s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056404779 (505h41m1.927487835s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056416652 (505h41m1.927499727s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056520950 (505h41m1.927604035s) [seg_pos={0 0}, upload=443734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:10:02.056636571 (505h41m1.927719616s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:10:02.056644276 (505h41m1.927727321s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:10:02.409013006 (505h41m2.280096051s) [upload=443734]: Done uploading segments... Dec 26 17:10:02.409034688 (505h41m2.280117733s) [upload=443734]: Upload finished. err: <nil> Duration: 9.919721945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...............X.......X....X............................] duration: 25.26759861s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:10:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 25.267627345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX] duration: 30.004006056s trace: https://jaeger.dev.storj.io/trace/2a33c8a0896e2812 FAIL (2025-12-26 17:10:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:09:59.412405166 (505h40m59.283488221s) [upload=443734]: Starting upload Dec 26 17:09:59.412488264 (505h40m59.283571299s) [upload=443734]: Done starting upload Dec 26 17:09:59.412584588 (505h40m59.283667633s) [upload=443734]: Uploading segments... Dec 26 17:09:59.412702623 (505h40m59.283785668s) [upload=443734]: Got next segment. Inline: false Dec 26 17:09:59.412739273 (505h40m59.283822318s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:59.412746376 (505h40m59.283829431s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:59.656808823 (505h40m59.527891868s) [seg_pos={0 0}, upload=443734]: Begin upload segment... Dec 26 17:09:59.656851325 (505h40m59.52793436s) [seg_pos={0 0}, upload=443734]: Done begin upload segment. Dec 26 17:09:59.656875851 (505h40m59.527958896s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.656927179 (505h40m59.528010224s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657442641 (505h40m59.528525676s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657819287 (505h40m59.528902322s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657989451 (505h40m59.529072486s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658360388 (505h40m59.529443433s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658632737 (505h40m59.529715772s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658977594 (505h40m59.530060639s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.661716348 (505h40m59.532799393s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.662944547 (505h40m59.534027592s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.663187159 (505h40m59.534270194s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664207722 (505h40m59.535290767s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664395620 (505h40m59.535478665s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.665648896 (505h40m59.536731931s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.667001091 (505h40m59.538084126s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.668250921 (505h40m59.539333956s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:10:01.276948018 (505h41m1.148031064s) [upload=443734]: Next returned nil segment Dec 26 17:10:01.276989468 (505h41m1.148072513s) [upload=443734]: Waiting for error group managing segments... Dec 26 17:10:01.660212240 (505h41m1.531295285s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.667666294 (505h41m1.538749329s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.670284259 (505h41m1.541367304s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.680257970 (505h41m1.551341016s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.683131332 (505h41m1.554214377s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.687154904 (505h41m1.558237969s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.696419345 (505h41m1.56750239s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.724453487 (505h41m1.595536532s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.916593334 (505h41m1.787676399s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.917178638 (505h41m1.788261683s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.918198120 (505h41m1.789281165s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.050222307 (505h41m1.921305362s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055406538 (505h41m1.926489583s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055439792 (505h41m1.926522837s) [seg_pos={0 0}, upload=443734]: Segment reached optimal threshold of 13 pieces. Dec 26 17:10:02.055713814 (505h41m1.926796869s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.055726138 (505h41m1.926809213s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056135126 (505h41m1.927218181s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056147029 (505h41m1.927230074s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056404779 (505h41m1.927487835s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056416652 (505h41m1.927499727s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056520950 (505h41m1.927604035s) [seg_pos={0 0}, upload=443734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:10:02.056636571 (505h41m1.927719616s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:10:02.056644276 (505h41m1.927727321s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:10:02.409013006 (505h41m2.280096051s) [upload=443734]: Done uploading segments... Dec 26 17:10:02.409034688 (505h41m2.280117733s) [upload=443734]: Upload finished. err: <nil> Duration: 30.004017388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X...............X.......X....X...........................] duration: 42.368620661s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.368651541s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X......XXX...X........X...............................] duration: 3.890908523s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:10:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.890968307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......X] duration: 30.003075515s trace: https://jaeger.dev.storj.io/trace/7fa74dcd6777d738 FAIL (2025-12-26 17:09:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ) context deadline exceeded ======LOGS====== Dec 26 17:09:29.133151177 (505h40m29.004234222s) [upload=443729]: Starting upload Dec 26 17:09:29.133227001 (505h40m29.004310056s) [upload=443729]: Done starting upload Dec 26 17:09:29.133312353 (505h40m29.004395388s) [upload=443729]: Uploading segments... Dec 26 17:09:29.133467960 (505h40m29.004551005s) [upload=443729]: Got next segment. Inline: false Dec 26 17:09:29.133505902 (505h40m29.004588937s) [upload=443729]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:29.133511061 (505h40m29.004594117s) [upload=443729]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:29.385162329 (505h40m29.256245364s) [seg_pos={0 0}, upload=443729]: Begin upload segment... Dec 26 17:09:29.385217574 (505h40m29.256300609s) [seg_pos={0 0}, upload=443729]: Done begin upload segment. Dec 26 17:09:29.385297316 (505h40m29.256380361s) [noise=true , address=145.79.227.49:4338 , piece_id=R5JMKO5GTTPVZEU2NF2KCTQHVSRBJHLEA4XLO2WALKTWSHZ25TZQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.385311804 (505h40m29.256394839s) [noise=true , address=145.79.227.5:4222 , piece_id=SM5EUHIF3TUKSNTJFYIAPJX3GK56PHCPNM6BYNYFQIU5AXO6VOOA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386019221 (505h40m29.257102266s) [noise=true , address=77.90.31.15:4700 , piece_id=MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386171460 (505h40m29.257254505s) [noise=true , address=145.79.227.52:4446 , piece_id=267V44INJDC35GFAVI3SF3ABEXZ2EEWCBKZNB2BMXXUZPA3CBUKA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386581772 (505h40m29.257664807s) [noise=true , address=145.79.227.42:4122 , piece_id=ND6SE7IYN2I7MEXE3XWY4TLHVR62OV6YMUD7RFTM4QWME4FVJYNQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386619003 (505h40m29.257702038s) [noise=true , address=145.79.227.38:4977 , piece_id=5HZR7ZNURSLY2WIRTL4V7QODZT4RCAX2A2EW6QG5OEJ72AKGJOHQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386876864 (505h40m29.257959899s) [noise=true , address=145.79.227.3:4149 , piece_id=UK5YDQW3ISQ4DGUVICJXSHWQYOMVD2XTGFULMAD5HGZS7M36TIHA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387152179 (505h40m29.258235214s) [noise=true , address=45.91.116.6:4974 , piece_id=2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387508908 (505h40m29.258591953s) [noise=true , address=145.79.227.47:4266 , piece_id=CV7AYDKW2QU36URGSMOUIM6JO4EU7AJA7HB2WQWXQ3Q4NYAX3WKQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387793720 (505h40m29.258876755s) [noise=true , address=145.79.227.46:4601 , piece_id=GGQ7LNBL2C2YUD42SBTTNDKTTAYTJI2VMD3E35F2HNNSPVOPC46A, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.388066008 (505h40m29.259149054s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.388185926 (505h40m29.259268971s) [noise=true , address=145.79.227.50:4374 , piece_id=P6WZAIBQQWA535GX4SGGPCJC6HDZYHMM4JBOXWFWPYZRYRLXJHLQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.392835450 (505h40m29.263918485s) [noise=true , address=77.90.31.14:4663 , piece_id=GVKAGH6B37ZXF5C5ITQZO726KRGDWTD4W3KFCLYQD6JKKNROOIEQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.393323039 (505h40m29.264406074s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.394641019 (505h40m29.265724064s) [noise=true , address=145.79.227.12:4767 , piece_id=SPVO53RPSN3JFWIG4HYD2BB4XIZ2F7AAGJYA7EYYJWDDN22LDOIQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.395598301 (505h40m29.266681346s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:31.005665530 (505h40m30.876748575s) [upload=443729]: Next returned nil segment Dec 26 17:09:31.005677944 (505h40m30.876760989s) [upload=443729]: Waiting for error group managing segments... Dec 26 17:09:31.389076481 (505h40m31.260159516s) [noise=true , address=145.79.227.52:4446 , piece_id=267V44INJDC35GFAVI3SF3ABEXZ2EEWCBKZNB2BMXXUZPA3CBUKA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.391690869 (505h40m31.262773904s) [noise=true , address=145.79.227.46:4601 , piece_id=GGQ7LNBL2C2YUD42SBTTNDKTTAYTJI2VMD3E35F2HNNSPVOPC46A, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.396665723 (505h40m31.267748768s) [noise=true , address=145.79.227.5:4222 , piece_id=SM5EUHIF3TUKSNTJFYIAPJX3GK56PHCPNM6BYNYFQIU5AXO6VOOA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.398209442 (505h40m31.269292487s) [noise=true , address=145.79.227.49:4338 , piece_id=R5JMKO5GTTPVZEU2NF2KCTQHVSRBJHLEA4XLO2WALKTWSHZ25TZQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.400697820 (505h40m31.271780866s) [noise=true , address=145.79.227.38:4977 , piece_id=5HZR7ZNURSLY2WIRTL4V7QODZT4RCAX2A2EW6QG5OEJ72AKGJOHQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.408174808 (505h40m31.279257853s) [noise=true , address=145.79.227.50:4374 , piece_id=P6WZAIBQQWA535GX4SGGPCJC6HDZYHMM4JBOXWFWPYZRYRLXJHLQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.409061566 (505h40m31.280144611s) [noise=true , address=145.79.227.42:4122 , piece_id=ND6SE7IYN2I7MEXE3XWY4TLHVR62OV6YMUD7RFTM4QWME4FVJYNQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.417519532 (505h40m31.288602567s) [noise=true , address=145.79.227.3:4149 , piece_id=UK5YDQW3ISQ4DGUVICJXSHWQYOMVD2XTGFULMAD5HGZS7M36TIHA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.424664949 (505h40m31.295747984s) [noise=true , address=145.79.227.47:4266 , piece_id=CV7AYDKW2QU36URGSMOUIM6JO4EU7AJA7HB2WQWXQ3Q4NYAX3WKQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.425492514 (505h40m31.296575549s) [noise=true , address=145.79.227.12:4767 , piece_id=SPVO53RPSN3JFWIG4HYD2BB4XIZ2F7AAGJYA7EYYJWDDN22LDOIQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.664797552 (505h40m31.535880597s) [noise=true , address=45.91.116.6:4974 , piece_id=2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.682185633 (505h40m31.553268668s) [noise=true , address=77.90.31.14:4663 , piece_id=GVKAGH6B37ZXF5C5ITQZO726KRGDWTD4W3KFCLYQD6JKKNROOIEQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.706708498 (505h40m31.577791543s) [noise=true , address=77.90.31.15:4700 , piece_id=MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.706727965 (505h40m31.577811011s) [seg_pos={0 0}, upload=443729]: Segment reached optimal threshold of 13 pieces. Dec 26 17:09:31.706934148 (505h40m31.578017193s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.706941081 (505h40m31.578024116s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707030611 (505h40m31.578113646s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.707038447 (505h40m31.578121482s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707187059 (505h40m31.578270105s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.707193411 (505h40m31.578276446s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707217648 (505h40m31.578300683s) [seg_pos={0 0}, upload=443729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:09:31.707296969 (505h40m31.578380004s) [upload=443729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:09:31.707301958 (505h40m31.578384993s) [upload=443729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:09:32.015660245 (505h40m31.88674329s) [upload=443729]: Done uploading segments... Dec 26 17:09:32.015718627 (505h40m31.886801662s) [upload=443729]: Upload finished. err: <nil> Duration: 30.003086706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......] duration: 28.674908803s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:09:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 28.674937548s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X......XXX...X........X..............................] duration: 3.430116345s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:09:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430143326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X...............X.......X....X..........................] duration: 30.634200625s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:08:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 30.63425041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X......XXX...X........X.............................] duration: 3.452427401s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:08:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452454311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X.....] duration: 7.473619703s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 7.473647777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X......XXX...X........X............................] duration: 3.490434331s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:08:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490467945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.724234689s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.724258975s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X......XXX...X........X...........................] duration: 3.545993906s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:07:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546018764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 21.472964818s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 21.472995765s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X...............X.......X....X.........................] duration: 30.321610847s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 30.321640295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X....] duration: 6.126098129s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 6.126131202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.139869508s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.139898913s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X...............X.......X....X........................] duration: 29.326064837s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:07:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 29.326100797s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)