Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X...............X...X....X..............................X] duration: 1m0.001481619s trace: https://jaeger.dev.storj.io/trace/4a423f4ac85de6c5 FAIL (2025-12-07 23:42:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001501007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.............X................] duration: 3.503487634s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:42:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503509647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.71738484s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:42:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.71751133s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.7719403s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.771967662s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.762180126s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.762211427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X.............X...............] duration: 3.523087934s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:41:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523114475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.726130808s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:41:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.726158469s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.602534068s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:41:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.60256159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.761445556s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:41:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.761476496s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.6815949s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:41:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.681623704s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X..............................X.............X..............] duration: 3.490453026s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:41:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490479457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.795786108s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:40:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.795810002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...............X...X....X..............................] duration: 5.911447875s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:40:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.911579678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.623652719s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:40:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.623679108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.336970181s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:40:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.336997103s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.824891747s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:40:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.824923327s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X..............................X.............X.............] duration: 3.35045077s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:39:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.350479655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.621839617s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:39:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.621893529s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...............X...X....X.............................] duration: 5.723163372s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:39:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.723189523s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X..............................X.............X............] duration: 3.378688565s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:39:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.378715196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.630239559s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:39:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.630261142s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.736584206s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:39:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.736609523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.003566999s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:39:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.003598981s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.960284176s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:39:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.960319202s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.702824963s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:39:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.702856565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...............X...X....X............................] duration: 5.721765542s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:38:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.721791481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X..............................X.............X...........] duration: 3.460812687s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:38:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460834517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.660261226s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:38:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.660303568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...............X...X....X...........................] duration: 5.537801092s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:38:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.537828976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.226978849s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:38:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.226998656s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.595429856s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:37:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.59545816s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.771233861s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:37:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.771261656s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X..............................X.............X..........] duration: 3.538652985s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:37:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538681239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.642997783s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:37:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.64302719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X...............X...X....X..........................] duration: 5.753634041s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:37:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.753664108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.807895703s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:37:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.807921112s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.897348134s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:37:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.897384504s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X..............................X.............X.........] duration: 3.435988599s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:36:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43602068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.711057461s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:36:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.711078681s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.508044673s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:36:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.508073578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.918625553s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:36:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.918651683s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X...............X...X....X.........................] duration: 5.830860564s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:36:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.830881905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.826129534s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:36:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.826146566s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X..............................X.............X........] duration: 3.505741014s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:35:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505769469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.758104988s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:35:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.758141385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.83649403s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:35:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.836521602s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.743853467s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:35:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.743887994s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X...............X...X....X........................] duration: 5.974390726s trace: https://jaeger.dev.storj.io/trace/62df288619bfa9bb FAIL (2025-12-07 23:35:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.97442438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X......X..............................X.............X.......] duration: 3.501607093s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:35:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501651598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.770489318s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:34:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.770518495s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)