Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.429933675s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:36:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.429960245s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.745939437s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:36:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.74597347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XX.....X.............................] duration: 8.650208477s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.650226141s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 54.470482ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:36:07 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 54.498816ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.587878875s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:35:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.587907278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XX.....X............................] duration: 8.867771629s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:35:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.867803739s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.500106607s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:35:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.500144038s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 73.646129ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:35:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 73.677478ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.478858231s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:35:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.478887097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX.....X...........................] duration: 8.607813946s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:35:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.607836157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX.....X..........................] duration: 8.912615797s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:34:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.912647367s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.424177594s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:34:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.424209894s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 80.024415ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:34:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 80.060983ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.008999708s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:34:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.009030747s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 67.569397ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:33:54 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 67.599023ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX.....X.........................] duration: 8.667984838s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:33:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.668013572s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.424327237s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:33:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.424354537s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.567781269s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:33:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.567881018s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 141.469358ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:32:54 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 141.553189ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.431711309s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:32:51 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.431739643s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX.....X........................] duration: 8.586152709s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.586172015s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 157.630582ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:32:30 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 157.665629ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.449874853s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:32:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.449903117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX.....X.......................] duration: 6.813802632s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:32:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.813838449s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 94.474741ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:32:05 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 94.505088ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.47431118s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:31:57 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.474339123s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.372158203s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:31:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.372189062s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 84.191079ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:31:44 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 84.260822ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.423331223s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:31:34 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.423376297s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.402761053s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:31:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.402789747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX.....X......................] duration: 7.923864965s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:31:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.923895224s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 351.795425ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:30:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 351.824902ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.434059974s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:30:30 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.434093957s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.388340779s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:30:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.38837692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX.....X.....................] duration: 7.802204634s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:29:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.802222258s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 91.604687ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:29:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 91.629524ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.45035373s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:29:50 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.45038003s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.425822466s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:29:20 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.425849317s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.505457373s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:29:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.505486268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.431651262s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:28:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.431699825s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 218.313367ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:28:56 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 218.338184ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX.....X....................] duration: 8.189782195s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:28:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.189814137s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000857129s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:28:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.000884882s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XX.....X...................] duration: 7.12409073s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:27:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.124118712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.113747856s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:27:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.113768616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.473109469s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:27:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.473143663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.95417875s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:27:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.95430516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XX.....X..................] duration: 7.688017354s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.688076057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX.....X.................] duration: 15.899984758s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:27:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.900009105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.011277265s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:26:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.011600701s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)